search

NTMC-Community / MatchZoo

Facilitating the design, comparison and sharing of deep text matching models.
Apache License 2.0
3.84k stars 900 forks source link

how can I increase gpu usage? #826

Closed arezooSarvi closed 4 years ago

arezooSarvi commented 4 years ago

I'm using MatchZoo for a while now, but when I'm running the models on gpu, it seems like it doesn’t use it properly. I'm using TensorFlow-gpu, it finds everything and starts running but gpu usage is very low (around 5%), I tried this on several models (dssm, arc-I, arc-II, drrm, mvlstm, etc), using different datasets (long/short documents), and parameters, for ranking task, and I'm mainly following the examples in the tutorial. Is there anything else I should consider when using gpu?

uduse commented 4 years ago

Is 5% memory usage? If so, try increasing batch sizes.

arezooSarvi commented 4 years ago

no it's gpu usage

uduse commented 4 years ago

@arezooSarvi I meant GPU memory usage. Crank up batch size as large as possible.

arezooSarvi commented 4 years ago

Thanks @uduse but it is not memory, by 5% I meant the average percentage of time spent on GPU computations, the memory usage is even smaller. As I mentioned I used several different set of parameters including batch size, with/without multiprocessing and different # of workers, but non of these seems to affect the computation time on gpu

uduse commented 4 years ago

That's weird. Possible reasons I can think of: (1) your script spent too much time on preprocessing compared to training (2) you are not actually using the GPU (3) you are still not setting batch size large enough, try something like 100000