search

HabanaAI / Model-References

Reference models for Intel(R) Gaudi(R) AI Accelerator
141 stars 67 forks source link

Memory of gaudi is occupied fully no mater how many batchsize is #15

Closed anti-machinee closed 2 years ago

anti-machinee commented 2 years ago

I use batch size = 32 or 64, gaudi is occupied all 32gb/32gb. Is it the mechanism of gaudi and Habana ? So that 32gb does not mean any thing?

greg-serochi commented 2 years ago

Hi @anti-machinee, in Gaudi, the driver allocates all the available memory to the card, even if the application or command may not need the full memory.

greg-serochi commented 2 years ago

in a future release, we are planning to give additional information on the on-card memory usage

greg-serochi commented 2 years ago

Hi @anti-machinee we have a new API in our new release to better check these Memory Usage https://docs.habana.ai/en/latest/PyTorch/PyTorch_User_Guide/Python_Packages.html#memory-stats-apis