search

google-research / circuit_training

Apache License 2.0
1.2k stars 189 forks source link

CPU RAM Usage #44

Open cr145 opened 2 years ago

cr145 commented 2 years ago

I'm having issues with the amount of CPU RAM being used by train_ppo.py. Specifically, over the course of training the memory usage steadily increases until there is no memory left, causing an error. This seems odd as I would expect the total memory usage to be roughly constant over the course of training as the model and dataset are both a fixed size. Does anyone have an idea as to why this is the case? Has anyone else experience similar issues, and been able to alleviate them?

ghost commented 1 year ago

@cr145
Hello, I thought there was one problem related to plc_main_wrapper which saved a lot of caches files during std cell placement. Pls try to delete all cached files by using this comment every time you start training to make sure all cached files are being deleted

cd /tmp/
sudo find -type f -name 'plc_wrapper_main*' -delete