ezpzbz / aiida-catmat

Collection of AiiDA WorkChains Developed in CATMAT project
MIT License
3 stars 1 forks source link

Heavy parsing of OUTCAR #32

Closed ezpzbz closed 4 years ago

ezpzbz commented 4 years ago

Now, we are completely dependent on pymatgen parsers. This is quite okay with parsing vasprun.xml but causes serious issues with OUTCAR. There is an open issue in AiiDA that discusses the problem in detail. https://github.com/aiidateam/aiida-core/issues/3973

Long story short, parsing OUTCAR can take termendous amount of time to be completed. In few cases that I followed, it takes up to 10-12 minutes to finish parsing and meanwhile it does not release the GIL. It consequently causes the RabbitMQ to miss two consecutive heartbeats (60+60 seconds) and therefore, it assumes process is lost/dead and gives the task to another worker to complete. Same thing will happen with the second worker. Meanwhile, first worker finishes parsing and its job and even registers the resulting nodes in the database. Right after it, second worker also does the same and AiiDA stops it as there is already sealed node in the database, and finally workchain fails.

There can be several solutions to this issue:

Steps to be done: