bforsbe
commented
8 years ago Original comment by Özkan Yildiz (Bitbucket: oeyildiz, GitHub: Unknown):
We also observed the same problem now with hang ups at the end of the expectation iteration using Relion 1.4 (no GPU usage of course) when we added the flag --dont_combine_weights_via_disc, which is the default in Relion 2.0. It did not matter if it was 2D classification or 3D classification or refinement. When the hang ups appeared it was reproducible when repeating the same run. We omitted this flag again and it worked fine afterwards with no hang ups.
Therefore, it is maybe necessary to check if this flag really fits to a certain system or not. We usually never used it with Relion 1.4 and never had problems so far.
Originally reported by: Dari Kimanius (Bitbucket: dkimanius, GitHub: dkimanius)
Reported by Özkan Yildiz in Issue #24, moved here due to being a different issue:
We repeated now the same 2D classification with v2.0.b10 after doing it with v2.0.b9 where we observed hang ups of the GPU cards at the end of its 7th iteration. The processes stop exactly at the same iteration 7 and at the same time and one GPU keeps on running on 100 % while the other goes to 0%. We therefore observed the behavior of the two GPU cards during this iteration. It seems that before the event of loosing one GPU, the memory consumption (and 3% does not seem to be much) that was equal on both our 2 GPU cards in previous iterations gets shifted during iteration 7 to only one GPU to the double amount (see below). And after a while the second zero-memory consuming GPU goes down (see below) after idling around some time at 100 %. So it looks like one GPU card took over the whole work of the second card. The iteration with the hang up takes longer than it should in theory, and the output hangs just before the mouse gets to its end. We noticed that after around 30 min (assuming this would be the time needed for the whole iteration) the time for the expectation starts rising to about 50 min and than one of the cards does not consume memory anymore. Maybe there is something wrong with parallelisation? The temperatures for both GPU cards seem to be OK and there is no sign that one GPU card has a temperature problem. We are doing 3D classifications on the same GPU cards and they work perfectly fine. We are also doing the same 2D classification on only CPUs and it is running fine. Now, we are going to try to do the same 2D classification on only one GPU card (with half the number of CPUs in order to see if the same thing happens. Usual output of nvidia-smi dmon during iteration 6
output of nvidia-smi dmon when the iteration 7 hangs and one GPU goes down
Commandline
Output of iteratin 6 and 7
stops at this point and no error message is shown.