search

SIMEXP / niak

GNU Octave library for connectivity analysis in large fMRI datasets
http://niak.simexp-lab.org
Other
28 stars 21 forks source link

niak clogs /tmp and kills the OS #391

Open bpinsard opened 5 years ago

bpinsard commented 5 years ago

niak puts a lot of stuff in /tmp and never clear it up... From looking into the code, the place for tmp is set by TMPDIR env variable. This should be documented to help people to place temp files in an appropriate location (eg. /scratch/).

pbellec commented 5 years ago

A niak run that completes should leave no trace in tmp.

If there is a crash, or many concurrent instances, it is very possible that this creates lots of left-over. It is better practice to use scratch as a tmp indeed, or whatever dedicated space there is for there. This is a standard configuration step on a HPC, and there is generally a recommended folder for it that needs to be specified. This indeed highly needs to be better documented, and has been a major source of issues for people who tried to install niak.

Typically, that aspect of the configuration is done once and for all on a given niak install through the psom_gb_vars_local.m file, and specifically the value of psom_gb_tmp. psom_gb_vars_local.m is a copy of the following file, which is edited by the user to set her own local configurations (whatever psom_gb_tmp_local is found first in the path of a given octave session will superseed all other configuration files) https://github.com/SIMEXP/psom/blob/master/psom_gb_vars.m#L70 ideally this parameter would be exposed to script users through opt.psom.tmp, but this is not currently implemented.

It should indeed be documented in http://psom.simexp-lab.org/psom_configuration.html and http://niak.simexp-lab.org/niak_installation.html Another way to deal with this is to mount whatever tmp location should be used on the /tmp of the docker container. This should be done in any case, because otherwise the container /tmp is an in-memory virtual folder, which gets clogged really fast and creates chaos for windows and mac users...

FYI if you feel inspired to try and start a PR, location for the niak install doc is: https://github.com/SIMEXP/niak/blob/gh-pages/niak_installation.md and PSOM config doc is there: https://github.com/SIMEXP/psom/blob/gh-pages/psom_configuration.md

pbellec commented 5 years ago

Another note: it may also be that the current niak install does leak files in the tmp. I have tested periodically it's not the case, but it's not part of our automated testing. If you can confirm such a leak, it would be appreciated. I'll create a separate issue to add this in the niak tests.

HanadS commented 5 years ago

found some documentation on this http://psom.simexp-lab.org/how_to_use_psom.html#misc-temporary-files

pbellec commented 5 years ago

Thanks for pointing out @hanads! It's not the best place for it, but at least it exists