search

sensein / etelemetry-client

Etelemetry client
Other
3 stars 9 forks source link

Multiproc tools submit multiple times #8

Closed effigies closed 4 years ago

effigies commented 4 years ago

The way multiprocessing works, each subprocess will end up importing a library that uses etelemetry, and so my usage will often get multiplied by the number of cores available. So, for instance, because fmriprep uses nipype with the MultiProc plugin, on an 8 core system I ping the server 8 times. Possibly more as workers are expired and restarted...

effigies commented 4 years ago

Update: Definitely more than 8 times, as workers are expired.

satra commented 4 years ago

@effigies - is this still the case? and can you post a short example to test this? my limited testing didn't return any version logging.

effigies commented 4 years ago

Latest fmriprep:

200111-18:01:10,741 nipype.workflow INFO:
     [Node] Setting-up "fmriprep_wf.single_subject_01_wf.anat_preproc_wf.surface_recon_wf.autorecon1" in "/scratch/fmriprep_wf/single_subject_01_wf/anat_preproc_wf/surface_recon_wf/autorecon1".
200111-18:01:10,745 nipype.interface INFO:
     recon-all complete : Not running
200111-18:01:10,745 nipype.workflow INFO:
     [Node] Running "autorecon1" ("smriprep.interfaces.freesurfer.ReconAll"), a CommandLine Interface with command:
echo recon-all: nothing to do
200111-18:01:10,748 nipype.interface INFO:
     recon-all complete : Not running
200111-18:01:11,65 nipype.utils INFO:
     Running nipype version 1.3.1 (latest: 1.4.0)
200111-18:01:11,111 nipype.workflow INFO:
     [Node] Finished "fmriprep_wf.single_subject_01_wf.anat_preproc_wf.surface_recon_wf.autorecon1".
200111-18:02:10,806 nipype.workflow INFO:
     [Node] Setting-up "fmriprep_wf.single_subject_01_wf.anat_preproc_wf.surface_recon_wf.skull_strip_extern" in "/scratch/fmriprep_wf/single_subject_01_wf/anat_preproc_wf/surface_recon_wf/skull_strip_extern".
200111-18:02:10,809 nipype.workflow INFO:
     [Node] Running "skull_strip_extern" ("niworkflows.interfaces.freesurfer.FSInjectBrainExtracted")
200111-18:02:10,813 nipype.workflow INFO:
     [Node] Finished "fmriprep_wf.single_subject_01_wf.anat_preproc_wf.surface_recon_wf.skull_strip_extern".
200111-18:02:15,105 nipype.workflow INFO:
     [Node] Setting-up "fmriprep_wf.single_subject_01_wf.anat_preproc_wf.surface_recon_wf.autorecon_resume_wf.autorecon2_vol" in "/scratch/fmriprep_wf/single_subject_01_wf/anat_preproc_wf/surface_recon_wf/autorecon_resume_wf/autorecon2_vol".
200111-18:02:15,109 nipype.interface INFO:
     recon-all complete : Not running
200111-18:02:15,109 nipype.workflow INFO:
     [Node] Running "autorecon2_vol" ("smriprep.interfaces.freesurfer.ReconAll"), a CommandLine Interface with command:
echo recon-all: nothing to do
200111-18:02:15,111 nipype.interface INFO:
     recon-all complete : Not running
200111-18:02:15,427 nipype.utils INFO:
     Running nipype version 1.3.1 (latest: 1.4.0)
200111-18:02:15,475 nipype.workflow INFO:
     [Node] Finished "fmriprep_wf.single_subject_01_wf.anat_preproc_wf.surface_recon_wf.autorecon_resume_wf.autorecon2_vol".
200111-18:02:21,113 nipype.workflow INFO:
     [Node] Setting-up "_autorecon_surfs0" in "/scratch/fmriprep_wf/single_subject_01_wf/anat_preproc_wf/surface_recon_wf/autorecon_resume_wf/autorecon_surfs/mapflow/_autorecon_surfs0".
200111-18:02:21,117 nipype.interface INFO:
     recon-all complete : Not running
200111-18:02:21,117 nipype.workflow INFO:
     [Node] Running "_autorecon_surfs0" ("smriprep.interfaces.freesurfer.ReconAll"), a CommandLine Interface with command:
echo recon-all: nothing to do
200111-18:02:21,120 nipype.interface INFO:
     recon-all complete : Not running
200111-18:02:21,447 nipype.utils INFO:
     Running nipype version 1.3.1 (latest: 1.4.0)
200111-18:02:21,473 nipype.workflow INFO:
     [Node] Finished "_autorecon_surfs0".
200111-18:02:25,416 nipype.workflow INFO:
     [Node] Setting-up "_autorecon_surfs1" in "/scratch/fmriprep_wf/single_subject_01_wf/anat_preproc_wf/surface_recon_wf/autorecon_resume_wf/autorecon_surfs/mapflow/_autorecon_surfs1".
200111-18:02:25,421 nipype.interface INFO:
     recon-all complete : Not running
200111-18:02:25,421 nipype.workflow INFO:
     [Node] Running "_autorecon_surfs1" ("smriprep.interfaces.freesurfer.ReconAll"), a CommandLine Interface with command:
echo recon-all: nothing to do
200111-18:02:25,424 nipype.interface INFO:
     recon-all complete : Not running
200111-18:02:25,754 nipype.utils INFO:
     Running nipype version 1.3.1 (latest: 1.4.0)
200111-18:02:25,790 nipype.workflow INFO:
     [Node] Finished "_autorecon_surfs1".
200111-18:02:29,397 nipype.workflow INFO:
     [Node] Setting-up "fmriprep_wf.single_subject_01_wf.anat_preproc_wf.surface_recon_wf.autorecon_resume_wf.autorecon_surfs" in "/scratch/fmriprep_wf/single_subject_01_wf/anat_preproc_wf/surface_recon_wf/autorecon_resume_wf/autorecon_surfs".
200111-18:02:29,405 nipype.workflow INFO:
     [Node] Setting-up "_autorecon_surfs0" in "/scratch/fmriprep_wf/single_subject_01_wf/anat_preproc_wf/surface_recon_wf/autorecon_resume_wf/autorecon_surfs/mapflow/_autorecon_surfs0".
200111-18:02:29,409 nipype.interface INFO:
     recon-all complete : Not running
200111-18:02:29,409 nipype.workflow INFO:
     [Node] Running "_autorecon_surfs0" ("smriprep.interfaces.freesurfer.ReconAll"), a CommandLine Interface with command:
echo recon-all: nothing to do
200111-18:02:29,412 nipype.interface INFO:
     recon-all complete : Not running
200111-18:02:29,721 nipype.utils INFO:
     Running nipype version 1.3.1 (latest: 1.4.0)
200111-18:02:29,736 nipype.workflow INFO:
satra commented 4 years ago

i'm not even getting a single nipype version with the current fmriprep docker unstable (do you disable ET in the docker container) - i'm running through singularity.

effigies commented 4 years ago

No, I'm using the docker image.

satra commented 4 years ago

the reason i wasn't seeing it is because my versions were the same.

posted a PR to nipype. will have to think about consequences.

satra commented 4 years ago

addressed in nipype via nipype-specific environment variable.

effigies commented 4 years ago

True, but any other libraries would need to do the same.

Maybe we do want to simply set NO_ET in multiproc.

satra commented 4 years ago

i think this is like OMP_NUM_THREADS. someone could set NO_ET at a global level, but since different packages in different settings could want their own counter. i think it should be left to the package how they want to deploy it. from the nipype side, which is often a library for other packages now, we can try to do something (which is what we have done with NIPYPE_NO_ET).