search

moloney / dcmstack

DICOM to Nifti conversion with meta data preservation
Other
72 stars 52 forks source link

Paralellization and multithreading support? #30

Closed TheChymera closed 9 years ago

TheChymera commented 9 years ago

Hi, seeing how grouping/conversion takes a lot of time, and I am still unable to use group_and_stack via nipype, I set out to paralellize my current dcmstack/for-loop workflow.

You can see the version I am refering to right now here.

Strangely, running a pool of size 2 or size 4, or 8, or 16 (4 cores on this machine), I see n marked speed improvement. Any idea why that could be? is your code already paralellizing stuff? Would group_and_stack (which I am not atm using) distribute its tasks over multiple threads?

TheChymera commented 9 years ago

@moloney ?

moloney commented 9 years ago

Nothing in dcmstack is parallel. You are probably mostly I/O bound, and if you are CPU bound then threads won't really help due the GIL.

If you don't need all the meta data, you could try using the minimal_extractor instead of the default_extractor. That should speed things up considerably.

If you do want all the meta data you could try using multiprocessing to parallelize the parsing. I usually just run multiple conversions in parallel on our cluster rather than trying to speed up the conversion of a single series.

Anyway, these discussions should really happen on the nipy mailing list rather than in github issues.

TheChymera commented 9 years ago

@moloney it seems the nipy mailing list is no longer? and discussions should move to github? https://groups.google.com/forum/#!topic/nipy-user/LMfhvcDgfck

moloney commented 9 years ago

Look here: http://mail.scipy.org/mailman/listinfo/nipy-devel