tee-ar-ex / trx-python

Python implementation of the TRX file format
https://tee-ar-ex.github.io/trx-python/
BSD 2-Clause "Simplified" License
22 stars 15 forks source link

create TRX from LazyTractogram or Generator #45

Closed frheault closed 1 year ago

frheault commented 2 years ago

Create TRX file from Generator or LazyTractogram and write to memmap in chunks. Support a buffer to avoid resizing the TRX at every chunk.

Added tests to verify if it is working.

frheault commented 2 years ago

@arokem It is still a bit rought, but is this what you had in mind? For me it is working pretty well.

In the tracking_introduction_eudx.py example from Dipy doing this works:

streamlines_generator = LocalTracking(csa_peaks, stopping_criterion, seeds,
                                      affine=affine, step_size=.5)
#streamlines = Streamlines(streamlines_generator)
trx = TrxFile.from_lazy_tractogram(streamlines_generator, hardi_img, extra_buffer=100000)

Using an extra_buffer of 0 will recreate a new TrxFile at each chunk (few thousand streamlines) which is a lot of new temporary folders and all and rewriting data. Using a bigger extra_buffer will create an empty TrxFile that can use preallocations and only get resized (if needed) a few times and the function finishes with a final resize.

I did not benchmark it.

arokem commented 1 year ago

Is this still WIP?

frheault commented 1 year ago

Is this still WIP?

With #37 passed and the test passing, I think this is ready for testing.

frheault commented 1 year ago

I need to test with larger files to quantify how much time is saved or how much allocations are avoided.

I know it works but I want to be able to have a few numbers to back it up.

arokem commented 1 year ago

Sounds good! No rush.