search

tee-ar-ex / trx-python

Python implementation of the TRX file format
https://tee-ar-ex.github.io/trx-python/
BSD 2-Clause "Simplified" License
22 stars 15 forks source link

Community file-format technological survey discussion #1

Open frheault opened 3 years ago

frheault commented 3 years ago

General goals

What is 'supported' by the memmap implementation?

(Initial thread: https://github.com/nipy/nibabel/issues/942)

frheault commented 3 years ago

Here is a draft of the (simple) specifications: Still ongoing work (I don't know exactly what to add since it has to cover multiple languages) https://drive.google.com/file/d/1DVKisuoENqU5Q_652wZdQNrcerxKC2xL/view?usp=sharing

Here is data to test the current implementation. It is pretty big because to truly test speed I had to generate a large tractogram with a lot of metadata. https://drive.google.com/drive/folders/1fjxyLskcFXYizg6sDNMrRPUu-N1PdvZu?usp=sharing

It would be nice to have comments on the current specifications, but if someone wanted to propose a new format to showcase. I can easily move "my" readme to my module, change the setup accordingly, or accommodate any new languages so the repository is simply hosting code and PDF. As for now, I made it easy to use and to test for my only proposition, but anyone that wants to expand can tell me and we can video chat and then plan a PR to accommodate new ideas.

If it is desired by someone, I could explain in detail the specifications and implementation or examples. To facilitate discussion it could be a conference call so we can go over it quickly as a group discussion about my implementation, limitations, etc. (largely inspired, if not all, by recommendations of @arokem @jdtournier @neurolabusc).

Again, it is important to re-iterate that this is not about the specific implementation or code or language, it is mostly about the specifications and file format descriptions/contents. My code is only there because I personally liked this idea and wanted to test if the idea could indeed achieve what was on the list of features.

Lestropie commented 3 years ago

Few little suggestions:

frheault commented 3 years ago

@arokem @francopestilli @frankyeh @MarcCote @neurolabusc @Garyfallidis @jchoude @mdesco @jdtournier @ppoulin @gabknight

If anyone wants to go over questions and discussions in the Issues sections, it would help to have a boost before the Dipy Meeting next Wednesday.

Sorry to add Github notification like that, but I don't who has time for that or not. https://github.com/dipy/dipy/issues/2229#issuecomment-733475223

emanuele commented 3 years ago

@frheault unfortunately I couldn't participate in the meeting but I am pretty interested in the discussion. Following a recent request of @MarcCote and a few others before him, I made available the large tractogram (500K streamlines) I used to benchmark my fast TRK loader (https://github.com/emanuele/load_trk): https://nilab.cimec.unitn.it/people/olivetti/data/sub-100206_var-FNAL_tract.trk . And here is another, much larger, test tractogram (10M streamlines, 2.9Gb): https://nilab.cimec.unitn.it/people/olivetti/data/sub-599469_var-10M_tract.trk

frheault commented 3 years ago

Hello everyone,

@arokem @francopestilli @frankyeh @MarcCote @neurolabusc @Garyfallidis @jchoude @jdtournier @ppoulin @gabknight @Lestropie @StongeEtienne

I don't want to leave the project die again, but I cannot unilaterally decide most of the issues, if no one else have opinions on the various issues raised by me or @Lestropie (in the issues section) I will have to make a decision which might not please some people. So if you have something to add, we should re-start this discussion immediately.

Also, I don't have the time to do a C++ (or rust, matlab, C) reader and writer to try out the specification. I think it would be important that someone else than me, independently tries to do a small&simple reader/writer to check if something is unclear if there is limitations or if something is impossible to implement.

Is there anyone interested in a video call to discuss implementation in any other language? This would be very important, once this is done we could settle a few points in the issues and go forward.

jdtournier commented 3 years ago

I'd love to volunteer for the C++ implementation, but my teaching load is pretty intense at the moment, I'm finding it difficult to get anything done other than that. I'll have a think, and maybe I can cobble together a really quick & dirty proof of concept - but no guarantees... If anyone else feels they have the time and the inclination, feel free to put your hand up!

neurolabusc commented 3 years ago

I also have a busy teaching load this term (and no fall break). However, in December I was on furlough and spent a little time looking at this problem. I found an elegant C wrapper for zip files. It is a very minimal, self contained wrapper, which seemed elegant to me. https://github.com/kuba--/zip

I created a basic project showing how it could be used to view the contents of the TRX format: https://github.com/neurolabusc/zip

-chris

On Jan 19, 2021, at 11:13 AM, J-Donald Tournier notifications@github.com<mailto:notifications@github.com> wrote:

I'd love to volunteer for the C++ implementation, but my teaching load is pretty intense at the moment, I'm finding it difficult to get anything done other than that. I'll have a think, and maybe I can cobble together a really quick & dirty proof of concept - but no guarantees... If anyone else feels they have the time and the inclination, feel free to put your hand up!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://protect2.fireeye.com/v1/url?k=a9694394-f6f27b7e-a9690d55-86861b72af13-50fc01c0355e3455&q=1&e=03c561cc-8053-4713-a207-215d4c524d86&u=https%3A%2F%2Fgithub.com%2Ffrheault%2Ftractography_file_format%2Fissues%2F1%23issuecomment-762948561, or unsubscribehttps://protect2.fireeye.com/v1/url?k=aa870693-f51c3e79-aa874852-86861b72af13-7d0f900ea8c15825&q=1&e=03c561cc-8053-4713-a207-215d4c524d86&u=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FACEEL57UV43WQ24JJLL2IC3S2WVTFANCNFSM4TVANJCA.

jdtournier commented 3 years ago

Good to see you've already had a go, @neurolabusc!

I was looking to use libzip here, it seems pretty active and well maintained. I'll take a look at what you've done when I get the chance...

Lestropie commented 3 years ago

I had naive plans to start looking into MRtrix3 support (which will be neither small nor simple) over the new year break, but unsurprisingly it didn't happen.

I'm not sure that a small & simple implementation will actually have the capability to identify issues with the specification. It's going to be working towards GUI support & the more exotic use cases that the pros and cons of different formulations are going to come out. So we may be more dependent on foresight rather than crashing into things after the fact.

frheault commented 3 years ago

Glad to see that attempts at this will be made in C++, @Lestropie about the small&simple it's true maybe it is not enough. If someone does it as they see fit and we encounter corner cases that's perfect!

@arokem @Garyfallidis I don't even know where to start to convert my existing code into something that would nicely fit into Nibabel. My current code is like a showcase, independent and in a vacuum. Would you be interested in a call to talk specifically about class, function design to refactor my code? I think re-implementing my code into a Nibabel branch right away would be nice.

Just to re-share, here is some example of my how to use my code. But also trx/tck/trk side by side to try to load the same data. https://vanderbilt365-my.sharepoint.com/:f:/g/personal/francois_rheault_vanderbilt_edu/ElXpfTBbbkVDq44-yy_FJicBYva86qHi5zBbFihelDsP9A?e=0gI5us