search

openworm / tracker-commons

Compilation of information and code bases related to open-source trackers for C. elegans
11 stars 12 forks source link

Files specification is too complicated #153

Closed Ichoran closed 7 years ago

Ichoran commented 7 years ago

The specification has us carrying around bits and pieces of file names. This is fiddly and error-prone, especially for languages that aren't very good at string manipulation.

Instead, we should simplify to the current file being a file name and/or path; and the next and prev files, if present, being the entire file name. Zipping the files separately should be okay (otherwise really huge data sets might be a pain to work with).

MichaelCurrie commented 7 years ago

This is a possible improvement. Since I have a large backlog of changes I need to implement for the Python version ahead of the conference, is it possible to leave this as a potential change for the future? I am curious to hear feedback from other labs about the format as well; perhaps they will have feedback that will cause us to prioritize certain changes.

Ichoran commented 7 years ago

I think I was the one who introduced the format but in retrospect I think it's just too error-prone to stay. The problem in addition to complexity is that it's easy to get it wrong; if you have files wm1.wcon through wm12.wcon and you have 1 as the last bit of the file, it's not very hard to start using the same rule from wm11.wcon as wm1.wcon--even though wm1.wcon says 11 in its next files, if you were using local just-the-previous-and-next files for wm11 you might encode its last digit as 1, etc..

So I'm fine with leaving it out as long as possible, but I think we'd better fix that design flaw before we release it to people. If I get time to fix the Python implementation, would that help?

Ichoran commented 7 years ago

@MichaelCurrie - I had a look at the Python code (around line 660 in wcon_parser.py) and it's pretty straightforward. The code already doesn't handle writing multiple files, just reading. I can make the change if it would help.

MichaelCurrie commented 7 years ago

Sure if you want to PR it that would be a big help! Thanks