search

soft-matter / trackpy

Python particle tracking toolkit
http://soft-matter.github.io/trackpy
Other
443 stars 131 forks source link

Linking using multiple HDF5 files. #751

Closed wheelern closed 8 months ago

wheelern commented 8 months ago

This isn't an issue, but the repo felt like the best place to ask this question. Let me know if I should go elsewhere.

I am tracking features in long-ish videos (1 hr, 65,000 frames) that, for a variety of performance and storage reasons, we split into chunks during recording - usually 18,000 frames per video, but this can be changed. I am using tp.batch() to parallelize location of features and writing the data to HDF5 files. Linking features within each file works well, but I'm not sure how to link features across files.

What's the best way to go about this? I've tried to merge HDF5s into a single file, while renaming each group (Frame) and replacing each frame column, but as a new user of PyTables, this has been tricky. Is there a better way to do it, or do you have any guidance for editing and merging HDF5s?

wheelern commented 8 months ago

Ended up reading and dumping into a single very large data frame.