search

rly / ndx-pose

NWB extension to store pose estimation data
BSD 3-Clause "New" or "Revised" License
13 stars 11 forks source link

Support for HRNet/Docs? #19

Open SkepticRaven opened 1 year ago

SkepticRaven commented 1 year ago

Our lab groups pose estimation (https://github.com/KumarLabJax/deep-hrnet-mouse).

However, figuring out the exact format+fields of stored data is difficult to follow. I couldn't get the docs to build successfully.

cd docs
make fulldoc

First error is easy, since there might be a typo in the template:

OSError: namespace file '..\spec/ndx-pose.namespace.yaml' not found
---
grep -Rn '\.\.\\'
docs/source/conf_doc_autogen.py:9:spec_input_spec_dir = '..\spec'

However, I can't get past the next one:

...
 File "[snip]/lib/python3.10/site-packages/hdmf_docutils/doctools/renderrst.py", line 644, in render_spec_table
    elif spec.data_type_def is not None:
AttributeError: 'LinkSpec' object has no attribute 'data_type_def'. Did you mean: 'data_type_inc'?

The usage on the main README is useful, but it doesn't describe how to add in skeleton information.

talmo commented 1 year ago

Hey @SkepticRaven,

If it helps, check out our implementation for NWB writing: https://github.com/talmolab/sleap-io/blob/e0ead827e484174a918ff091e05a6ae90ded5e43/sleap_io/io/nwb.py

If you have some test data, we'd be happy to add a parser from your HRNet outputs to our Labels data structure so you have an NWB converter for free through that library!

Here's a general example of what that would look like if you want to take a stab at it yourself:

https://github.com/talmolab/sleap-io#create-labels-from-raw-data

SkepticRaven commented 1 year ago

Yeah, I was aware that SLEAP had a slew of nice converters!

Our pose data should be pretty straight forward to swap between formats. We have some "test" data with in our JABS project over in this folder. The description of pose_v2 and v3 are over in the repo that produces them here. We haven't released v4+, but those add better identity, static objects, and segmentation.

I was investigating the limits of what information is stored (in all the different formats).

We seem to store a handful more info than what's currently out there. Our latest models (not yet shared publicly) include segmentation, learned identity embedding vectors (n-dimensional), and static objects (eg location of water spout). We also have plans for dynamic objects (non-mouse objects that can be moved around).

DLC has some functionality for static objects with their "Uniquebodyparts", but I was unsure if this made their way into NWB format.