Does deepof work with Sleap (.slp) files?

[catubc]

Hello We are looking forward to trying your pipeline, but weren't sure about 2 issues:

1. Our tracks are in .slp files from sleap (https://github.com/talmolab/sleap). They are essentially simple arrays such as this : [n_frames, n_animals, n_features, 2dimensions].

2. Some of the animals features are periodically occluded (e.g. on some frames we may be missing an animal, or some of the features).

We are happy to write code adapting these tracks to deepof. Do you have any suggestions or can provide some starter notebooks?

Thanks so much! catubc

[lucasmiranda42]

Hi, @catubc! And thank you for your interest in DeepOF 😊

At the moment the program does unfortunately not directly support .slp files as input. However, we'd be happy to help create an integration (which shouldn't be harder than a couple of functions, as far as I gather from your issue). Regarding occlusions, DeepOF masks them from further analysis based on the likelihood returned by DLC. Would we have an equivalent metric in the sleap files? If we do, this should not be a problem in principle.

Feel free to post a reduced version of an example file that we could use as a starting point (or point us towards a public dataset with the same characteristics). Contributions are also welcome if you'd like to provide code as a pull request! (you can find the contribution guide in the main README file 🙂).

Best, thank you once again, and looking forward! Lucas

[lisadiez]

Hi Lucas, I'm Lisa - I work with Cat.

Here's a link to a npy file from our Sleap tracks: https://drive.google.com/file/d/1bZ1Rw6LwEao1Lt7MG6DcZX4HIKxRDhb9/view?usp=sharing It's a (28802, 6, 6, 2) array, corresponding to (n_frames, n_animals, n_nodes, x & y locations).

Our probabilities are essentially 1 for any non NaN value, and 0 for all NaN values.

Let me know if you need anything else from us! Best, Lisa

[lucasmiranda42]

Dear Lisa,

Amazing, thank you! The only missing detail would be the ID of each node (i.e. which body part does each node represent :)). Does sleap output anything (such as a metadata file) including this information? With that we should we good to go!

Best, Lucas

[lisadiez]

Hi again Lucas!

Node IDs: 0 = Nose, 1 to 4 = Spine points, 5 = Base of tail

Sleap does provide metadata - I put the original .slp file here if you want to take a look.

Best, Lisa

[lucasmiranda42]

Dear Lisa,

excellent! We're preparing an integration for the next release. I'll leave the issue open and let you know when this is solved :)

Best, Lucas

[lucasmiranda42]

Dear @lisadiez, dear @catubc,

You can find a first implementation of our SLEAP integration in the development branch.

For now, only .npy input is supported, and you have to pass your body part names to the Project object as an extra parameter, called rename_bodyparts. The function has been integrated into our testing pipeline, but we'd really appreciate it if you could test it on your own and report issues / make suggestions!

Here is an example of how you'd create a SLEAP/DeepOF project:

my_deepof_project = deepof.data.Project(
                project_path=os.path.join("tutorial_files"),
                video_path=os.path.join("tutorial_files/Videos/"),
                table_path=os.path.join("tutorial_files/Tables/"), # Where your .npy files are stored
                project_name="deepof_project",
                arena="polygonal-autodetect",
                rename_bodyparts=["your", "bodyparts", "as", "a", "list"],
                animal_ids=["B", "W"],
                table_format=".npy",
                video_scale=380, # Set to the arena 
)

Please note that the development branch also has some new features regarding arena detection, which is now fully automatic. See this issue for details.

Best, and hope it helps! Feel free to open a new issue if you have any questions and/or feedback :) Lucas