error during sleap-track

[aminerva]

I have run into the same issue with sleap-track as previously reported in #1871.

When executing the sleap-track command in a script submitted to a SLURM scheduler, TensorFlow generates the following warning repeatedly (see attached log as well):

W tensorflow/core/data/root_dataset.cc:163] Optimization loop failed: CANCELLED: Operation was cancelled

sleap_predict_221414_221414.log

As mentioned in the previous issue report, I updated sleap to the most recent version (1.4.1a2), however now it looks like sleap never actually begins predictions as it never moves past the following line (see attached log as well):

2024-10-06 09:42:52.843370: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

sleap_predict_221415_221415.log

[github-actions[bot]]

Thank you for opening a new issue!

We appreciate your input and will review it soon.

[!WARNING]
A friendly reminder that this is a public forum. Please be cautious when clicking links, downloading files, or running scripts posted by others.

• Always verify the credibility of links and code.
• Avoid running scripts or installing files from untrusted sources.
• If you're unsure, ask for clarification before proceeding.

Stay safe and happy SLEAPing!

Best regards, The Team

[eberrigan]

Hi @aminerva,

Thanks for sending your log. How did you install SLEAP? If you could send me your environment info that would be super helpful!

A combination of conda list and pip list should suffice.

Best,

Elizabeth

[eberrigan]

@aminerva here is an image with SLEAP and its dependencies installed. This should work on your cluster. Please follow the directions in the README to use it. https://gitlab.com/salk-tm/sleap-train