Bug - keypoint_moseq kernel doesnt work in jupyter notebook if launched from the same environment

[stowerslab]

So not sure if im doing anything wrong but i have run into a strange bug. I have done the step of registering the keypoint moseq kernel. If I keypoint moseq in a jupyter notebook from its own environment though with that kernel selected as the backend it throws me this error

XlaRuntimeError: UNKNOWN: no kernel image is available for execution on the device in external/org_tensorflow/tensorflow/compiler/xla/stream_executor/cuda/cuda_asm_compiler.cc(64): 'status'

However if i launch it from another environment it goes through with no issues

[calebweinreb]

Interesting... can you provide the OS you were using and which installation method you used? Also which command triggered the error?

[stowerslab]

Win 10 pro, and installed using cond0a.

Not sure if relevant, but because our institute blocks the anaconda repo, we had to remove the "default" repo and use conda-forge but no errors during installation.

Tried it again and problem still persists with the newest version moseq 0.0.5. The command that threw this error was "import keypoint_moseq as kpms"

Other than the error in the jupyter notebook above, there is also an error in the terminal

2023-05-02 13:02:17.325253: E external/org_tensorflow/tensorflow/compiler/xla/stream_executor/cuda/cuda_asm_compiler.cc:61] cuLinkAddData fails. This is usually caused by stale driver version.

2023-05-02 13:02:17.329033: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/gpu_compiler.cc:1369] The CUDA linking API did not work. Please use XLA_FLAGS=--xla_gpu_force_compilation_parallelism=1 to bypass it, but expect to get longer compilation time due to the lack of multi-threading.

These 2 errors do not appear if i launch keypoint moseq in a jupyter notebook in different environment, but with the keypoint moseq kernel selected

[calebweinreb]

Hmm weird. I wasn't able to replicate this on our own Windows computer and also it seems like you have a workaround so I'm going to close this for now. But thanks for the all the info! It's helpful in case anyone else runs into this too.

[kanhabatra]

Hi, I just ran into the same error on linux! I haven't tried the workaround yet but was wondering if you found a fix for this?

[calebweinreb]

Hmm interesting! I haven't found a fix. In case it helps, the workaround is to run the following from within the keypoint_moseq env

python -m ipykernel install --user --name=keypoint_moseq

launch jupyter from a separate env (such as base), and then select the keypoint_moseq kernel

[kanhabatra]

I figured it out. The cuda related packages that are needed for moseq require some specific cuda drivers to be installed which I didn't earlier.

Thank you a lot anyway!