[BUG] NBLAST searches fail using Neuron Similarity Widget

[mwinding]

• CATMAID version: 2021.12.21-62-gfe0a57ad9
• CATMAID instance: https://neurophyla.mrc-lmb.cam.ac.uk/catmaid/drosophila/l1/seymour/
• Browser: Chrome, Version 98.0.4758.80 (Official Build) (x86_64)
• Operating system: macOS, Big Sur 11.2.3

Short description

NBLAST searches fail without verbose error message in the Neuron Similarity Widget. After computing, the status is reported as "error" and no results are present. This issue seems to have started between 2022-01-13 and 2022-01-20 based on the log.

To reproduce

Add a skeleton for NBLAST search, use default parameters with Config = "Brain R->L (2)" selected.

[HEyinhui]

the same question!

[clbarnes]

I believe this coincides with the update to the latest release. @tomka , were any changes made to this functionality shortly before the release?

[tomka]

Looking at the error message (hover mouse over error status text in table), this might have to do with a recent Django upgrade and the way async code is called by django-channels. @clbarnes when you updated the server the last time, did you also upgrade the async dependency file (pip install -r requirements-async.txt)? Also, I remember running into some issues with RabbitMQ and switched at least one server to Redis, which seems to interact better with django-channels. I will check on this setup and another setup that sill uses RabbitMQ to see if I made some changes that didn't make it into the upgrade docs.

[clbarnes]

Those seem to be installed, but when the fibsem instance was set up it was very low-traffic and so did not have websockets configured for async, as that was optional at the time - if that's now non-optional for cross-instance stuff, I can set it up, but it will require a bit of downtime.

[tomka]

WebSockets are still optional, I just thought they were set up, because the error suggested the back-end was trying to use them using channels. Is CHANNEL_LAYERS set to anything in settings.py?

[tomka]

While this isn't solved yet, I finally can reproduce this issue reliably, making debugging easier.

[tomka]

Thanks for sending your config files! They show that you have CHANNEL_LAYERS defined in settings.py, even though you don't have ASGI/Daphne configured. With a setup with comparable settings to yours, I can get it to work when I comment the CHANNEL_LAYERS layers setting. Since you don't need WebSockets, it would be interesting to see if it works without this setting.

Also, I noticed that I had the line numprocs = 1 to the supervisor config for Celery for the latest stable and dev versions. Otherwise I would get threading errors (we upgraded the Celery dependency not too long ago). The docs have already been updated.

[clbarnes]

Sorry, which instance is that on? On seymour (0111-8) I thought we did have daphne and celery set up and configured (it's certainly in the supervisor config); it's fibsem/lsm where we don't, which I think is not where Michael's seeing this problem. Fibsem doesn't have CHANNEL_LAYERS.

[tomka]

I have seen this on seymour. I have it now working with CHANNELS_LAYERS enabled as well, using a patch I just pushed to master and dev.

[tomka]

While there was no error with the previous patch, and the NBLAST computation was performed as well, the (optional) use of WebSockets was not working properly yet: messages (like "NBLAST computation complete") were not sent to clients, because the async coroutine sending the message wasn't actually run (not awaited). I have updated the way messages are sent from Celery tasks and now this part should work again as well. All main branches are updated.

[clbarnes]

All neurophyla instances are updated to dev.

[tomka]

Thanks @clbarnes! I just ran a test NBLAST comparison and it worked fine, including WebSockets update messages. @mwinding ans @HEyinhui, maybe you can try as well and run/rerun some queries that failed for you before.

[HEyinhui]

Whatever I computed, the score is 0 or very low, why?

[clbarnes]

You have the "only positive scores" box checked, which might explain the zeros - depending on other factors, you could easily be getting negative scores, which don't have any special meaning and can disrupt normalisation.

Are these skeletons being transformed at all? If not, the scores should be very low because the morphologies aren't spatially/ directionally similar. If they are transformed and look like they should be very similar in the 3D viewer, it may be a problem.

[HEyinhui]

I transformed the skeleton, and it worked. Thanks so much.