Integration of live classification with RELION or cryosparc

[genisvalentin]

Dear WARP developers,

In our facility, we have modified WARP v1 so that it can automatically submit 2D/3D classification jobs to cryoSPARC during live pre-processing. We found that this is very useful in assessing grid/sample quality.

While we would not like to miss out on the newest WARP developments, we would also like to keep using this feature.

Now that there is a new open source code repository, would you consider integrating this feature? Would you accept code contributions from us to this end?

Best regards

Genis

[alisterburt]

@genisvalentin cool! Could you explain what the integration and associated configuration looks like?

My gut feeling is that we are probably not interested in maintaining this feature. This is because cryoSPARC may change and an integration like this would force us to stay in sync with it.

I'll defer to @dtegunov for the final say though 🙂

[genisvalentin]

Hi @alisterburt!

One can communicate with a CryoSPARC master instance using their web API, so one can send an http request with a JSON string that allows you to do several things, such as build and queue a job, or query the status of a job (e.g. completed or failed). This is greatly inspired by the scipion cryosparc plugin.

In our live processing implementation, we have a dispatcher timer that runs every few seconds to keep track of running cryoSPARC jobs and queues new classification jobs when required. The user can choose to have a new run every x minutes or every x particles.

This is how it looks in the GUI.

In the main window, there is a link that brings up the latest classification job in a web browser.

We are doing this on Windows, so one needs to have a network file system where the cryoSPARC linux workstation can access the particle stacks. Here, we also have a sftp-based file picker so that one can browse the cryoSPARC workstation filesystem and locate the WARP project directory. Hence the ssh username and key.

To use the web API, the cryosparc license key must be in the http request header. The license key is a new string in the global.settings file.

I must admit that we have already encountered two (small but breaking) changes in the web API, one with the introduction of CryoSPARC 3 and another one in a 4.x release, so indeed it might be a problem to stay up to date.

Looking forward to hearing your thoughts!

Genis

[alisterburt]

@genisvalentin this is really, really awesome 🙂 thank for taking the time to explain! I still have my reservations but that doesn't take anything away from the cool thing you've built

[dtegunov]

Hey @genisvalentin, this is amazing! I've never seen a UI mod for Warp before. Good color choice for the icon 😉

However, I think this falls outside of Warp's operation domain. Ideally, we'd provide REST endpoints that allow an external application to orchestrate the co-existence of Warp and CS rather than interact with CS directly from Warp. Can you think of endpoints that would enable this?

I'll loop in @cdienem who wrote a script for triggering CS jobs based on Warp's outputs a long time ago.