A Few Practicality Questions with Integrating Suite2P

[jmdelahanty]

Hello MouseLand!

I'll be starting to analyze some 2P data from the Tye Lab here at Salk and I was wondering about how I might best integrate Suite2P with some infrastructure I've been trying to make.

I've been working on using a repository from the Deisseroth Lab for containerizing conversions of Bruker's RAW file format into tiffs with Bruker's Image Ripping Utility and then appending those tiffs into a base NWB file I create at the end of a recording session. Our recordings aren't too big, but they generate around 20-25k images and managing that many tiffs isn't fun or efficient. I could use tiff stacks, but Bruker's image ripper doesn't produce them and my goal is that all my pipeline has to handle is one NWB file which is easier to track and manipulate.

My next goal is to have another container that does the preprocessing steps (motion correction, cell registration, etc) upon that NWB file using Suite2P's awesome algorithms and append those processed objects to the file as well until it's ready for someone on their local machine to touch up/investigate using the GUI. So the questions:

• Is there a way to read from an NWB file and then append results to that same file?
• Can I containerize these processes and, if so, do you have any good examples or resources I can look at for doing so?
• I'd like to have the configurations used for processing the files be part of the NWB file's metadata for completeness. Is there a way to provide some sort of configuration for all the parameters before processing? Does it even make sense to do that?
• I also wanted to confirm that Suite2P doesn't use any GPU resources. I think our campus' Docker containers are meant for people running things that use GPUs preferentially over CPU run pipelines so I need to ask our IT about that. Am I correct that it's using Numba JIT for CPU acceleration but not with GPU acceleration?

Thank you for your time and I'm so excited to use your platform! Thank you also for building it!

[carsen-stringer]

Sounds great!

1. we can now read in NWB files for processing (use ops['input_format']='nwb' and use latest, pip install git+https://www.github.com/mouseland/suite2p.git) but it is not well-tested. the results are not appended to that file by default, but it could be added as an option. see nwb.py with all the functionality we currently have. we are happy to accept pull requests expanding it.
2. I have never made a container for suite2p but all the packages are now installable via pip so I don't see any reason why containerization would be difficult.
3. you could save it as "ops_original" so they are not overwritten by the processing, but there are very few parameters we add to the ops.npy file so you could also use the end ops.
4. Yes, no GPU resources. we are using pytorch now for FFTs in registration but without GPU, and we are also using numba to accelerate registration and the multiplication for fluorescence extraction.

[jmdelahanty]

Thank you for your response! I'll be trying to use it this week and I'll do my best to use what Chris has made for the Deisseroth lab. I'll update this thread with anything I run into.

[jmdelahanty]

Hello again @carsen-stringer! I have no idea what this entails or if it's something a newbie like me could do, but is doing things with a GPU a good idea or possible? Is this something I could somehow try and learn how to do and implement? It just sounds really neat and like a great opportunity to see how code is different when using tools like Numba for CPU acceleration vs GPU acceleration for your algorithms. Also for understanding how things work under the hood in general.

[carsen-stringer]

now that we are using pytorch, the GPU could be utilized easily, but the overhead of copying data to the GPU is often too large if you aren't doing many operations on the same chunk of data

[jmdelahanty]

Interesting! So just because you can use a GPU doesn't mean you should! I didn't know that, I basically thought it was always faster to throw a GPU at stuff if you can. Thank you for the tip!