search

michaellee1 / CellSeg

MIT License
21 stars 25 forks source link

compute_stats() fails with large number of cells #38

Open qu4drupole opened 2 years ago

qu4drupole commented 2 years ago

I have processed CODEX data (that I assume came from the new PhenoCycler-Fusion system) saved as a single qptiff file. This format isn't quite what CellSeg expects for codex. With some small tweaks to the CellSeg code I was able to successfully complete the workflow on a cropped section (1000x1000) of the images. I used the 'CellSeg_stepbystep' notebook. The test image had an estimated 2.7k cells, and the overlays and statistics looked good. I reran everything on the entire image (~200k cells) and my kernel crashes at walkthrough_utils.compute_stats. The Jupyter log isn't very helpful:

[I 2022-05-20 14:44:19.756 ServerApp] Saving file at /CellSeg_stepbystep.ipynb [W 2022-05-20 14:44:19.757 ServerApp] Notebook CellSeg_stepbystep.ipynb is not trusted [I 2022-05-20 15:05:30.095 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (1/5), keep random ports kernel 815a70eb-3373-413b-84f1-3fa4c437534a restarted kernel 815a70eb-3373-413b-84f1-3fa4c437534a restarted [I 2022-05-20 15:05:30.168 ServerApp] Starting buffering for 815a70eb-3373-413b-84f1-3fa4c437534a:b07fe2b1-dfed-4a1f-a482-1c7f08a65774

I tried using more memory (500G). No avail. Any idea on what's causing the problem? What information would be helpful for troubleshooting?

qu4drupole commented 2 years ago

It's working now. It was (partly) a memory error. With 1 TB, I could get through to lstsq() step, but would have taken forever to complete with 200k cells and 44 markers. I made some edits to the output section and cvmask class to crop up the image into smaller chunks for quantification. Now it just takes <10min. If you think this may be a recurring situation for users, I can submit a pull request.

charles-xue commented 10 months ago

Hello, I have recently been trying to segment cells in CODEX's qptiff files and found this software, but I have some problems when using it. Can you provide me with some help?