search

nadeemlab / SPT

Spatial profiling toolbox for spatial characterization of tumor immune microenvironment in multiplex images
https://oncopathtk.org
Other
21 stars 2 forks source link

Memory limits during application initialization #219

Closed jimmymathews closed 1 year ago

jimmymathews commented 1 year ago

Currently there are application initialization steps (indexing, cache creation) which are still unnecessarily resource-intensive and cannot complete correctly in some environments. For this issue, improve the centroid cache creation and the expression matrix cache creation to do per-sample serialization-to-file during processing, in order to vastly decrease RAM usage requirements.

jimmymathews commented 1 year ago

As of 0536244d3938bc2f669bf06278a90646ddfc0ae6 (branch issue219) , the per-specimen during-processing serialization is enabled in spt ondemand cache-expressions-data-array (for the feature matrix part, not the centroids/positional info).

jimmymathews commented 1 year ago

As of 4364ea99a234e2fe8a9e3da63a2887e238585e83 (branch issue219), enabled for the centroids as well.

jimmymathews commented 1 year ago

I think the memory adjustment has solved this issue. The cache creation script completes and proceeds.

However, it seems there is a problem with the volume mounting. Files created by one container (counts) are only sometimes available from other containers or at different times.

jimmymathews commented 1 year ago

This leads to the problem that the other containers (proximity, squidpy) never start up, because they are waiting to see the cache files created.

jimmymathews commented 1 year ago

This latter problem is really related to spt-dashboard #63

The primary issue described here is resolved, I think.