search

AI4SCR / ATHENA

BSD 3-Clause "New" or "Revised" License
31 stars 6 forks source link

Memory issues when building the graph #3

Open YuanningEric opened 1 year ago

YuanningEric commented 1 year ago

Hi! I am using the software to analyze whole-slide images, and my images contained around 70K+ spots/patches. When I used the ath.graph.build_graph function to built a radius graph, the process was always "killed" because of memory issues. I am using 4 CPUs with 8GB RAM of each CPU. Is this behavior expected?

adrianomartinelli commented 1 year ago

Hi, yes this is expected as the current implementation does not exploit the spatial information from the images but regards each spot as independent sample (thus computing the pairwise distances between all pairs with sklearn functionalities). The framework was designed to work on smaller tissue patches as generated by imaging mass cytometry. These patches have up to 10k cells / spots / patches which is feasible to compute on a local machine.

As a work around, would it be possible to process your data as multiple tiles? The problematic regions would then only be the edges.

Best, Adriano

YuanningEric commented 1 year ago

Hi Adriano. Thanks for your reply! By "process your data as multiple tiles", do you mean I can try to split one image into multiple parts or regions? What do you mean by "the problematic regions would then only be the edges?"

adrianomartinelli commented 1 year ago

Yes exactly, that you process the whole slide in smaller crops.

What do you mean by "the problematic regions would then only be the edges?" I mean that if you crop the image in non-overlapping smaller images then the constructed graph is not correct at the image edges. Assume you crop your image in for parts like

A | B -- -- C | D

For example in the KNN graph, it might be possible that the true nearest neighbors are not in the current cropped image but actually in the one adjacent to it. In other words, if you have a cell close to the left border of a cropped image B, it is likely that some of the nearest neighbors are actually in the cropped image A that is on the left hand side of the left border of cropped image B.

Best, Adriano