jian-shu-lab / Bering

Other
14 stars 1 forks source link

Managing VRAM #4

Open pakiessling opened 1 year ago

pakiessling commented 1 year ago

Hi,

I was now able to apply Bering to my data in image-free mode.

Next, I would like to try with the addition of nuclei and membrane stain.

Unfortunately, my GPU has only 16 GB VRAM, not enough to accomondate the whole slide images, even on a modest measurement.

What do you recommend? Would it be advisable to split to only train on a subset of the slide / images, or split it into batches somehow?

KANG-BIOINFO commented 1 year ago

Hi pakiessling,

Have you solved this issue?

Thank you for your careful observation. Image-free mode is much less memory-intensive compared with the mode with image input, due to the number of images and image sizes if we want to train binary edge prediction for tens of thousands of edges in the training phase.

When we trained the model, our GPU size was not too much larger than your GPU. We tried reducing the number of edges during training (e.g. 5000), which can be set here. We also tried limiting the number of training samples when we used image-mode.

Besides, in the prediction stage, we sometimes reduce num_edges_perSpot in cell_segmentation function to reduce the total number of edges in the prediction.

Your suggestion sounds reasonable. Currently, we use all images in selected edges during training. We will consider using a subset of images for selected edges in training in the future as a potential way to reduce VRAM consumption. Hope these are helpful and thank you again for your feedback.