search

iceberg-project / Seals

This repo will be used for ICEBERG-related science: scripts, models, figures, and findings related to ICEBERG and its Seal detection use case.
MIT License
7 stars 3 forks source link

Memory Utilization of Tilling task #50

Closed iparask closed 5 years ago

iparask commented 5 years ago

Hello @bentocg, I just fund out that the GPU prediction task requires 80GB of memory. Is that expected? Is there something in the implementation that causes it to reach those levels?

This is the case regardless the image size. Thank you!

iparask commented 5 years ago

This apparently is my mistake. Closing until I investigate further

iparask commented 5 years ago

I'm realizing that rasterio is responsible for the large memory consumption. Please take a look at this notebook Cell 12 (Password can be provided via our slack channel).

Regardless of the image size, the memory consumption can be either high or low. I was discussing this with @AymenFJA and we think that the memory consumption is related with the image attributes when they are opened by rasterio.

@bentocg, do you agree with this understanding. Also, @mmacferrin and @AymenFJA are using GDAL for tilling. Would that work here as well?

mmacferrin commented 5 years ago

For what it’s worth, we’ve had good luck using GDAL in Python for reading image segments if you don’t need the entire image. It only helps, I think, if the TIF images are tiled (in the TIF format) to begin with, but if so it’s pretty good about using only necessary memory to read portions of the image.

This may not help, I haven’t been following this conversation closely (if it doesn’t, feel free to disregard). Just thought I’d toss that out here if it helps.

iparask commented 5 years ago

@mmacferrin this is actually helpful! Thank you!

mmacferrin commented 5 years ago

In the ASIFT (mmacferrin branch)/phase_1.../tile_writer.py file, lines 46, 48 & 80 are what you'd need to do that, copy/paste and test it out, see if it helps. (Line 48, "GetRasterBand", doesn't actually read the band data, it just creates the object and gets metadata).

https://github.com/iceberg-project/ASIFT/blob/mmacferrin/src/phase_1_image_preprocessing/tile_writer.py