Memory leak after saving raster data.

[davidvardan]

Hi,

I am using django-raster to save raster data into postgis database. Here is my code

   resistanceTile = rm.RasterLayer(id=resistanceMap[0].id, name="Resistance", rasterfile=tmp_path, datatype='co')
   resistanceTile.save()

where 'tmp_path' is a path to tiff file with 2960x2580 resolution and the maximum zoom level is 15 by default. Before saving tile, app is using 110MB of memory,but after calling resistanceTile.save() the memory increase to 1.7GB and not free.

Regards, David Vardanyan

[yellowcap]

@davidvardan not sure why this is happening. What version of Django and django-raster are you using? Do you have Celery configured?

[davidvardan]

Django-raster== 0.5 and Django==2.0.4. I am not using Celery.

[yellowcap]

Python does not always free the memory immediately. Can you run the following and check the if the memory was freed afterwards?

import gc
gc.collect()

[davidvardan]

I tried this before write you, but it didn't help me. Maybe problem will be solve with celery?

[davidvardan]

Hi @yellowcap,

I found the source of my problem. Memory leak cause when we used debug=true in django settings, because django save every SQL statement in memory and don't free it if debug=true.

[yellowcap]

Glad you found the origin of the problem @davidvardan ! Hope django-raster is useful for you!

[davidvardan]

And also I have a question. Does django-raster have method to parse raster data without using file, directly from band data?

[yellowcap]

Not directly, but you can use the RasterLayerParser class "manually" to do what you want.

You would need to create an in-memory raster from your band data, and then assign that to a rasterlayer parser instance.

Here is an example of how this could look like:

import numpy
from django.contrib.gis.gdal import GDALRaster
from raster.tiles.const import WEB_MERCATOR_SRID
from raster.tiles.parser import RasterLayerParser

# Identify what rasterlayer will hold the data.
rasterlayer_id = 1

# Instantiate parser for this rasterlayer id.
parser = RasterLayerParser(rasterlayer_id)

# Create in-memory raster from array (you need to know the pixel size, srid and origin of the pixel data).
data = numpy.random.randint(0, 256, size=(100, 100)).astype('uint8')
rst = GDALRaster({
    'width': 100,
    'height': 100,
    'datatype': 1,
    'srid': 4326,
    'origin': (2.345, -3.234),
    'scale': (0.001, -0.002),
    'bands': [
        {'data': data, 'nodata_value': 0},
    ]
})

# Reproject to web mercator (the raster needs to be in 3857)
rst_3857 = rst.transform(WEB_MERCATOR_SRID)

# Assign the raster as dataset.
parser.dataset = rst_3857

# Extract metadata and store in rasterlayer.
parser.extract_metadata()

# Create tiles for a list of zoom levels.
zoom = 15
parser.create_tiles(list(range(zoom + 1)))

[davidvardan]

Thank you very much, this is what I need.