search

geodesign / django-raster

Django-raster allows you to create tiled map services (TMS) and raster map algebra end points for web maps. It is Python-based, and requires GeoDjango with a PostGIS backend.
BSD 3-Clause "New" or "Revised" License
96 stars 39 forks source link

RasterTile.objects.bulk_create(batch) in parser.py result in Out of Memory (when processing big Raster file) #61

Closed justRishi closed 2 years ago

justRishi commented 3 years ago

Problem when adding raster layer, when processing a Large raster file(2.4GB) with max zoom level 14 the creation of tiles runs into an error at level 13 with:

psycopg2.errors.ProgramLimitExceeded: out of memory 
DETAIL: Cannot enlarge string buffer containing 0 bytes by 1233179600 more bytes.   
....   
File "/opt/venv/lib/python3.8/site-packages/raster/tiles/parser.py", line 412, in process_quadrant   
RasterTile.objects.bulk_create(batch)   
File "/opt/venv/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method

(see complete stack at the end)

So code where it goes wrong is here at line 412: /raster/tiles/parser.py

 # Commit batch to database and reset it
                if len(batch) == self.batch_step_size:
                    RasterTile.objects.bulk_create(batch)
                    batch = []
        # Commit remaining objects
        if len(batch):
            RasterTile.objects.bulk_create(batch)

last line is 412. Batch string is to long, to many objects.

Complete error log:

[2021-06-15 12:07:39] Creating 968 tiles in 1 quadrants at zoom 14.
[2021-06-15 12:07:39] Starting tile creation for quadrant 1 at zoom level 14
[2021-06-15 12:07:39] Traceback (most recent call last):
File "/opt/venv/lib/python3.8/site-packages/raster/tasks.py", line 39, in create_tiles
parser.create_tiles(zoom)
File "/opt/venv/lib/python3.8/site-packages/raster/tiles/parser.py", line 287, in create_tiles
self.populate_tile_level(zoom_levels)
File "/opt/venv/lib/python3.8/site-packages/raster/tiles/parser.py", line 313, in populate_tile_level
self.process_quadrant(indexrange, zoom)
File "/opt/venv/lib/python3.8/site-packages/raster/tiles/parser.py", line 347, in process_quadrant
snapped_dataset = self.dataset.warp({
File "/opt/venv/lib/python3.8/site-packages/django/contrib/gis/gdal/raster/source.py", line 410, in warp
target = GDALRaster(ds_input, write=True)
File "/opt/venv/lib/python3.8/site-packages/django/contrib/gis/gdal/raster/source.py", line 143, in __init__
self._ptr = capi.create_ds(
File "/opt/venv/lib/python3.8/site-packages/django/contrib/gis/gdal/prototypes/errcheck.py", line 127, in check_pointer
raise GDALException('Invalid pointer returned from "%s"' % func.__name__)
django.contrib.gis.gdal.error.GDALException: Invalid pointer returned from "GDALCreate"

[2021-06-15 12:07:51] Traceback (most recent call last):
File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.ProgramLimitExceeded: out of memory
DETAIL: Cannot enlarge string buffer containing 0 bytes by 1233179600 more bytes.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/venv/lib/python3.8/site-packages/raster/tasks.py", line 39, in create_tiles
parser.create_tiles(zoom)
File "/opt/venv/lib/python3.8/site-packages/raster/tiles/parser.py", line 287, in create_tiles
self.populate_tile_level(zoom_levels)
File "/opt/venv/lib/python3.8/site-packages/raster/tiles/parser.py", line 313, in populate_tile_level
self.process_quadrant(indexrange, zoom)
File "/opt/venv/lib/python3.8/site-packages/raster/tiles/parser.py", line 412, in process_quadrant
RasterTile.objects.bulk_create(batch)
File "/opt/venv/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/opt/venv/lib/python3.8/site-packages/django/db/models/query.py", line 514, in bulk_create
returned_columns = self._batched_insert(
File "/opt/venv/lib/python3.8/site-packages/django/db/models/query.py", line 1287, in _batched_insert
inserted_rows.extend(self._insert(
File "/opt/venv/lib/python3.8/site-packages/django/db/models/query.py", line 1270, in _insert
return query.get_compiler(using=using).execute_sql(returning_fields)
File "/opt/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
cursor.execute(sql, params)
File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/opt/venv/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.OperationalError: out of memory
DETAIL: Cannot enlarge string buffer containing 0 bytes by 1233179600 more bytes.
justRishi commented 3 years ago

Will I deleted my comment, without having to change the code , RASTER_BATCH_STEP_SIZE can be set to a value in Django settings file, will make the batch that is written to db bulk_create in tiles\parser.py smaller.