Running pyhgtmap throws error.

[Kradenko]

Hi,

I have moved away from the original app to this branch. When I run the function below:

pyhgtmap -o contour --max-nodes-per-tile=0 -s 10 -0 --pbf warp-90.tif

on the warp-90.tif file I get the below error. I have tried every combination but I can't seem to work out what is wrong. Documentation is limited and I can't make sense of this error. Would someone be able to tell me what I am doing wrong?

user@b2a337b418dd:/data/srtm# pyhgtmap -o contour --max-nodes-per-tile=0 -s 10 -0 --pbf warp-90.tif
Traceback (most recent call last):
  File "/usr/local/bin/pyhgtmap", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/dist-packages/pyhgtmap/main.py", line 66, in main
    main_internal(sys.argv[1:])
  File "/usr/local/lib/python3.8/dist-packages/pyhgtmap/main.py", line 58, in main_internal
    HgtFilesProcessor(opts.nJobs, opts.startId, opts.startWayId, opts).process_files(
  File "/usr/local/lib/python3.8/dist-packages/pyhgtmap/hgt/processor.py", line 281, in process_files
    self.process_file(file_name, check_poly)
  File "/usr/local/lib/python3.8/dist-packages/pyhgtmap/hgt/processor.py", line 239, in process_file
    self.process_tile(file_name, tile)
  File "/usr/local/lib/python3.8/dist-packages/pyhgtmap/hgt/processor.py", line 212, in process_tile
    self.process_tile_internal(file_name, tile)
  File "/usr/local/lib/python3.8/dist-packages/pyhgtmap/hgt/processor.py", line 119, in process_tile_internal
    tile_contours = tile.get_contours(
  File "/usr/local/lib/python3.8/dist-packages/pyhgtmap/hgt/tile.py", line 207, in _get_contours
    elevations, contour_data = self.contourLines(
  File "/usr/local/lib/python3.8/dist-packages/pyhgtmap/hgt/tile.py", line 148, in contourLines
    z: numpy.typing.ArrayLike = numpy.ma.array(
  File "/usr/local/lib/python3.8/dist-packages/numpy/ma/core.py", line 6647, in array
    return MaskedArray(data, mask=mask, dtype=dtype, copy=copy,
  File "/usr/local/lib/python3.8/dist-packages/numpy/ma/core.py", line 2894, in __new__
    mask = np.resize(mask, _data.shape)
  File "<__array_function__ internals>", line 200, in resize
  File "/usr/local/lib/python3.8/dist-packages/numpy/core/fromnumeric.py", line 1495, in resize
    a = concatenate((a,) * repeats)[:new_size]
  File "<__array_function__ internals>", line 200, in concatenate
SystemError: <built-in function concatenate> returned NULL without setting an error

Kind Regards, K

[agrenott]

Hi, Could you please share the file causing you issue?

[Kradenko]

Hi agrenott,

Thank you for the quick reply, was out of town shortly after I logged this ticket. I have been trying to resolve, or at least understand what I am doing right and wrong.

I have the below script that I am running:

mkdir -p /data/srtm

cd /mnt/srtm

for zipfile in *.zip; do unzip -j -o "$zipfile" -d /data/srtm; done

echo "Fill all the Voids"
cd /data/srtm
for hgtfile in *.hgt; do gdal_fillnodata.py $hgtfile $hgtfile.tif; done
echo "Fill all the voids complete!"

echo "Merge all tifs"
gdal_merge.py -n 32767 -co BIGTIFF=YES -co TILED=YES -co COMPRESS=LZW -co PREDICTOR=2 -o /data/srtm/raw.tif *.hgt.tif
echo "Merge all tifs complete"

echo "Convert the raw file into Mercator projection, interpolate and shrink"
cd /data/srtm
gdalwarp -co BIGTIFF=YES -co TILED=YES -co COMPRESS=LZW -co PREDICTOR=2 -t_srs "+proj=merc +ellps=sphere +R=6378137 +a=6378137 +units=m" -r bilinear -tr 1000 1000 raw.tif warp-1000.tif
gdalwarp -co BIGTIFF=YES -co TILED=YES -co COMPRESS=LZW -co PREDICTOR=2 -t_srs "+proj=merc +ellps=sphere +R=6378137 +a=6378137 +units=m" -r bilinear -tr 5000 5000 raw.tif warp-5000.tif
gdalwarp -co BIGTIFF=YES -co TILED=YES -co COMPRESS=LZW -co PREDICTOR=2 -t_srs "+proj=merc +ellps=sphere +R=6378137 +a=6378137 +units=m" -r bilinear -tr 500 500 raw.tif warp-500.tif
# 60m - just for contour lines
gdalwarp -co BIGTIFF=YES -co TILED=YES -co COMPRESS=LZW -co PREDICTOR=2 -t_srs "+proj=merc +ellps=sphere +R=6378137 +a=6378137 +units=m" -r bilinear -tr 700 700 raw.tif warp-700.tif
# 30m - for the detail hillshade
gdalwarp -co BIGTIFF=YES -co TILED=YES -co COMPRESS=LZW -co PREDICTOR=2 -t_srs "+proj=merc +ellps=sphere +R=6378137 +a=6378137 +units=m" -r bilinear -tr 90 90 raw.tif warp-90.tif
echo "Convert the raw file into Mercator projection, interpolate and shrink complete!"

echo "Create color relief for different zoom level"
# relief for zoom factors 1-4
gdaldem color-relief -co COMPRESS=LZW -co PREDICTOR=2 -alpha warp-5000.tif /data/OpenTopoMap/mapnik/relief_color_text_file.txt relief-5000.tif
# relief for zoom factors 5-8
gdaldem color-relief -co COMPRESS=LZW -co PREDICTOR=2 -alpha warp-500.tif /data/OpenTopoMap/mapnik/relief_color_text_file.txt relief-500.tif
echo "Create color relief for different zoom level complete"

# create hillshade
echo "Create hillshade for different zoom levels"
gdaldem hillshade -z 7 -compute_edges -co COMPRESS=LZW warp-5000.tif hillshade-5000.tif
gdaldem hillshade -z 7 -compute_edges -co BIGTIFF=YES -co TILED=YES -co COMPRESS=LZW warp-1000.tif hillshade-1000.tif
gdaldem hillshade -z 4 -compute_edges -co BIGTIFF=YES -co TILED=YES -co COMPRESS=LZW warp-700.tif hillshade-700.tif
gdaldem hillshade -z 2 -co compress=lzw -co predictor=2 -co bigtiff=yes -compute_edges warp-90.tif hillshade-90.tif && gdal_translate -co compress=LZW -co bigtiff=yes -co tiled=yes hillshade-90.tif hillshade-90-jpeg.tif
echo "Create hillshade for different zoom levels complete"

echo "Create contour lines"
pyhgtmap -o contour --max-nodes-per-tile=0 -0 --pbf -l warp-90.tif
echo "Create contour lines complete"

mv *.osm.pbf contours.pbf

Here is the download link to the tiff file (https://www.aonline.co.za/warp-90.tif) It doesn't give that error in this case, just responds with "Killed". Any idea why?

I have based the above on this tutorial: https://github.com/der-stefan/OpenTopoMap/blob/master/mapnik/README.md I suspect I am doing unnecessary commands to achieve my goal of generating the contour lines pbf file.

Any advice would be greatly appreciated! K.

[agrenott]

Hi, Thanks for the file, I've been able to reproduce your latest "Killed" issue:

$ pyhgtmap -o contour --max-nodes-per-tile=0 -s 10 -0 --pbf -l DEBUG /mnt/f/DOWNLOAD/warp-90.tif
$ dmesg | tail
...
[  228.112091] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=pyhgtmap,pid=4428,uid=1000
[  228.112253] Out of memory: Killed process 4428 (pyhgtmap) total-vm:23444412kB, anon-rss:13312964kB, file-rss:8kB, shmem-rss:0kB, UID:1000 pgtables:31284kB oom_score_adj:0

At least in my case, this is because the process is consuming too much memory. Probably because your input file is quite big (323MB).

2024-10-10 18:31:06,985 - pyhgtmap.hgt.file:439:INFO - 9567 - tif file /mnt/f/DOWNLOAD/warp-90.tif: 33396 x 27498 points, bbox: (10.99999, -35.00034, 37.99930, -14.99997)

You may try to split it into smaller tiles.

[Kradenko]

Just saw that now. Damn memory :P What I do wonder, could the compressing of the raw.tif down to the warp-90.tif file cause the error?

I have another machine here with 384 GB of RAM I am going to test it on now. Will report back. Again, thank you for the awesome quick responses. Appreciate it!

K.