search

tilemill-project / tilemill

TileMill is a modern map design studio
https://tilemill-project.github.io/tilemill/
BSD 3-Clause "New" or "Revised" License
3.11k stars 527 forks source link

Coredump during Tilemill export #2402

Open nickponline opened 10 years ago

nickponline commented 10 years ago

I'm getting a core dump exporting from Tilemill using the command line. I'm trying to do an export of a map consisting of 179 layers (each a tif). Total of 11GB of tifs. Each tif look fine visually and all data inside seems to be correct. 

$ gdalinfo imagelayer95.tif
Driver: GTiff/GeoTIFF
Files: imagelayer95.tif
Size is 4087, 4105
Coordinate System is:
PROJCS["WGS 84 / Pseudo-Mercator",
    GEOGCS["WGS 84",
        DATUM["WGS_1984",
            SPHEROID["WGS 84",6378137,298.257223563,
                AUTHORITY["EPSG","7030"]],
            AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0],
        UNIT["degree",0.0174532925199433],
        AUTHORITY["EPSG","4326"]],
    PROJECTION["Mercator_1SP"],
    PARAMETER["central_meridian",0],
    PARAMETER["scale_factor",1],
    PARAMETER["false_easting",0],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    EXTENSION["PROJ4","+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext  +no_defs"],
    AUTHORITY["EPSG","3857"]]
Origin = (-13639890.148661922663450,4579938.928738852031529)
Pixel Size = (0.022403255814027,-0.022403255814027)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (-13639890.149, 4579938.929) (122d31'45.18"W, 38d 0'13.08"N)
Lower Left  (-13639890.149, 4579846.963) (122d31'45.18"W, 38d 0'10.73"N)
Upper Right (-13639798.587, 4579938.929) (122d31'42.22"W, 38d 0'13.08"N)
Lower Right (-13639798.587, 4579846.963) (122d31'42.22"W, 38d 0'10.73"N)
Center      (-13639844.368, 4579892.946) (122d31'43.70"W, 38d 0'11.90"N)
Band 1 Block=4087x1 Type=Byte, ColorInterp=Red
  Mask Flags: PER_DATASET ALPHA 
Band 2 Block=4087x1 Type=Byte, ColorInterp=Green
  Mask Flags: PER_DATASET ALPHA 
Band 3 Block=4087x1 Type=Byte, ColorInterp=Blue
  Mask Flags: PER_DATASET ALPHA 
Band 4 Block=4087x1 Type=Byte, ColorInterp=Alpha

Output of Tilemill is:

[millstone] linking '/home/layers/imagelayer0.tif' -> '/mnt/data/ortho-6-6.tif'
[millstone] linking '/home/layers/imagelayer1.tif' -> '/mnt/data/ortho-10-2.tif'
... SNIP (ones line for each image)
[millstone] linking '/home/layers/imagelayer167.tif' -> '/mnt/data/ortho-10-14.tif'
[millstone] linking '/home/layers/imagelayer169.tif' -> '/mnt/data/ortho-4-9.tif'
[millstone] finished processing '/home/ortho'

done.
Rendering export
Creating new job /mnt/data/ortho/ortho.export
Persisting state in /mnt/data/ortho/ortho.export every minute
Writing errors to /mnt/data/ortho/ortho.export-failed
ERROR 4: `/home/layers/imagelayer86.tif' not recognised as a supported file format.
ERROR 4: `/home/layers/imagelayer95.tif' not recognised as a supported file format.
17/20922/50554: `/home/layers/imagelayer95.tif' not recognised as a supported file format.
encountered during parsing of layer 'imagelayer95' in Layer
ERROR 4: `/home/layers/imagelayer10.tif' not recognised as a supported file format.
17/20922/50555: `/home/layers/imagelayer95.tif' not recognised as a supported file format.
encountered during parsing of layer 'imagelayer95' in Layer
17/20923/50554: `/home/layers/imagelayer95.tif' not recognised as a supported file format.
encountered during parsing of layer 'imagelayer95' in Layer
7/20923/50555: `/home/layers/imagelayer95.tif' not recognised as a supported file format.

  encountered during parsing of layer 'imagelayer95' in Layer

[3s] Part(1/1) 0.0118% 4/33.9k @ 1/s | 7h 29m 44s left | ✓ 0 ■ 0 □ 0 fail 4render: this map appears to be in use by 1 other thread(s) which is not allowed. You need to use a map pool to avoid sharing map objects between concurrent rendering

render: this map appears to be in use by 1 other thread(s) which is not allowed. You need to use a map pool to avoid sharing map objects between concurrent rendering
... SNIP (similar lines) 
render: this map appears to be in use by 1 other thread(s) which is not allowed. You need to use a map pool to avoid sharing map objects between concurrent rendering

16/10462/25278: `/home/layers/imagelayer86.tif' not recognised as a supported file format.
16/10462/25279: `/home/layers/imagelayer86.tif' not recognised as a supported file format.
16/10463/25278: `/home/layers/imagelayer86.tif' not recognised as a supported file format.

... SNIP (similar lines) 

ERROR 4: `/home/layers/imagelayer2.tif' not recognised as a supported file format.
ERROR 4: `/home/layers/imagelayer17.tif' not recognised as a supported file format.
ERROR 4: `/home/layers/imagelayer38.tif' not recognised as a supported file format.
ERROR 4: `/home/layers/imagelayer28.tif' not recognised as a supported file format.

... SNIP (similar lines) 

17/20926/50556: `/home/layers/imagelayer10.tif' not recognised as a supported file format.
17/20926/50557: `/home/layers/imagelayer10.tif' not recognised as a supported file format.
17/20927/50556: `/home/layers/imagelayer10.tif' not recognised as a supported file format.
17/20927/50557: `/home/layers/imagelayer10.tif' not recognised as a supported file format.

... SNIP (similar lines) 

ERROR 4: `/home/layers/imagelayer74.tif' not recognised as a supported file format.

18/41851/101111: `/home/layers/imagelayer21.tif' not recognised as a supported file format.

Export process died: Error: SQLITE_CANTOPEN: unable to open database file
Exiting process [tm-ortho.mbtiles]
*** Error in `tm-ortho.mbtiles': free(): corrupted unsorted chunks: 0x00007f38442ce540 ***
Aborted (core dumped)

The only difference on our end is that this export job is larger than previous ones. Nothing else in our process has changed. Any help would be greatly appreciated!

The only difference on our end is that this export job is larger than previous ones. Nothing else in our process has changed. Any help would be greatly appreciated!

Running Tilemill e740479cede5c0286fd0c15cf127758ceb07735f
Node v0.10.29
springmeyer commented 10 years ago

Can you share a backtrace (same as we did in #2338) please?

nickponline commented 10 years ago

Sure, doing it now.

nickponline commented 10 years ago

Running exactly the same export and it has progressed past the point that it core dumped last run, although still giving the same errors:

19/83698/202218: `/home/layers/imagelayer89.tif' not recognised as a supported file format.
19/83698/202219: `/home/layers/imagelayer89.tif' not recognised as a supported file format.
19/83699/202218: `/home/layers/imagelayer89.tif' not recognised as a supported file format.
19/83699/202219: `/home/layers/imagelayer89.tif' not recognised as a supported file format.
[3s] Part(1/1) 0.4245% 144/33.9k @ 39/s | 14m 35s left | ✓ 2 ■ 0 □ 30 fail 112
ERROR 4: `/home/layers/imagelayer144.tif' not recognised as a supported file format.
19/83700/202218: `/home/layers/imagelayer144.tif' not recognised as a supported file format.
19/83700/202219: `/home/layers/imagelayer144.tif' not recognised as a supported file format.
19/83701/202218: `/home/layers/imagelayer144.tif' not recognised as a supported file format.
19/83701/202219: `/home/layers/imagelayer144.tif' not recognised as a supported file format.
[5m 55s] Part(1/1) 6.5820% 2.2k/33.9k @ 13/s | 1h 24m 4s left | ✓ 1.1k ■ 0 □ 1.0k fail 117render: this map appears to be in use by 1 other thread(s) which is not allowed. You need to use a map pool to avoid sharing map objects between concurrent rendering
render: this map appears to be in use by 1 other thread(s) which is not allowed. You need to use a map pool to avoid sharing map objects between concurrent rendering
render: this map appears to be in use by 1 other thread(s) which is not allowed. You need to use a map pool to avoid sharing map objects between concurrent rendering
render: this map appears to be in use by 1 other thread(s) which is not allowed. You need to use a map pool to avoid sharing map objects between concurrent rendering
render: this map appears to be in use by 1 other thread(s) which is not allowed. You need to use a map pool to avoid sharing map objects between concurrent rendering
render: this map appears to be in use by 1 other thread(s) which is not allowed. You need to use a map pool to avoid sharing map objects between concurrent rendering
render: this map appears to be in use by 1 other thread(s) which is not allowed. You need to use a map pool to avoid sharing map objects between concurrent rendering
render: this map appears to be in use by 1 other thread(s) which is not allowed. You need to use a map pool to avoid sharing map objects between concurrent rendering
render: this map appears to be in use by 1 other thread(s) which is not allowed. You need to use a map pool to avoid sharing map objects between concurrent rendering
render: this map appears to be in use by 1 other thread(s) which is not allowed. You need to use a map pool to avoid sharing map objects between concurrent rendering
render: this map appears to be in use by 1 other thread(s) which is not allowed. You need to use a map pool to avoid sharing map objects between concurrent rendering
render: this map appears to be in use by 1 other thread(s) which is not allowed. You need to use a map pool to avoid sharing map objects between concurrent rendering
render: this map appears to be in use by 1 other thread(s) which is not allowed. You need to use a map pool to avoid sharing map objects between concurrent rendering
render: this map appears to be in use by 1 other thread(s) which is not allowed. You need to use a map pool to avoid sharing map objects between concurrent rendering
render: this map appears to be in use by 1 other thread(s) which is not allowed. You need to use a map pool to avoid sharing map objects between concurrent rendering
render: this map appears to be in use by 1 other thread(s) which is not allowed. You need to use a map pool to avoid sharing map objects between concurrent rendering
render: this map appears to be in use by 1 other thread(s) which is not allowed. You need to use a map pool to avoid sharing map objects between concurrent rendering
[6m 8s] Part(1/1) 7.3660% 2.5k/33.9k @ 16/s | 1h 17m 13s left | ✓ 1.1k ■ 0 □ 1.3k fail 117

Will post the trace if it crashes.

springmeyer commented 10 years ago

/home/layers/imagelayer89.tif not recognised as a supported file format. is cause for concern. I presume that this file does actually exist and is valid but it just failing to be opened during the export due to some other issue. Can you confirm its valid? Perhaps you are seeing this same error not recognised as a supported file format but randomly for different files each run?

nickponline commented 10 years ago

Yeah the images are valid, here are the two mentioned: https://www.dropbox.com/s/lcvsb3fuhi67rsh/Archive.zip?dl=0

For the first run the erroneous images are reported in this order:

imagelayer86.tif
imagelayer95.tif
imagelayer10.tif
imagelayer95.tif
imagelayer86.tif
imagelayer6.tif
imagelayer2.tif
imagelayer17.tif
imagelayer38.tif
imagelayer28.tif
imagelayer10.tif
imagelayer57.tif
imagelayer117.tif
imagelayer2.tif
imagelayer17.tif
imagelayer38.tif
imagelayer6.tif
imagelayer48.tif
imagelayer14.tif
imagelayer6.tif
imagelayer21.tif
imagelayer28.tif
imagelayer57.tif
imagelayer117.tif
imagelayer35.tif
imagelayer42.tif
imagelayer49.tif
imagelayer9.tif
imagelayer14.tif
imagelayer48.tif
imagelayer7.tif
imagelayer27.tif
imagelayer48.tif
imagelayer21.tif
imagelayer15.tif
imagelayer21.tif
imagelayer74.tif
imagelayer21.tif
imagelayer8.tif
export core dumped

Where as for the second run:

imagelayer148.tif
imagelayer14.tif
imagelayer49.tif
imagelayer95.tif
imagelayer123.tif
imagelayer14.tif
imagelayer95.tif
imagelayer49.tif
imagelayer148.tif
imagelayer118.tif
imagelayer123.tif
imagelayer118.tif
imagelayer123.tif
imagelayer166.tif
imagelayer58.tif
imagelayer91.tif
imagelayer98.tif
imagelayer165.tif
imagelayer166.tif
imagelayer58.tif
imagelayer13.tif
imagelayer152.tif
imagelayer58.tif
imagelayer56.tif
imagelayer91.tif
imagelayer98.tif
imagelayer83.tif
imagelayer94.tif
imagelayer165.tif
imagelayer56.tif
imagelayer152.tif
imagelayer13.tif
imagelayer148.tif
imagelayer2.tif
imagelayer113.tif
imagelayer83.tif
imagelayer94.tif
imagelayer2.tif
imagelayer33.tif
imagelayer2.tif
imagelayer148.tif
imagelayer77.tif
imagelayer113.tif
imagelayer33.tif
imagelayer77.tif
imagelayer11.tif
imagelayer28.tif
imagelayer11.tif
imagelayer73.tif
imagelayer28.tif
imagelayer23.tif
imagelayer8.tif
imagelayer28.tif
imagelayer8.tif
imagelayer23.tif
imagelayer82.tif
imagelayer110.tif
imagelayer73.tif
imagelayer82.tif
imagelayer78.tif
imagelayer110.tif
imagelayer78.tif
imagelayer89.tif
imagelayer144.tif
export still running
nickponline commented 10 years ago

There are images in the first run that aren't in the second, images in the second run that aren't in the first, and images in both runs.

springmeyer commented 10 years ago

Okay, so yeah, something is making GDAL think they are invalid. What if you pass export CPL_DEBUG=ON - can you get an indication what GDAL thinks is wrong? Is the process running out of file handles ulimit?

nickponline commented 10 years ago

The only difference between the first and second (which has now completed without a core dumped albeit the errors reported above) was running ulimit -c unlimited in preparation for the core dump. I ran with CPL_DEBUG enabled and it core dumped again. Console output: https://gist.github.com/nickponline/485cb66ac8519b8f97e5 (will be truncated, so useful stuff might be at the end) trace coming now.

nickponline commented 10 years ago

Backtrace: https://gist.github.com/nickponline/c836f90f7364a6d6f1a0