std::bad_alloc core dump example

[jsta]

I'm trying to extract a raster from the gdb located at:

https://gisdata.mn.gov/dataset/water-lake-bathymetry

• ftp://ftp.gisdata.mn.gov/pub/gdrs/data/pub/us_mn_state_dnr/water_lake_bathymetry/fgdb_water_lake_bathymetry.zip

Running into an error (copied below) that I thought I'd share:

arc_raster.exe data/mn_bathy/water_lake_bathymetry.gdb/ 0 lake_bathymetric_elevation_model

arc_raster.exe data/mn_bathy/water_lake_bathymetry.gdb/ 0 lake_bathymetric_elevation_model Arc Raster Rescue (url=github.com/r-barnes/ArcRasterRescue, hash=2f4140e9d209355e, compiled=2019-08-23 12:48:01 UTC) Allocating 4x104576x117376 = 49098850304 terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc Aborted (core dumped)

[Dave-Evans]

Hi @jsta did you end up figuring this out? I don't know anything about C++ but I was assuming that this meant that the file was too large to fit in RAM. Do you know if this is not the issue?

Thanks, Dave

[r-barnes]

@jsta: That's a 49GB allocation, so it's not surprising it would fail. Are those dimensions (104576x117376) cells correct?

[jsta]

I'm getting slightly under those numbers but pretty close. It is quite large.

[r-barnes]

@jsta : Given that the file is only 1.1GB, I'm struggling to see how the dataset itself could be that large.

[jsta]

It's 1.7 GB when extracted as a tif. I don't know if it'll help but here is the gdalinfo readout:

gdalinfo lake_bathymetric_elevation_model.tif

``` Driver: GTiff/GeoTIFF Files: lake_bathymetric_elevation_model.tif Size is 104468, 117233 Coordinate System is: PROJCRS["NAD_1983_UTM_Zone_15N", BASEGEOGCRS["NAD83", DATUM["North American Datum 1983", ELLIPSOID["GRS 1980",6378137,298.257222101004, LENGTHUNIT["metre",1]]], PRIMEM["Greenwich",0, ANGLEUNIT["degree",0.0174532925199433]], ID["EPSG",4269]], CONVERSION["UTM zone 15N", METHOD["Transverse Mercator", ID["EPSG",9807]], PARAMETER["Latitude of natural origin",0, ANGLEUNIT["degree",0.0174532925199433], ID["EPSG",8801]], PARAMETER["Longitude of natural origin",-93, ANGLEUNIT["degree",0.0174532925199433], ID["EPSG",8802]], PARAMETER["Scale factor at natural origin",0.9996, SCALEUNIT["unity",1], ID["EPSG",8805]], PARAMETER["False easting",500000, LENGTHUNIT["metre",1], ID["EPSG",8806]], PARAMETER["False northing",0, LENGTHUNIT["metre",1], ID["EPSG",8807]]], CS[Cartesian,2], AXIS["easting",east, ORDER[1], LENGTHUNIT["metre",1]], AXIS["northing",north, ORDER[2], LENGTHUNIT["metre",1]], ID["EPSG",26915]] Data axis to CRS axis mapping: 1,2 Origin = (199105.000000000000000,5403520.000000000000000) Pixel Size = (5.000000000000000,-5.000000000000000) Metadata: AREA_OR_POINT=Area Image Structure Metadata: COMPRESSION=LZW INTERLEAVE=BAND Corner Coordinates: Upper Left ( 199105.000, 5403520.000) ( 97d 5'26.65"W, 48d42'43.25"N) Lower Left ( 199105.000, 4817355.000) ( 96d43' 6.99"W, 43d26'55.03"N) Upper Right ( 721445.000, 5403520.000) ( 89d59'15.24"W, 48d44'43.05"N) Lower Right ( 721445.000, 4817355.000) ( 90d15'43.10"W, 43d28'34.80"N) Center ( 460275.000, 5110437.500) ( 93d30'51.79"W, 46d 8'46.88"N) Band 1 Block=128x128 Type=Float32, ColorInterp=Gray NoData Value=-3.40282306073709653e+38 Unit Type: metre Metadata: RepresentationType=ATHEMATIC ```

[r-barnes]

Thanks, @jsta. Does the raster contain large areas of no_data or with pixels that are all the same value? (Does it look like it would compress nicely?)

[andersmd7]

I'm encountering a similar issue trying to rescue a File Geodatabase raster from https://www.fs.usda.gov/rds/archive/products/RDS-2019-0030/RDS-2019-0030.zip (U.S. Forest Service Riparian Areas layer).

Allocating 1x462083x286682 = 132470878606
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Aborted

This is a national (U.S.) layer, at 10m resolution (though w/ only 2 integer values, and lots of NoData, so the geodatabase is only around 2.15 GB, though an uncompressed version would be much larger. Does ArcRasterRescue have a method for handling rasters that are too large to fit in RAM?