search

zkwurst / GSoC2017-GRASS-GIS

GNU General Public License v2.0
2 stars 1 forks source link

NAIP API return crashes g.message, r.import fails #11

Open zkwurst opened 7 years ago

zkwurst commented 7 years ago
  1. data_info will not print with "i" flag or without
  2. NAIP tiles will download but r.import fails

Below are notes I took while running tests:

08-10-17 r.in.usgs NAIP tests

########## NAIP, 2011, Land Cover ########## Test 1 Info

Conditions: Clean directory

Results: Completed script without returning anything...

Makes it to:

gscript.verbose("TNM API Query URL:\t{0}".format(TNM_API_URL))

try:
    # Query TNM API
    TNM_API_GET = urllib2.urlopen(TNM_API_URL, timeout=12)
except urllib2.URLError:
    gscript.fatal("USGS TNM API query has timed out. Check network configuration. Please try again.")

try:
    # Parse return JSON object
    return_JSON = json.load(TNM_API_GET)
except:
    gscript.fatal("Unable to load USGS JSON object.")

...but is exiting for some reason... not sure why it doesn't get caught by the fatal message.

GRASS output: (Thu Aug 10 17:11:15 2017)
/home/zechariah/git/GSoC2017-GRASS-GIS/r.in.usgs/r.in.usgs.py -i --overwrite --verbose product=naip output_directory=/home/zechariah/Downloads/08-09_test output=08-09_test The default resampling method for product naip is nearest Assuming current location as input Input parameters: '+proj=longlat +no_defs +a=6378137 +rf=298.257222101 +towgs84=0,0,0,0,0,0,0' Output parameters: '+proj=longlat +ellps=GRS80 +datum=NAD83 +nodefs' Assuming current location as input Input parameters: '+proj=longlat +no_defs +a=6378137 +rf=298.257222101 +towgs84=0,0,0,0,0,0,0' Output parameters: '+proj=longlat +ellps=GRS80 +datum=NAD83 +nodefs' TNM API Query URL: https://viewer.nationalmap.gov/tnmaccess/api/products?datasets=USDA+National+Agriculture+Imagery+Program+%28NAIP%29&bbox=-79.12972222,35.29583333,-78.13166667,36.27194444&prodFormats=JPEG2000 To download USGS data, remove flag, and rerun r.in.usgs. (Thu Aug 10 17:11:20 2017) Command finished (5 sec)

########## Test 2 Download

Conditions: same as test 1

Results: Began to download 306 files. Script stopped manually.

GRASS Output: (Thu Aug 10 17:20:52 2017)
/home/zechariah/git/GSoC2017-GRASS-GIS/r.in.usgs/r.in.usgs.py --overwrite --verbose product=naip output_directory=/home/zechariah/Downloads/08-09_test output=08-09_test The default resampling method for product naip is nearest Assuming current location as input Input parameters: '+proj=longlat +no_defs +a=6378137 +rf=298.257222101 +towgs84=0,0,0,0,0,0,0' Output parameters: '+proj=longlat +ellps=GRS80 +datum=NAD83 +nodefs' Assuming current location as input Input parameters: '+proj=longlat +no_defs +a=6378137 +rf=298.257222101 +towgs84=0,0,0,0,0,0,0' Output parameters: '+proj=longlat +ellps=GRS80 +datum=NAD83 +nodefs' TNM API Query URL: https://viewer.nationalmap.gov/tnmaccess/api/products?datasets=USDA+National+Agriculture+Imagery+Program+%28NAIP%29&bbox=-79.12972222,35.29583333,-78.13166667,36.27194444&prodFormats=JPEG2000 Downloading USGS Data... Download 1 of 306: COMPLETE Please note that the data are left in inconsistent state and may be corrupted
(Thu Aug 10 17:21:09 2017) Command aborted (16 sec)

########## Test 3 - FAIL Info

Conditions: Empty download directory Changed g.region to smaller single-tile region

Results: Single tile region returns ~20 tiles instead of 306 the multi-tile region did. Module still exits without warning and does not print info. g.message crashes in GRASS terminal.

GRASS Terminal Output: buffer overflow detected : g.message terminated ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7febbefda7e5] /lib/x86_64-linux-gnu/libc.so.6(fortify_fail+0x5c)[0x7febbf07c11c] /lib/x86_64-linux-gnu/libc.so.6(+0x117120)[0x7febbf07a120] /lib/x86_64-linux-gnu/libc.so.6(+0x116689)[0x7febbf079689] /lib/x86_64-linux-gnu/libc.so.6(_IO_default_xsputn+0x80)[0x7febbefde6b0] /lib/x86_64-linux-gnu/libc.so.6(_IO_vfprintf+0x139b)[0x7febbefb150b] /lib/x86_64-linux-gnu/libc.so.6(__vsprintf_chk+0x84)[0x7febbf079714] /home/zechariah/dev/grass7_trunk/dist.x86_64-pc-linux-gnu/lib/libgrass_gis.7.3.svn.so(+0x134af)[0x7febbf3404af] /home/zechariah/dev/grass7_trunk/dist.x86_64-pc-linux-gnu/lib/libgrass_gis.7.3.svn.so(G_message+0xb3)[0x7febbf340af3] g.message(main+0x3ec)[0x400f9c] /lib/x86_64-linux-gnu/libc.so.6(libc_start_main+0xf0)[0x7febbef83830] g.message(_start+0x29)[0x400fe9] ======= Memory map: ======== 00400000-00402000 r-xp 00000000 08:01 416133 /home/zechariah/dev/grass7_trunk/dist.x86_64-pc-linux-gnu/bin/g.message 00601000-00602000 r--p 00001000 08:01 416133 /home/zechariah/dev/grass7_trunk/dist.x86_64-pc-linux-gnu/bin/g.message 00602000-00603000 rw-p 00002000 08:01 416133 /home/zechariah/dev/grass7_trunk/dist.x86_64-pc-linux-gnu/bin/g.message 022c6000-022e7000 rw-p 00000000 00:00 0 [heap] 7febbe349000-7febbe35f000 r-xp 00000000 08:01 267420 /lib/x86_64-linux-gnu/libgcc_s.so.1 7febbe35f000-7febbe55e000 ---p 00016000 08:01 267420 /lib/x86_64-linux-gnu/libgcc_s.so.1 7febbe55e000-7febbe55f000 rw-p 00015000 08:01 267420 /lib/x86_64-linux-gnu/libgcc_s.so.1 7febbe55f000-7febbe837000 r--p 00000000 08:01 1310801 /usr/lib/locale/locale-archive 7febbe837000-7febbe93f000 r-xp 00000000 08:01 265175 /lib/x86_64-linux-gnu/libm-2.23.so 7febbe93f000-7febbeb3e000 ---p 00108000 08:01 265175 /lib/x86_64-linux-gnu/libm-2.23.so 7febbeb3e000-7febbeb3f000 r--p 00107000 08:01 265175 /lib/x86_64-linux-gnu/libm-2.23.so 7febbeb3f000-7febbeb40000 rw-p 00108000 08:01 265175 /lib/x86_64-linux-gnu/libm-2.23.so 7febbeb40000-7febbeb59000 r-xp 00000000 08:01 287683 /lib/x86_64-linux-gnu/libz.so.1.2.8 7febbeb59000-7febbed58000 ---p 00019000 08:01 287683 /lib/x86_64-linux-gnu/libz.so.1.2.8 7febbed58000-7febbed59000 r--p 00018000 08:01 287683 /lib/x86_64-linux-gnu/libz.so.1.2.8 7febbed59000-7febbed5a000 rw-p 00019000 08:01 287683 /lib/x86_64-linux-gnu/libz.so.1.2.8 7febbed5a000-7febbed62000 r-xp 00000000 08:01 415209 /home/zechariah/dev/grass7_trunk/dist.x86_64-pc-linux-gnu/lib/libgrass_datetime.7.3.svn.so 7febbed62000-7febbef61000 ---p 00008000 08:01 415209 /home/zechariah/dev/grass7_trunk/dist.x86_64-pc-linux-gnu/lib/libgrass_datetime.7.3.svn.so 7febbef61000-7febbef62000 r--p 00007000 08:01 415209 /home/zechariah/dev/grass7_trunk/dist.x86_64-pc-linux-gnu/lib/libgrass_datetime.7.3.svn.so 7febbef62000-7febbef63000 rw-p 00008000 08:01 415209 /home/zechariah/dev/grass7_trunk/dist.x86_64-pc-linux-gnu/lib/libgrass_datetime.7.3.svn.so 7febbef63000-7febbf123000 r-xp 00000000 08:01 265180 /lib/x86_64-linux-gnu/libc-2.23.so 7febbf123000-7febbf323000 ---p 001c0000 08:01 265180 /lib/x86_64-linux-gnu/libc-2.23.so 7febbf323000-7febbf327000 r--p 001c0000 08:01 265180 /lib/x86_64-linux-gnu/libc-2.23.so 7febbf327000-7febbf329000 rw-p 001c4000 08:01 265180 /lib/x86_64-linux-gnu/libc-2.23.so 7febbf329000-7febbf32d000 rw-p 00000000 00:00 0 7febbf32d000-7febbf376000 r-xp 00000000 08:01 415271 /home/zechariah/dev/grass7_trunk/dist.x86_64-pc-linux-gnu/lib/libgrass_gis.7.3.svn.so 7febbf376000-7febbf576000 ---p 00049000 08:01 415271 /home/zechariah/dev/grass7_trunk/dist.x86_64-pc-linux-gnu/lib/libgrass_gis.7.3.svn.so 7febbf576000-7febbf577000 r--p 00049000 08:01 415271 /home/zechariah/dev/grass7_trunk/dist.x86_64-pc-linux-gnu/lib/libgrass_gis.7.3.svn.so 7febbf577000-7febbf578000 rw-p 0004a000 08:01 415271 /home/zechariah/dev/grass7_trunk/dist.x86_64-pc-linux-gnu/lib/libgrass_gis.7.3.svn.so 7febbf578000-7febbf579000 rw-p 00000000 00:00 0 7febbf579000-7febbf59f000 r-xp 00000000 08:01 265126 /lib/x86_64-linux-gnu/ld-2.23.so 7febbf777000-7febbf77b000 rw-p 00000000 00:00 0 7febbf79b000-7febbf79e000 rw-p 00000000 00:00 0 7febbf79e000-7febbf79f000 r--p 00025000 08:01 265126 /lib/x86_64-linux-gnu/ld-2.23.so 7febbf79f000-7febbf7a0000 rw-p 00026000 08:01 265126 /lib/x86_64-linux-gnu/ld-2.23.so 7febbf7a0000-7febbf7a1000 rw-p 00000000 00:00 0 7ffef075d000-7ffef077f000 rw-p 00000000 00:00 0 [stack] 7ffef07ce000-7ffef07d0000 r--p 00000000 00:00 0 [vvar] 7ffef07d0000-7ffef07d2000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

########## Test 4 - FAIL Download

Conditions: Same as test 3 but without "i" flag. Info still doesn't print, but ran download anyway.

Results: All 20 tiles downloaded Info didn't print Module failed on import

GRASS Output: (Fri Aug 11 12:37:36 2017)
/home/zechariah/git/GSoC2017-GRASS-GIS/r.in.usgs/r.in.usgs.py product=naip output_directory=/home/zechariah/Downloads/08-11_test output=08-11_test_out Downloading USGS Data... Download 1 of 20: COMPLETE Download 2 of 20: COMPLETE Download 3 of 20: COMPLETE Download 4 of 20: COMPLETE Download 5 of 20: COMPLETE Download 6 of 20: COMPLETE Download 7 of 20: COMPLETE Download 8 of 20: COMPLETE Download 9 of 20: COMPLETE Download 10 of 20: COMPLETE Download 11 of 20: COMPLETE Download 12 of 20: COMPLETE Download 13 of 20: COMPLETE Download 14 of 20: COMPLETE Download 15 of 20: COMPLETE Download 16 of 20: COMPLETE Download 17 of 20: COMPLETE Download 18 of 20: COMPLETE Download 19 of 20: COMPLETE Download 20 of 20: COMPLETE Importing and reprojecting m_3507811_se_17_1_20140618_20141118.jp2... Importing 4 raster bands... Importing raster map ... Warning 1: Empty SOT marker detected: Psot=12. Importing raster map ... Importing raster map ... Importing raster map ... WARNING: pj_transform() failed: failed to load datum shift file ERROR: Unable to re-project vector map vreg_tmp_1634@PERMANENT from ERROR: Unable to reproject to source location Error in atexit._run_exitfuncs: Traceback (most recent call last): File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs func(*targs, *kargs) File "/home/zechariah/dev/grass7_trunk/dist.x86_64-pc- linux-gnu/scripts/r.import", line 151, in cleanup mapset=grass.gisenv()['MAPSET'])['fullname']: KeyError: 'MAPSET' Error in sys.exitfunc: Traceback (most recent call last): File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs func(targs, **kargs) File "/home/zechariah/dev/grass7_trunk/dist.x86_64-pc- linux-gnu/scripts/r.import", line 151, in cleanup mapset=grass.gisenv()['MAPSET'])['fullname']: KeyError: 'MAPSET' ERROR: Unable to import 'm_3507811_se_17_1_20140618_20141118.jp2' (Fri Aug 11 12:41:43 2017) Command finished (4 min 7 sec)

I found this thread that mentions vaguely similar issues: http://osgeo-org.1560.x6.nabble.com/r-sim-water-buffer-overflow-td3999042.html

Not sure which of the issues discussed in the thread could be at play here... something with my GRASS compilation, latlong, resolution settings?

petrasovaa commented 7 years ago

This is way to long to read, I am little bit short on time. If I understand correctly the main problem is the g.message - in that case just use print. You are trying to print a lot of data and g.message is not meant for that I guess. Regarding the NAIP problem, that is exactly what I was writing you about in my last email. Unfortunately I don't have a solution or a workaround for it at this point. So don't worry about NAIP, keep the NAIP code there, it will be useful once that is resolved.

zkwurst commented 7 years ago

No worries and thanks for the feedback! I hadn't associated the NAIP projection issue with this.