search

GenericMappingTools / gmt

The Generic Mapping Tools
https://www.generic-mapping-tools.org
Other
848 stars 355 forks source link

Segmentation fault in grdpaste #746

Closed sebastic closed 5 years ago

sebastic commented 5 years ago

Description of the problem

As reported by Joachim Breitner in Debian Bug #929264:

this sequence of commands causes a segementation fault. This is a regression; this is from a project from a few years back and worked then:

wget -c http://dds.cr.usgs.gov/srtm/version2_1/SRTM3/Eurasia/N45E0{06,07}.hgt.zip
unzip N45E006.hgt.zip
gmt xyz2grd N45E006.hgt -R6/7/45/46 -I3s -di-32768 -ZTLhw -GN45E006.grd
gmt grd2xyz N45E006.grd >N45E006.txt
gmt surface N45E006.txt -RN45E006.grd -T0.35 -GN45E006.grd
unzip N45E007.hgt.zip
gmt xyz2grd N45E007.hgt -R7/8/45/46 -I3s -di-32768 -ZTLhw -GN45E007.grd
gmt grd2xyz N45E007.grd >N45E007.txt
gmt surface N45E007.txt -RN45E007.grd -T0.35 -GN45E007.grd
cp N45E006.grd 45tmp.grd
gmt grdpaste 45tmp.grd N45E007.grd -G45tmp.grd
ERROR: Caught signal number 11 (Segmentation fault) at
/usr/lib/x86_64-linux-gnu/libnetcdf.so.13(NC_check_nulls+0xc7)[0x7f9dbb8af9a7]
[0x0]
Stack backtrace:
/usr/lib/x86_64-linux-gnu/libgmt.so.5(sig_handler+0x2ac)[0x7f9dbb9f006c]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12670)[0x7f9dbb4c9670]
/usr/lib/x86_64-linux-gnu/libnetcdf.so.13(NC_check_nulls+0xc7)[0x7f9dbb8af9a7]
/usr/lib/x86_64-linux-gnu/libnetcdf.so.13(+0x2df9f)[0x7f9dbb8aff9f]
/usr/lib/x86_64-linux-gnu/libnetcdf.so.13(nc_get_varm_float+0xf)[0x7f9dbb8b133f]
/usr/lib/x86_64-linux-gnu/libgmt.so.5(+0x91c24)[0x7f9dbba54c24]
/usr/lib/x86_64-linux-gnu/libgmt.so.5(gmt_nc_read_grd+0x826)[0x7f9dbba55bb6]
/usr/lib/x86_64-linux-gnu/libgmt.so.5(gmtlib_read_grd+0x9e)[0x7f9dbba37ede]
/usr/lib/x86_64-linux-gnu/libgmt.so.5(+0x488e8)[0x7f9dbba0b8e8]
/usr/lib/x86_64-linux-gnu/libgmt.so.5(GMT_Read_Data+0x4c3)[0x7f9dbba0d213]
/usr/lib/x86_64-linux-gnu/libgmt.so.5(GMT_grdpaste+0x3e0)[0x7f9dbbbaddc0]
/usr/lib/x86_64-linux-gnu/libgmt.so.5(GMT_Call_Module+0xc0)[0x7f9dbb9faf50]
gmt(main+0x5c6)[0x55ef48806716]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)[0x7f9db8ca709b]
gmt(_start+0x2a)[0x55ef48806cda]

System information

welcome[bot] commented 5 years ago

👋 Thanks for opening your first issue here! Please make sure you filled out the template with as much detail as possible. We appreciate that you took the time to contribute!

Please make sure you read our Contributing Guide and abide by our Code of Conduct.

joa-quim commented 5 years ago

Waw, what a complicated way of doing

grdpaste N45E006.hgt N45E007.hgt -G45tmp.grd

Anyway, it doesn't crash with the developing version (soon GMT6), which means the problem has been fixed, but the result is wrong probably due to some bad parameter in the convoluted recipe for pasting the two grids (didn't check any further).

nomeata commented 5 years ago

Thanks for letting me know that I can pass .hgt files directly to grdpaste. But I still get segmentation fails when I want to paste more together:

gmt grdpaste N45E006.hgt N45E007.hgt -G45tmp.grd
gmt grdpaste 45tmp.grd N45E008.hgt -G45tmp.grd
ERROR: Caught signal number 11 (Segmentation fault) at
/usr/lib/x86_64-linux-gnu/libnetcdf.so.13(NC_check_nulls+0xc7)[0x7f482a8469a7]
[0x0]
Stack backtrace:
/usr/lib/x86_64-linux-gnu/libgmt.so.5(sig_handler+0x2ac)[0x7f482a98706c]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12670)[0x7f482a460670]
/usr/lib/x86_64-linux-gnu/libnetcdf.so.13(NC_check_nulls+0xc7)[0x7f482a8469a7]
/usr/lib/x86_64-linux-gnu/libnetcdf.so.13(+0x2df9f)[0x7f482a846f9f]
/usr/lib/x86_64-linux-gnu/libnetcdf.so.13(nc_get_varm_float+0xf)[0x7f482a84833f]
/usr/lib/x86_64-linux-gnu/libgmt.so.5(+0x91c24)[0x7f482a9ebc24]
/usr/lib/x86_64-linux-gnu/libgmt.so.5(gmt_nc_read_grd+0x826)[0x7f482a9ecbb6]
/usr/lib/x86_64-linux-gnu/libgmt.so.5(gmtlib_read_grd+0x9e)[0x7f482a9ceede]
/usr/lib/x86_64-linux-gnu/libgmt.so.5(+0x488e8)[0x7f482a9a28e8]
/usr/lib/x86_64-linux-gnu/libgmt.so.5(GMT_Read_Data+0x4c3)[0x7f482a9a4213]
/usr/lib/x86_64-linux-gnu/libgmt.so.5(GMT_grdpaste+0x3e0)[0x7f482ab44dc0]
/usr/lib/x86_64-linux-gnu/libgmt.so.5(GMT_Call_Module+0xc0)[0x7f482a991f50]
gmt(main+0x5c6)[0x557fc0686716]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)[0x7f4827c3e09b]
gmt(_start+0x2a)[0x557fc0686cda]
joa-quim commented 5 years ago

Right, but that's GMT5. What I said is that the problem was fixed meanwhile. The developing version (GMT6dev) has it fixed.

nomeata commented 5 years ago

Ok, thanks. (I discovered that I can work-around the issue by not pasting the files, but rather calling grdimage on each individual tile, ignoring warnings when the whole tile is outside the requested range).

joa-quim commented 5 years ago

Hmm, it's actually more strange. Your example works fine for me on Windows

grdpaste -
grdpaste(core) 5.4.5 [64-bit] [MP] - Join two grids along their common edge

usage: grdpaste <grid1> <grid2> -G<outgrid> [-V[<level>]] [-f[i|o]<info>]

grdpaste N45E006.hgt N45E007.hgt -G45tmp.grd

grdinfo 45tmp.grd
45tmp.grd: Title: N45E006
45tmp.grd: Command: grdpaste N45E006.hgt N45E007.hgt -G45tmp.grd
45tmp.grd: Remark: Assumed to be a SRTM3 tile
45tmp.grd: Gridline node registration used [Geographic grid]
45tmp.grd: Grid file format: nf = GMT netCDF format (32-bit float), COARDS, CF-1.5
45tmp.grd: x_min: 6 x_max: 8 x_inc: 0.000833333333333 name: longitude [degrees_east] n_columns: 2401
45tmp.grd: y_min: 45 y_max: 46 y_inc: 0.000833333333333 name: latitude [degrees_north] n_rows: 1201
45tmp.grd: z_min: 156 z_max: 4672 name: z
45tmp.grd: scale_factor: 1 add_offset: 0
45tmp.grd: format: netCDF-4 chunk_size: 134,134 shuffle: on deflation_level: 3
joa-quim commented 5 years ago

But I see the your crash comes from netcdf. I remember there was something about a bug in netcdf checking NULLs. Paul made a patch to GMT because of it and apparently the bug has been fixed. Try to update your netcdf lib to latest version (if the linux gods allow such heresy)

PaulWessel commented 5 years ago

if your netCDF is not 4.6.3 then you have a buggy netcdf.

sebastic commented 5 years ago

Do you know which commit from NetCDF 4.6.3 is the fix?

We have 4.6.2 in buster, later versions were released after the freeze.

PaulWessel commented 5 years ago

Not at the moment - would have to look in their README/logs - and headed to dentist as we speak. I think 4.6.2 was the problem but need to check. Distros are like elephants, big and slow. At macports we are living with a buggy ghostscript 9.26 yet 9.27 was released...

PaulWessel commented 5 years ago

4.6.2 introduced the bug and 4.6.3 fixed it.

sebastic commented 5 years ago

4.6.3 also bumped the SONAME which requires a transition.

If issue in question is https://github.com/Unidata/netcdf-c/issues/1265, that has a bunch of commits claiming to fix it, and by the looks of it far to invasive to backport.

I guess we'll just have to live with a buggy NetCDF in buster.

PaulWessel commented 5 years ago

Yes, that is the one. Is 4.6.1 an option?

Paul Wessel, Professor and Chair Dept. of Earth Sciences (formerly Geology & Geophysics) SOEST, U of Hawaii at Manoa

On May 20, 2019 at 8:59:06 AM, Bas Couwenberg (notifications@github.com) wrote:

4.6.3 also bumped the SONAME which requires a transition.

If issue in question is Unidata/netcdf-c#1265 https://github.com/Unidata/netcdf-c/issues/1265, that has a bunch of commits claiming to fix it, and by the looks of it far to invasive to backport.

I guess we'll just have to live with a buggy NetCDF in buster.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/GenericMappingTools/gmt/issues/746?email_source=notifications&email_token=AGJ7IX4HL4DVSVGKGI25A2DPWLYHVA5CNFSM4HOAQIS2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVZYN3I#issuecomment-494110445, or mute the thread https://github.com/notifications/unsubscribe-auth/AGJ7IX5GEDXNGATXUJZIH3LPWLYHVANCNFSM4HOAQISQ .

sebastic commented 5 years ago

Downgrading to 4.6.1 is not an option.

Looking at the changes between .1 & .2, I guess https://github.com/Unidata/netcdf-c/pull/1001 introduced the issue. Reverting that also looks too invasive.

PaulWessel commented 5 years ago

We implemented a workaround in GMT to avoid the bug while we waited for relief, so both the 5.4 branch and 6 are immune.

sebastic commented 5 years ago

Do you mean https://github.com/GenericMappingTools/gmt/commit/1a5a11a29eaa242258d088fdf2690c0452986988?

That looks good to cherry-pick as patch for the Debian package.

@nomeata, can you test that?

PaulWessel commented 5 years ago

Yep, that is the one. It crashed on NULL due to their bug.

nomeata commented 5 years ago

@nomeata, can you test that?

Not easily (I'm not building it from source), but the instructions above do not require any private data, so you should easily be able to test the package before uploading.

sebastic commented 5 years ago

I've included the changes from 1a5a11a29eaa242258d088fdf2690c0452986988 as a patch in the gmt Debian package, and that fixes the issue.

I'll request an unblock to get it into testing and upcoming buster stable release once it's in the archive.

We can close this issue as it's a duplicate of #512.

nomeata commented 5 years ago

Thanks, nice work!