joa-quim
commented
2 years ago You can try this.
gmt grdconvert 60062_L05_32631_DTM_18_S02_20180814_1M.XYZ -G60062_L05_32631_DTM_18_S02_20180814_1M.ncClosed anbj closed 2 years ago
joa-quim
commented
2 years ago You can try this.
gmt grdconvert 60062_L05_32631_DTM_18_S02_20180814_1M.XYZ -G60062_L05_32631_DTM_18_S02_20180814_1M.nc
anbj
commented
2 years ago Thanks @joa-quim.
I get:
$ gmt grdconvert 60062_L05_32631_DTM_18_S02_20180814_1M.XYZ -G60062_L05_32631_DTM_18_S02_20180814_1M.nc
ERROR 1: Ungridded dataset: At line 111098, change of Y direction
grdconvert (GMT_grdconvert): Not a supported grid format [60062_L05_32631_DTM_18_S02_20180814_1M.XYZ]I think this is gridded, but have not checked all 29448347 lines. But, that wouldn't matter to xyz2grd, right? It just takes the value at the given x,y, and puts it into the the z-box? It does not care how the lines are sorted in the input?
anbj
commented
2 years ago Not very related, but still; this is bathymetry (thin strip) acquired in a 'diagonal' direction (relative to N/S), so that (I think) >~ 90% of the cells of the encompassing grid rectangle will be NaN's. Stupid! Is a custom projection a solution here?
joa-quim
commented
2 years ago The grdconvert way is using GDAL under the hood and it expects the data to be gridded. But xyz2grd does expect that too. The pick the closest point and put it into the node is an (pseudo) interpolation algorithm called nearest-neighbor, but GMT does not have it (although it has one with a similar name).
So your data is not gridded and you need to do it, but the file is big so ... by chunks?
And yes, if it's a MB transit you could in principle project the data into an oblique Mercator and grid that.
anbj
commented
2 years ago Thanks again @joa-quim. I thought xyz2grd didn't care about the sorting. A bit puzzled by it saying it's not gridded..
Yup, chunk is probably an good option. I would like to test oblique Mercator also. Several MB transits as you guessed.
joa-quim
commented
2 years ago I don't think it cares about sorting but it cares on the data being previously gridded.
joa-quim
commented
2 years ago Several MB transits as you guessed.
You should probably be using MB-system and its module mbgrid.
anbj
commented
2 years ago Trying to replicate this, but not sure how to restrict amount of memory available for gmt to use. Not even sure if memory shortage is the problem.
Making a bogus grid gives similar output, but it stops after gmt [DEBUG]: Entering GMT_Destroy_Session. There is no such line in the debug output of my original post.
$ echo 0.5 0.5 0.5 | gmt xyz2grd -I0.00001 -Gtest.nc -R0/1/0/1 -Vd
gmt [DEBUG]: GMT_Create_Session: Terminal width = 211
gmt [DEBUG]: Obtained the ppid from parent: 15894
gmt [DEBUG]: Enter: gmtinit_new_GMT_ctrl
gmt [DEBUG]: GMT->session.SHAREDIR = /usr/local/share
gmt [DEBUG]: GMT->session.HOMEDIR = /home/anbj
gmt [DEBUG]: GMT->session.USERDIR = /home/anbj/.gmt [created]
gmt [DEBUG]: GMT->session.CACHEDIR = /home/anbj/.gmt/cache [created]
gmt [DEBUG]: GMT: 0. Will try to find subdir=postscriptlight stem = PSL_custom_fonts suffix=.txt
gmt [DEBUG]: GMT: 1. gmt_getsharepath trying current dir
gmt [DEBUG]: GMT: 2. gmt_getsharepath trying USERDIR /home/anbj/.gmt
gmt [DEBUG]: GMT: 3. gmt_getsharepath trying USERDIR subdir /home/anbj/.gmt/postscriptlight
gmt [DEBUG]: GMT: 4. gmt_getsharepath trying SHAREDIR subdir /usr/local/share/postscriptlight
gmt [DEBUG]: GMT: 5. gmt_getsharepath trying SHAREDIR /usr/local/share
gmt [DEBUG]: GMT: 6. gmt_getsharepath failed
gmt [DEBUG]: Map distance calculation will be Cartesian
gmt [DEBUG]: Exit: gmtinit_new_GMT_ctrl
gmt [DEBUG]: Enter: New_PSL_Ctrl
gmt [DEBUG]: Exit: New_PSL_Ctrl
gmt [DEBUG]: Enter: gmt_manage_workflow
gmt [DEBUG]: Exit : gmt_manage_workflow
gmt [DEBUG]: Enter: PSL_beginsession
gmt [DEBUG]: Exit : PSL_beginsession
gmt [DEBUG]: Enter: PSL_setdefaults
gmt [DEBUG]: Exit : PSL_setdefaults
gmt [DEBUG]: Enter: gmtlib_io_init
gmt [DEBUG]: Exit : gmtlib_io_init
gmt [DEBUG]: Enter: gmt_hash_init
gmt [DEBUG]: Exit: gmt_hash_init
gmt [DEBUG]: Enter: gmt_hash_init
gmt [DEBUG]: Exit: gmt_hash_init
gmt [DEBUG]: Enter: gmt_reload_settings
gmt [DEBUG]: The PROJ_GEODESIC set to Vincenty
gmt [DEBUG]: Look for file /home/anbj/gmt.conf
gmt [DEBUG]: Look for file /home/anbj/.gmt/gmt.conf
gmt [DEBUG]: Look for file /home/anbj/.gmt/server/gmt.conf
gmt [DEBUG]: Look for file /home/anbj/.gmt/cache/gmt.conf
gmt [DEBUG]: Could not find file gmt.conf
gmt [DEBUG]: Exit: gmt_reload_settings
gmt [DEBUG]: Enter: gmtlib_plot_C_format
gmt [DEBUG]: Exit: gmtlib_plot_C_format
gmt [DEBUG]: Enter: gmtinit_get_history
gmt [DEBUG]: Enter: gmt_hash_init
gmt [DEBUG]: Exit: gmt_hash_init
gmt [DEBUG]: Exit: gmtinit_get_history
gmt [DEBUG]: Initialize FFTW with 8 threads.
gmt [DEBUG]: GMT_Create_Session initialized GMT structure
gmt [DEBUG]: Loading core GMT shared library: libgmt.so
gmt [DEBUG]: Shared Library # 0 (core). Path = libgmt.so
gmt [DEBUG]: Loading GMT plugins from: /usr/local/lib/gmt/plugins
gmt [DEBUG]: Shared Library # 1 (supplements). Path = /usr/local/lib/gmt/plugins/supplements.so
gmt [DEBUG]: Revised options: -I0.00001 -Gtest.nc -R0/1/0/1 -Vd
xyz2grd [DEBUG]: History: Process -R0/1/0/1
xyz2grd [DEBUG]: Look for file 0/1/0/1 in /home/anbj/.gmt
xyz2grd [DEBUG]: Look for file 0/1/0/1 in /home/anbj/.gmt/cache
xyz2grd [DEBUG]: Look for file 0/1/0/1 in /home/anbj/.gmt/server
xyz2grd [DEBUG]: Got regular w/e/s/n for region (0/1/0/1)
xyz2grd [INFORMATION]: Processing input table data
xyz2grd [INFORMATION]: Cartesian input grid
xyz2grd [DEBUG]: Chosen boundary condition for all edges: natural
xyz2grd [INFORMATION]: Cartesian input grid
xyz2grd (gmtapi_alloc_grid): Could not reallocate memory [37.25 Gb, 10000200001 items of 4 bytes]
gmt [DEBUG]: Entering GMT_Destroy_SessionAll these lines are 'missing' from this run, comprated to the one in the first post;
xyz2grd [DEBUG]: gmtapi_init_import: Passed family = Data Table and geometry = Point
xyz2grd [DEBUG]: Found readable file 60062_L05_32631_DTM_18_S02_20180814_1M.XYZ
xyz2grd [DEBUG]: Replace file 60062_L05_32631_DTM_18_S02_20180814_1M.XYZ with path 60062_L05_32631_DTM_18_S02_20180814_1M.XYZ
xyz2grd [DEBUG]: Found readable file 60062_L05_32631_DTM_18_S02_20180814_1M.XYZ
xyz2grd [DEBUG]: Object ID 1 : Registered Data Table File 60062_L05_32631_DTM_18_S02_20180814_1M.XYZ as an Input resource with geometry Point [n_objects = 2]
xyz2grd [DEBUG]: gmtapi_init_import: Added 1 new sources
xyz2grd [DEBUG]: GMT_Init_IO: Returned first Input object ID = 1
xyz2grd [DEBUG]: GMT_Begin_IO: Mode value 1 not considered (ignored)
xyz2grd [DEBUG]: GMT_Begin_IO: Initialize record-by-record access for Input
xyz2grd [DEBUG]: gmtapi_next_io_source: Selected object 1
xyz2grd [DEBUG]: Found readable file 60062_L05_32631_DTM_18_S02_20180814_1M.XYZ
xyz2grd [DEBUG]: Replace file 60062_L05_32631_DTM_18_S02_20180814_1M.XYZ with path 60062_L05_32631_DTM_18_S02_20180814_1M.XYZ
xyz2grd [DEBUG]: Found readable file 60062_L05_32631_DTM_18_S02_20180814_1M.XYZ
xyz2grd [DEBUG]: Calling nc_open on 60062_L05_32631_DTM_18_S02_20180814_1M.XYZ, ncid = 0, err = -51
xyz2grd [DEBUG]: Found readable file 60062_L05_32631_DTM_18_S02_20180814_1M.XYZ
xyz2grd [INFORMATION]: Reading Data Table from file 60062_L05_32631_DTM_18_S02_20180814_1M.XYZ
xyz2grd [DEBUG]: GMT_Begin_IO: Input resource access is now enabled [record-by-record]
xyz2grd [DEBUG]: Source col types: (Number,Number,Number)
xyz2grd [DEBUG]: ASCII source scanned: Numerical columns: 3, Trailing text: N, Record type: Numerical only
ERROR: Caught signal number 11 (Segmentation fault) at
/lib64/libgmt.so.6(GMT_xyz2grd+0x22b6)[0x7fb70b2ae746]
[0x253ba8534]
Stack backtrace:
/lib64/libgmt.so.6(sig_handler_unix+0xf1)[0x7fb70b053b11]
/lib64/libpthread.so.0(+0x12ce0)[0x7fb706a0bce0]
/lib64/libgmt.so.6(GMT_xyz2grd+0x22b6)[0x7fb70b2ae746]
/lib64/libgmt.so.6(GMT_Call_Module+0x34f)[0x7fb70af7793f]
gmt(main+0x27b)[0x559b4d0cc57b]
/lib64/libc.so.6(__libc_start_main+0xf3)[0x7fb70666eca3]
gmt(_start+0x2e)[0x559b4d0cd5de]Memory is surely the culprit. Your first post had
xyz2grd (GMT_xyz2grd): Could not reallocate memory [9.31 Gb, 2498741960 items of 4 bytes]
Ah, and the last one too
xyz2grd (gmtapi_alloc_grid): Could not reallocate memory [37.25 Gb, 10000200001 items of 4 bytes]
PaulWessel
commented
2 years ago xyz2grd (gmtapi_alloc_grid): Could not reallocate memory [37.25 Gb, 10000200001 items of 4 bytes]
What are you trying to do here? -I0.00001 is ~1 meter on a 10^5 times 10^5 grid. You most certainly will run out of memory on that.
anbj
commented
2 years ago
xyz2grd (gmtapi_alloc_grid): Could not reallocate memory [37.25 Gb, 10000200001 items of 4 bytes]What are you trying to do here? -I0.00001 is ~1 meter on a 10^5 times 10^5 grid. You most certainly will run out of memory on that.
Yes, but I was deliberately trying to run out of memory to see if I could provoke a similar crash as in 1st post. Both cases run out of memory - one crashes, the other exits gracefully.
anbj
commented
2 years ago Memory is surely the culprit. Your first post had
xyz2grd (GMT_xyz2grd): Could not reallocate memory [9.31 Gb, 2498741960 items of 4 bytes]
Ah, and the last one too
xyz2grd (gmtapi_alloc_grid): Could not reallocate memory [37.25 Gb, 10000200001 items of 4 bytes]
But why the crash in the xyz2grd (GMT_xyz2grd): Could not reallocate memory [9.31 Gb, 2498741960 items of 4 bytes] case and not in the other? Both 6.3.0.
Why doesn't gmt exit after that message in both cases?
PaulWessel
commented
2 years ago It is hard to know without recreating. Since we create the grid up front before reading you would think this will be caught immediately. But perhaps in 6.3.0 we did not exit at the right place. We have made many changes to error exits recently 6.4.0 but that is not helping you here - and I dont know if it necessarily would. Also, I would recommend against using the $(gmt info) with that huge files since you are taxing things doubly even though gmtinfo does not allocate any memorhy. Does it crash if you get -R first then run xyz2grd.
anbj
commented
2 years ago Thanks for the tip @PaulWessel, and @joa-quim. I use $() for scripting (i.e. execute and leave).
I'm sorry I can't test this on git head. If it's interesting I can supply a tar.gz that is 112M (679M uncompressed). But you probably won't have the memory issue, so not sure if it's any point in doing that.
It crashes if I supply -R as well:
$ gmt xyz2grd -I1 60062_L05_32631_DTM_18_S02_20180814_1M.XYZ -R457048/532265/8046145/8079364 -G60062_L05_32631_DTM_18_S02_20180814_1M.nc
xyz2grd (GMT_xyz2grd): Could not reallocate memory [9.31 Gb, 2498741960 items of 4 bytes]
ERROR: Caught signal number 11 (Segmentation fault) at
/lib64/libgmt.so.6(GMT_xyz2grd+0x22b6)[0x7fecfa5c2746]
[0x253ba8534]
Stack backtrace:
/lib64/libgmt.so.6(sig_handler_unix+0xf1)[0x7fecfa367b11]
/lib64/libpthread.so.0(+0x12ce0)[0x7fecf5d1fce0]
/lib64/libgmt.so.6(GMT_xyz2grd+0x22b6)[0x7fecfa5c2746]
/lib64/libgmt.so.6(GMT_Call_Module+0x34f)[0x7fecfa28b93f]
gmt(main+0x27b)[0x56428961c57b]
/lib64/libc.so.6(__libc_start_main+0xf3)[0x7fecf5982ca3]
gmt(_start+0x2e)[0x56428961d5de]If I change -I1 to -I10 (i.e. no memory shortage), it runs fine:
$ gmt xyz2grd -I10 60062_L05_32631_DTM_18_S02_20180814_1M.XYZ -R457048/532265/8046145/8079364 -G60062_L05_32631_DTM_18_S02_20180814_1M.nc
xyz2grd [WARNING]: (x_max-x_min) must equal (NX + eps) * x_inc), where NX is an integer and |eps| <= 0.0001.
xyz2grd [WARNING]: (y_max-y_min) must equal (NY + eps) * y_inc), where NY is an integer and |eps| <= 0.0001.
xyz2grd (gmtapi_init_grdheader): Please select compatible -R and -I valuesI'll try to convince someone to install latest gmt and see if the problem persists.
anbj
commented
2 years ago I just installed 6.3.0 on a Windows machine (with not enough memory), and tried the same thing. No crash:
gmt xyz2grd -I1 -R457048/532265/8046145/8079364 60062_L05_32631_DTM_18_S02_20180814_1M.XYZ -Gtest.nc
xyz2grd (xyz2grd.c:587(GMT_xyz2grd)): Could not reallocate memory [9.31 Gb, 2498741960 items of 4 bytes]
xyz2grd (xyz2grd.c:587(GMT_xyz2grd)): gmt_M_memory [realloc] calledgmt --version
6.3.0The only test will be when someone (you or others) try master on that Linux machine. Or when 6.4.0 is released.
anbj
commented
2 years ago Confirmed; bug and crash is gone after installing latest:
$ gmt --version
6.4.0_fd7ec50_2022.05.31$ bash skript.sh
gmt [DEBUG]: GMT_Create_Session: Terminal width = 237
gmt [DEBUG]: Obtained the ppid from parent: 8994
gmt [DEBUG]: Enter: gmtinit_new_GMT_ctrl
gmt [DEBUG]: GMT->session.SHAREDIR = /home/anbj@od.local/gmt/build/share
gmt [DEBUG]: GMT->session.HOMEDIR = /home/anbj@od.local
gmt [DEBUG]: GMT->session.USERDIR = /home/anbj@od.local/.gmt [created]
gmt [DEBUG]: GMT->session.CACHEDIR = /home/anbj@od.local/.gmt/cache [created]
gmt [DEBUG]: GMT: 0. Will try to find subdir=postscriptlight stem = PSL_custom_fonts suffix=.txt
gmt [DEBUG]: GMT: 1. gmt_getsharepath trying current dir
gmt [DEBUG]: GMT: 2. gmt_getsharepath trying USERDIR /home/anbj@od.local/.gmt
gmt [DEBUG]: GMT: 3. gmt_getsharepath trying USERDIR subdir /home/anbj@od.local/.gmt/postscriptlight
gmt [DEBUG]: GMT: 4. gmt_getsharepath trying SHAREDIR subdir /home/anbj@od.local/gmt/build/share/postscriptlight
gmt [DEBUG]: GMT: 5. gmt_getsharepath trying SHAREDIR /home/anbj@od.local/gmt/build/share
gmt [DEBUG]: GMT: 6. gmt_getsharepath failed
gmt [DEBUG]: Map distance calculation will be Cartesian
gmt [DEBUG]: Exit: gmtinit_new_GMT_ctrl
gmt [DEBUG]: Enter: New_PSL_Ctrl
gmt [DEBUG]: Exit: New_PSL_Ctrl
gmt [DEBUG]: Enter: gmt_manage_workflow
gmt [DEBUG]: Exit : gmt_manage_workflow
gmt [DEBUG]: Enter: PSL_beginsession
gmt [DEBUG]: Exit : PSL_beginsession
gmt [DEBUG]: Enter: PSL_setdefaults
gmt [DEBUG]: Exit : PSL_setdefaults
gmt [DEBUG]: Enter: gmtlib_io_init
gmt [DEBUG]: Exit : gmtlib_io_init
gmt [DEBUG]: Enter: gmt_hash_init
gmt [DEBUG]: Exit: gmt_hash_init
gmt [DEBUG]: Enter: gmt_hash_init
gmt [DEBUG]: Exit: gmt_hash_init
gmt [DEBUG]: Enter: gmt_reload_settings
gmt [DEBUG]: The PROJ_GEODESIC set to Vincenty
gmt [DEBUG]: Look for file /home/anbj@od.local/gmt.conf
gmt [DEBUG]: Look for file /home/anbj@od.local/.gmt/gmt.conf
gmt [DEBUG]: Look for file /home/anbj@od.local/.gmt/server/gmt.conf
gmt [DEBUG]: Look for file /home/anbj@od.local/.gmt/cache/gmt.conf
gmt [DEBUG]: Could not find file gmt.conf
gmt [DEBUG]: Exit: gmt_reload_settings
gmt [DEBUG]: Enter: gmtlib_plot_C_format
gmt [DEBUG]: Exit: gmtlib_plot_C_format
gmt [DEBUG]: Enter: gmtinit_get_history
gmt [DEBUG]: Initialize FFTW with 2 threads.
gmt [DEBUG]: GMT_Create_Session initialized GMT structure
gmt [DEBUG]: Loading core GMT shared library: libgmt.so
gmt [DEBUG]: Shared Library # 0 (core). Path = libgmt.so
gmt [DEBUG]: Loading GMT plugins from: /home/anbj@od.local/gmt/build/lib64/gmt/plugins
gmt [DEBUG]: Shared Library # 1 (supplements). Path = /home/anbj@od.local/gmt/build/lib64/gmt/plugins/supplements.so
gmt [DEBUG]: Revised options: -I1 -R478888/544024/8035671/8067912 60062_L08_32631_DTM_18_S07_20180818_1M.XYZ -G60062_L08_32631_DTM_18_S07_20180818_1M.nc -Vd
xyz2grd [DEBUG]: History: Process -R478888/544024/8035671/8067912
xyz2grd [DEBUG]: Look for file 478888/544024/8035671/8067912 in /home/anbj@od.local/.gmt
xyz2grd [DEBUG]: Look for file 478888/544024/8035671/8067912 in /home/anbj@od.local/.gmt/cache
xyz2grd [DEBUG]: Look for file 478888/544024/8035671/8067912 in /home/anbj@od.local/.gmt/server
xyz2grd [DEBUG]: Got regular w/e/s/n for region (478888/544024/8035671/8067912)
xyz2grd [INFORMATION]: Processing input table data
xyz2grd [INFORMATION]: Cartesian input grid
xyz2grd [DEBUG]: Chosen boundary condition for all edges: natural
xyz2grd [INFORMATION]: Cartesian input grid
xyz2grd [DEBUG]: Object ID 0 : Registered Grid Memory Reference 1cc1690 as an Input resource with geometry Surface [n_objects = 1]
xyz2grd [DEBUG]: Successfully created a new Grid container
xyz2grd [INFORMATION]: n_columns = 65137 n_rows = 32242 nm = 2100147154 size = 2100147154
xyz2grd (GMT_xyz2grd): Could not reallocate memory [7.82 Gb, 2100147154 items of 4 bytes]
xyz2grd [ERROR]: Unable to allocate flag memory
xyz2grd [DEBUG]: gmtlib_garbage_collection: Destroying object: C=0 A=0 ID=0 W=Input F=Grid M=Memory Reference S=Unused P=1cc1690 N=(null)
xyz2grd [DEBUG]: GMTAPI_Garbage_Collection freed 1 memory objects
xyz2grd [DEBUG]: gmtlib_unregister_io: Unregistering object no 0 [n_objects = 0]
gmt [DEBUG]: Entering GMT_Destroy_SessionClosing this as it no longer crashes my use case.
This crash in on version 6.3.0. I'm not able to install a newer version on the system. Suspected cause is not enough memory. I installed 6.3.0 on a system with sufficient memory, and I do not experience any crash. My input file is several hundred megabytes so not sure how I can share it, and if it is necessary.
Equivalent smaller input files that do fit memory is gridded without problems.
Command:
Result (with
-Vd):Are there any smart methods to make big files fit into memory? Remove decimals etc.?