search

GenericMappingTools / pygmt

A Python interface for the Generic Mapping Tools.
https://www.pygmt.org
BSD 3-Clause "New" or "Revised" License
760 stars 218 forks source link

call_module() for convert -S kills my jupyter kernel #1609

Closed tomeracles closed 3 years ago

tomeracles commented 3 years ago

Description of the problem

When trying to call the module gmtconvert with the -S flag in a jupyter notebook, the kernel dies with no error message. I'm doing this on a file created with ogr2ogr from a kml file. This code worked fine using pygmt 0.4.0, but breaks now that I have updated to 0.5.0!

Full code that generated the error

Here is a section of my input file faults.txt

# @VGMT1.0
# @R32.3110698139/34.5801629672/34.5708028839/35.6872439877               
# @Je4326
# @Jp"+proj=longlat +datum=WGS84 +no_defs "
# @Jw"GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]]"
# @GMULTILINESTRING
# @NName|description|timestamp|begin|end|altitudeMode|tessellate|extrude|visibility|drawOrder|icon|snippet
# @Tstring|string|datetime|datetime|datetime|string|integer|integer|integer|integer|string|string
# FEATURE_DATA
>
# @D"Inferred Fault"||||||-1|0|-1|||
33.0102099338746 34.571560739077 0
32.9945552439243 34.5736013843409 0
32.9843003168117 34.574612774927 0
32.9765740213708 34.5736984567195 0
32.9743256717152 34.5738400391915 0
32.9696058965068 34.5737893558167 0
32.9646609606958 34.5738105695601 0
32.9628679328768 34.5743150493937 0
32.959219971995 34.5744762104806 0
32.9559168272917 34.5744660441812 0
32.9512179473419 34.5742523320563 0
32.9450426820596 34.5736950965054 0
32.9404869727102 34.5726333367537 0
32.936443930804 34.5708028838632 0
>
# @D"Definite Fault"||||||-1|0|-1|||
32.9990596137843 34.7937354852047 0
32.9908891450293 34.7991886286412 0
32.9848650692329 34.803762275527 0
32.9805609668175 34.8071072422452 0
32.9762643069475 34.8092121192229 0
32.9726022634527 34.8125591793077 0
32.9678874074511 34.8167084305494 0
32.9657090145242 34.818666934113 0
32.9615702002069 34.822315614569 0
32.9585199968519 34.8250740300371 0
32.9551674809636 34.8283485197953 0
32.9530042912567 34.8301963307876 0
32.9530038538104 34.8301963306186 0

Code run in jupyter notebook:

import pygmt
infile = 'faults.txt'
tmpfile = pygmt.helpers.GMTTempFile()
with pygmt.clib.Session() as session:
    session.call_module('convert', f'{infile} -SName="Inferred Fault" -Vd ->{tmpfile.name}')

I was able to run convert when I didn't try to use -S.

(I have also tried using -SName=Inferred in case there was a problem with the quote marks or space but same thing happens).

Full error message

No error message given except Jupyter Notebooks tells me "The kernel appears to have died. It will restart automatically."

Here is the debug output from using -Vd

gmtconvert [DEBUG]: Found readable file faults.txt
gmtconvert [DEBUG]: Replace file faults.txt with path faults.txt
gmtconvert [DEBUG]: Replace file faults.txt with faults.txt
gmtconvert [INFORMATION]: Processing input table data
gmtconvert [DEBUG]: gmtapi_init_import: Passed family = Data Table and geometry = Point
gmtconvert [DEBUG]: Found readable file faults.txt
gmtconvert [DEBUG]: Replace file faults.txt with path faults.txt
gmtconvert [DEBUG]: Found readable file faults.txt
gmtconvert [DEBUG]: Object ID 0 : Registered Data Table File faults.txt as an Input resource with geometry Point [n_objects = 1]
gmtconvert (gmtapi_init_import): tried to free unallocated memory
gmtconvert [DEBUG]: gmtapi_init_import: Added 1 new sources
gmtconvert [DEBUG]: GMT_Init_IO: Returned first Input object ID = 0
gmtconvert [DEBUG]: gmtapi_begin_io: Input resource access is now enabled [container]
gmtconvert [DEBUG]: gmtapi_import_dataset: Passed ID = -1 and mode = 0
gmtconvert [INFORMATION]: Reading Data Table from File faults.txt
gmtconvert [DEBUG]: Found readable file faults.txt
gmtconvert [DEBUG]: Calling nc_open on faults.txt, ncid = 0, err = -51
gmtconvert [DEBUG]: Found readable file faults.txt
gmtconvert [DEBUG]: Source col types: (Number,Number,Number)
gmtconvert [DEBUG]: ASCII source scanned: Numerical columns: 3, Trailing text: N, Record type: Numerical only
gmtconvert [DEBUG]: GMT memory: Initialize 3 temporary column double arrays, each of length : 0
gmtconvert [DEBUG]: GMT_End_IO: Input resource access is now disabled

System information

Please paste the output of python -c "import pygmt; pygmt.show_versions()":

PyGMT information:
  version: v0.5.0
System information:
  python: 3.8.10 | packaged by conda-forge | (default, May 10 2021, 22:58:09)  [Clang 11.1.0 ]
  executable: /Applications/anaconda3/envs/pygmt/bin/python
  machine: macOS-10.14.6-x86_64-i386-64bit
Dependency information:
  numpy: 1.19.1
  pandas: 1.1.1
  xarray: 0.16.0
  netCDF4: 1.5.7
  packaging: 20.4
  ghostscript: 9.54.0
  gmt: 6.2.0
GMT library information:
  binary dir: /Applications/anaconda3/envs/pygmt/bin
  cores: 8
  grid layout: rows
  library path: /Applications/anaconda3/envs/pygmt/lib/libgmt.dylib
  padding: 2
  plugin dir: /Applications/anaconda3/envs/pygmt/lib/gmt/plugins
  share dir: /Applications/anaconda3/envs/pygmt/share/gmt
  version: 6.2.0
welcome[bot] commented 3 years ago

👋 Thanks for opening your first issue here! Please make sure you filled out the template with as much detail as possible. You might also want to take a look at our contributing guidelines and code of conduct.

weiji14 commented 3 years ago

Hi @tomeracles! Thanks for your detailed bug report, and for trying out PyGMT! I don't think there were any major changes to the clib between PyGMT v0.4.1 and v0.5.0 after scanning through https://github.com/GenericMappingTools/pygmt/compare/v0.4.1...v0.5.0, other than some code formatting fixes. Also, I've tried running your example on try-gmt which uses PyGMT v0.5.0 on Linux, but the kernel doesn't crash.

The debug output is this:

gmtconvert [DEBUG]: Found readable file faults.txt
gmtconvert [DEBUG]: Replace file faults.txt with path faults.txt
gmtconvert [DEBUG]: Replace file faults.txt with faults.txt
gmtconvert [INFORMATION]: Processing input table data
gmtconvert [DEBUG]: gmtapi_init_import: Passed family = Data Table and geometry = Point
gmtconvert [DEBUG]: Found readable file faults.txt
gmtconvert [DEBUG]: Replace file faults.txt with path faults.txt
gmtconvert [DEBUG]: Found readable file faults.txt
gmtconvert [DEBUG]: Object ID 0 : Registered Data Table File faults.txt as an Input resource with geometry Point [n_objects = 1]
gmtconvert (gmtapi_init_import): tried to free unallocated memory
gmtconvert [DEBUG]: gmtapi_init_import: Added 1 new sources
gmtconvert [DEBUG]: GMT_Init_IO: Returned first Input object ID = 0
gmtconvert [DEBUG]: gmtapi_begin_io: Input resource access is now enabled [container]
gmtconvert [DEBUG]: gmtapi_import_dataset: Passed ID = -1 and mode = 0
gmtconvert [INFORMATION]: Reading Data Table from File faults.txt
gmtconvert [DEBUG]: Found readable file faults.txt
gmtconvert [DEBUG]: Calling nc_open on faults.txt, ncid = 0, err = -51
gmtconvert [DEBUG]: Found readable file faults.txt
gmtconvert [DEBUG]: Source col types: (Number,Number,Number)
gmtconvert [DEBUG]: ASCII source scanned: Numerical columns: 3, Trailing text: N, Record type: Numerical only
gmtconvert [DEBUG]: GMT memory: Initialize 3 temporary column double arrays, each of length : 0
gmtconvert [DEBUG]: GMT_End_IO: Input resource access is now disabled
gmtconvert [DEBUG]: Object ID 1 : Registered Data Table Memory Reference 56462f0b25d0 as an Input resource with geometry Line [n_objects = 2]
gmtconvert [DEBUG]: Successfully duplicated a Data Table
gmtconvert [DEBUG]: Object ID 2 : Registered Data Table File /tmp/pygmt-md54mg08.txt as an Output resource with geometry Line [n_objects = 3]
gmtconvert [DEBUG]: gmtapi_begin_io: Output resource access is now enabled [container]
gmtconvert [DEBUG]: gmtapi_export_dataset: Passed ID = 2 and mode = 0
gmtconvert [INFORMATION]: Write Data Table to file /tmp/pygmt-md54mg08.txt
gmtconvert (gmtlib_free_ogr): tried to free unallocated memory
gmtconvert (gmtlib_free_ogr): tried to free unallocated memory
gmtconvert [DEBUG]: GMT_End_IO: Output resource access is now disabled
gmtconvert [INFORMATION]: 1 tables concatenated, 27 records passed (input cols = 3; output cols = 3)
gmtconvert [INFORMATION]: Extracted 1 from a total of 2 segments
gmtconvert [DEBUG]: gmtlib_garbage_collection: Destroying object: C=0 A=1 ID=0 W=Input F=Data Table M=File S=Used P=56462ef12730 N=faults.txt
gmtconvert (gmt_free_table): tried to free unallocated memory
gmtconvert [DEBUG]: gmtlib_garbage_collection: Destroying object: C=0 A=0 ID=1 W=Input F=Data Table M=Memory Reference S=Unused P=56462f0b25d0 N=(null)
gmtconvert (gmt_free_table): tried to free unallocated memory
gmtconvert [DEBUG]: GMTAPI_Garbage_Collection freed 2 memory objects
gmtconvert [DEBUG]: gmtlib_unregister_io: Unregistering object no 0 [n_objects = 2]
gmtconvert [DEBUG]: gmtlib_unregister_io: Unregistering object no 1 [n_objects = 1]
gmtconvert [DEBUG]: gmtlib_unregister_io: Unregistering object no 2 [n_objects = 0]
gmtconvert (gmtlib_free_ogr): tried to free unallocated memory
gmtconvert (gmtlib_free_ogr): tried to free unallocated memory
gmtconvert [DEBUG]: GMT memory: Free 3 temporary column arrays, each of length : 2097152

The output of !cat {tmpfile.name} is:

>
33.0102099339   34.5715607391   0
32.9945552439   34.5736013843   0
32.9843003168   34.5746127749   0
32.9765740214   34.5736984567   0
32.9743256717   34.5738400392   0
32.9696058965   34.5737893558   0
32.9646609607   34.5738105696   0
32.9628679329   34.5743150494   0
32.959219972    34.5744762105   0
32.9559168273   34.5744660442   0
32.9512179473   34.5742523321   0
32.9450426821   34.5736950965   0
32.9404869727   34.5726333368   0
32.9364439308   34.5708028839   0

and the pygmt.show_versions() for reference is:

PyGMT information:
  version: v0.5.0
System information:
  python: 3.9.7 | packaged by conda-forge | (default, Sep 29 2021, 19:20:46)  [GCC 9.4.0]
  executable: /srv/conda/envs/notebook/bin/python
  machine: Linux-5.4.129+-x86_64-with-glibc2.27
Dependency information:
  numpy: 1.21.3
  pandas: 1.3.4
  xarray: 0.19.0
  netCDF4: 1.5.7
  packaging: 20.4
  ghostscript: 9.54.0
  gmt: 6.2.0
GMT library information:
  binary dir: /srv/conda/envs/notebook/bin
  cores: 6
  grid layout: rows
  library path: /srv/conda/envs/notebook/lib/libgmt.so
  padding: 2
  plugin dir: /srv/conda/envs/notebook/lib/gmt/plugins
  share dir: /srv/conda/envs/notebook/share/gmt
  version: 6.2.0

So I don't see anything suspicious really... Maybe @meghanrjones can doublecheck if this is an issue on macOS (we seem to be getting lots of them recently)? Otherwise, you could try running gmt clear all in your terminal and/or reinstalling to see if it fixes your issue:

gmt clear all
conda env remove -n pygmt
conda create --name pygmt --channel conda-forge pygmt jupyterlab
conda activate pygmt
jupyterlab --no-browser
maxrjones commented 3 years ago

I didn't check the output but the kernel didn't crash for me on MacOS with notebook 6.4.4.

weiji14 commented 3 years ago

Hmm, just to see if this is a problem on the PyGMT or GMT side, @tomeracles could you try running this in a jupyter code cell:

!gmt convert faults.txt -SName="Inferred Fault" -Vd > converted_faults.txt
tomeracles commented 3 years ago

Thanks very much for looking into this! Clearly there was some issue with my conda environment, as I reinstalled pygmt in a new environment and the issue has gone, so very sorry to bother you with that. However, I'm still curious. In the previous environment, as @weiji14 suggested I tried the command

!gmt convert faults.txt -SName="Inferred Fault" -Vd > converted_faults.txt

in a jupyter cell and got the following output:

gmt [DEBUG]: Obtained the ppid from parent: 80990
gmt [DEBUG]: Enter: gmtinit_new_GMT_ctrl
gmt [DEBUG]: GMT->session.SHAREDIR = /Applications/GMT-6.2.0.app/Contents/Resources/share
gmt [DEBUG]: GMT->session.HOMEDIR = /Users/tmerry
gmt [DEBUG]: GMT->session.USERDIR = /Users/tmerry/.gmt [created]
gmt [DEBUG]: GMT->session.CACHEDIR = /Users/tmerry/.gmt/cache [created]
gmt [DEBUG]: GMT: 0. Will try to find subdir=postscriptlight stem = PSL_custom_fonts suffix=.txt
gmt [DEBUG]: GMT: 1. gmt_getsharepath trying current dir
gmt [DEBUG]: GMT: 2. gmt_getsharepath trying USERDIR /Users/tmerry/.gmt
gmt [DEBUG]: Map distance calculation will be Cartesian
gmt [DEBUG]: Exit:  gmtinit_new_GMT_ctrl
gmt [DEBUG]: Enter: New_PSL_Ctrl
gmt [DEBUG]: Exit:  New_PSL_Ctrl
gmt [DEBUG]: Enter: gmt_manage_workflow
gmt [DEBUG]: Exit : gmt_manage_workflow
gmt [DEBUG]: Enter: PSL_beginsession
gmt [DEBUG]: Exit : PSL_beginsession
gmt [DEBUG]: Enter: PSL_setdefaults
gmt [DEBUG]: Exit : PSL_setdefaults
gmt [DEBUG]: Enter: gmtlib_io_init
gmt [DEBUG]: Exit : gmtlib_io_init
gmt [DEBUG]: Enter: gmt_hash_init
gmt [DEBUG]: Exit:  gmt_hash_init
gmt [DEBUG]: Enter: gmt_hash_init
gmt [DEBUG]: Exit:  gmt_hash_init
gmt [DEBUG]: Enter: gmt_reload_settings
gmt [DEBUG]: The PROJ_GEODESIC set to Vincenty
gmt [DEBUG]: Look for file /Users/tmerry/gmt.conf
gmt [DEBUG]: Look for file /Users/tmerry/.gmt/gmt.conf
gmt [DEBUG]: Look for file /Users/tmerry/.gmt/server/gmt.conf
gmt [DEBUG]: Look for file /Users/tmerry/.gmt/cache/gmt.conf
gmt [DEBUG]: Could not find file gmt.conf
gmt [DEBUG]: Exit:  gmt_reload_settings
gmt [DEBUG]: Enter: gmtlib_plot_C_format
gmt [DEBUG]: Exit:  gmtlib_plot_C_format
gmt [DEBUG]: Enter: gmtinit_get_history
gmt [DEBUG]: Initialize FFTW with 8 threads.
gmt [DEBUG]: GMT_Create_Session initialized GMT structure
gmt [DEBUG]: Loading core GMT shared library: libgmt.dylib
gmt [DEBUG]: Shared Library # 0 (core). Path = libgmt.dylib
gmt [DEBUG]: Loading GMT plugins from: /Applications/GMT-6.2.0.app/Contents/Resources/lib/gmt/plugins
gmt [DEBUG]: Shared Library # 1 (supplements). Path = /Applications/GMT-6.2.0.app/Contents/Resources/lib/gmt/plugins/supplements.so
gmt [DEBUG]: Revised options: faults.txt -S"Name=Inferred Fault" -Vd
gmtconvert [DEBUG]: Found readable file faults.txt
gmtconvert [DEBUG]: Replace file faults.txt with path faults.txt
gmtconvert [DEBUG]: Replace file faults.txt with faults.txt
gmtconvert [INFORMATION]: Processing input table data
gmtconvert [DEBUG]: gmtapi_init_import: Passed family = Data Table and geometry = Point
gmtconvert [DEBUG]: Found readable file faults.txt
gmtconvert [DEBUG]: Replace file faults.txt with path faults.txt
gmtconvert [DEBUG]: Found readable file faults.txt
gmtconvert [DEBUG]: Object ID 0 : Registered Data Table File faults.txt as an Input resource with geometry Point [n_objects = 1]
gmtconvert (gmtapi_init_import): tried to free unallocated memory
gmtconvert [DEBUG]: gmtapi_init_import: Added 1 new sources
gmtconvert [DEBUG]: GMT_Init_IO: Returned first Input object ID = 0
gmtconvert [DEBUG]: gmtapi_begin_io: Input resource access is now enabled [container]
gmtconvert [DEBUG]: gmtapi_import_dataset: Passed ID = -1 and mode = 0
gmtconvert [INFORMATION]: Reading Data Table from File faults.txt
gmtconvert [DEBUG]: Found readable file faults.txt
gmtconvert [DEBUG]: Calling nc_open on faults.txt, ncid = 0, err = -51
gmtconvert [DEBUG]: Found readable file faults.txt
gmtconvert [DEBUG]: Source col types: (Number,Number,Number)
gmtconvert [DEBUG]: ASCII source scanned: Numerical columns: 3, Trailing text: N, Record type: Numerical only
gmtconvert [DEBUG]: GMT memory: Initialize 3 temporary column double arrays, each of length : 0
gmtconvert [DEBUG]: GMT_End_IO: Input resource access is now disabled
/bin/bash: line 1: 80991 Abort trap: 6           gmt convert faults.txt -SName="Inferred Fault" -Vd > converted_faults.txt

and an empty file converted_faults.txt was made. Meanwhile, running the same gmt command in the terminal (with the same conda environment activated) worked perfectly well.

Interestingly, when I type which gmt in a terminal with my old pygmt environment activated, I get /Applications/anaconda3/envs/pygmt/bin/gmt. However, in the jupyter notebook in the same environment, when I type !which gmt I get /Users/tmerry/this_gmt/bin/gmt. I don't expect anyone to help me with this but I do wonder what I've done to make that happen.

Thanks again anyway and sorry to bring my own system issues here!

maxrjones commented 3 years ago

Glad it's working now, @tomeracles! Without looking in too much detail, I suspect that there's some confusion between the GMT 6.2.0 bundle (/Applications/GMT-6.2.0.app/) and the conda-installed GMT. My recommendation would be to uninstall the bundle and just use conda environments for GMT moving forward, since users on the forum have occasionally brought up issues related to conflicts between the bundle and conda installations. In addition, there's a known issue with the bundle not working for the wrappers.