catch cdunif core dumps

durack1 commented 5 years ago

The following will reproduce the issue:

(cdat81py2) duro@ocean:[~]:[12177]> ipython
Python 2.7.15 | packaged by conda-forge | (default, Feb 28 2019, 04:00:11) 
Type "copyright", "credits" or "license" for more information.

IPython 5.8.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import cdms2

In [2]: file = 'tmp/file.nc'

In [3]: fileHandle = cdms2.open(file,'rw+')
CDMS I/O error: Looking up file 0
cdunif: Unknown Error
Segmentation fault (core dumped)

jypeter commented 5 years ago

@durack1 I think this is a duplicate of #77

(cdatm_py3) jypeter@obelix2 - ...jypeter - 45 >python
Python 3.6.7 | packaged by conda-forge | (default, Feb 28 2019, 09:07:38)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cdms2
>>> f = cdms2.open('tmp/file.nc', 'rw+')
CDMS I/O error: Looking up file 796095077
cdunif: Unknown Error
Segmentation fault (core dumped)

(cdatm_py3) jypeter@obelix2 - ...jypeter - 46 >python
Python 3.6.7 | packaged by conda-forge | (default, Feb 28 2019, 09:07:38)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cdms2
>>> f = cdms2.open('tmp/file.nc', 'r+')
CDMS system error: No such file or directory
CDMS I/O error: Opening file /home/users/jypeter/tmp/file.nc
Traceback (most recent call last):
  File "/home/share/unix_files/cdat/miniconda3/envs/cdatm_py3/lib/python3.6/site-packages/cdms2/dataset.py", line 1281, in __init__
    _fileobj_ = Cdunif.CdunifFile(path, mode)
OSError: No error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/share/unix_files/cdat/miniconda3/envs/cdatm_py3/lib/python3.6/site-packages/cdms2/dataset.py", line 499, in openDataset
    return CdmsFile(path, mode, mpiBarrier=CdMpi)
  File "/home/share/unix_files/cdat/miniconda3/envs/cdatm_py3/lib/python3.6/site-packages/cdms2/dataset.py", line 1283, in __init__
    raise CDMSError('Cannot open file %s (%s)' % (path, err))
cdms2.error.CDMSError: Cannot open file /home/users/jypeter/tmp/file.nc (No error)
>>>

durack1 commented 5 years ago

@dnadeau4 @jypeter is right, when this bug is fixed you can close out both this #335 and #77 as dupes

dnadeau4 commented 5 years ago

https://github.com/CDAT/cdms/blob/master/Src/Cdunifmodule.c#L1458-L1462

durack1 commented 5 years ago

@dnadeau4 so the link above means that rw+ is an accepted mode, whereas this causes a segfault?

dnadeau4 commented 5 years ago

This will destroy the self object and return Null. Either destroying the self object causes the segfault or the calling function. Actually "rw+" is not acceptable, only "r,w,a,r+,w+,a+".

durack1 commented 5 years ago

It would help that a user that provides the option rw+ then gets a warning to say that it's defaulting to w+ and the process continues. I think defaulting to w+ is a safe default to me.

Not sure what @pochedls or @doutriaux1 think

pochedls commented 5 years ago

@durack1 - I don't remember. w+ doesn't clobber the file does it? I think this is what I would want assuming it doesn't trash what is already in the file if it exists.

durack1 commented 5 years ago

@pochedls good point, I am not 100% certain that w+ will not clobber files, I think you're right, any default mode should leave a file (and data) intact, and just allow editing by default

lee1043 commented 5 years ago

I think I am hitting same issue after installing recent nightly version. I cannot even import cdms2. Any progress?

(pmp_nightly_20190520) -bash-4.1$ p 
Python 3.6.7 | packaged by conda-forge | (default, Feb 28 2019, 09:07:38) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cdms2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/export/lee1043/anaconda2/envs/pmp_nightly_20190520/lib/python3.6/site-packages/cdms2/__init__.py", line 8, in <module>
    from . import dataset
  File "/export/lee1043/anaconda2/envs/pmp_nightly_20190520/lib/python3.6/site-packages/cdms2/dataset.py", line 8, in <module>
    from . import Cdunif
ImportError: libcdms.so: cannot open shared object file: No such file or directory
>>>

(pmp_nightly_20190520) -bash-4.1$ conda list cdms2
# packages in environment at /export/lee1043/anaconda2/envs/pmp_nightly_20190520:
#
# Name                    Version                   Build  Channel
cdms2                     3.1.3                    pypi_0    pypi

doutriaux1 commented 5 years ago

conda-forge updated netcdf and it broke cdms2... Please use:

conda install -f -c cdat/label/nightly -c conda-forge libnetcdf=4.6.2

lee1043 commented 5 years ago

@doutriaux1 unfortunately it is not solving my issue...

dnadeau4 commented 5 years ago

@lee1043 this is a different issue, but run to for an update of libcdms.

conda install -f -c cdat/label/nightly -c conda-forge libcdms libnetcdf=4.6.2

khouakhi commented 4 years ago

cannot import name 'Cdunif'

import cdms2

ImportError Traceback (most recent call last)

in () ----> 1 import cdms2 /usr/local/lib/python3.7/site-packages/cdms2/__init__.py in () 6 from .error import CDMSError # noqa 7 from lazy_object_proxy import Proxy ----> 8 from . import dataset 9 from . import selectors 10 from . import avariable /usr/local/lib/python3.7/site-packages/cdms2/dataset.py in () 6 from .error import CDMSError 7 import sys ----> 8 from . import Cdunif 9 import numpy 10 from . import cdmsNode ImportError: cannot import name 'Cdunif'

CDAT / cdms

catch cdunif core dumps #336