Closed stefanomattia closed 6 years ago
nehaljwani
commented
6 years ago @stefanomattia Could you please share the file: S5P_OFFL_L2NO2__20171129T010127_20171129T024256_00662_01_001107_20171202T195637.nc or a similar one which one can use to reproduce the issue?
stefanomattia
commented
6 years ago You can download file test.nc showing the same issue here.
With Anaconda netCDF4 1.2.4:
(netcdf4_124) $ python
Python 3.6.3 |Anaconda, Inc.| (default, Dec 5 2017, 17:30:25)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import netCDF4 as nc
>>> nc.__version__
'1.2.4'
>>> nc.Dataset('7845.nc')
<class 'netCDF4._netCDF4.Dataset'>
root group (NETCDF4 data model, file format HDF5):
Conventions: CF-1.7
[...]With latest Anaconda netCDF4 1.3.1:
$ source activate netcdf4_latest
(netcdf4_latest) $ python
Python 3.6.3 |Anaconda, Inc.| (default, Dec 5 2017, 17:30:25)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import netCDF4 as nc
>>> nc.__version__
'1.3.1'
>>> nc.Dataset('7845.nc')
Abort trap: 6
mingwandroid
commented
6 years ago Thanks, I know what the problem is and have submitted a PR to the upstream project here
We can roll out new builds with this patch tomorrow though, I expect.
For full details, it's a run-of-the-mill buffer overrun. The reason it gets caught by our build and not by other builds is because we pass the -fstack-protector-strong CFLAG to all of our builds. This means that as soon as a buffer overrun is detected the program aborts. This is better than corrupting your data and/or the program crashing at some other point with not way to track it back to the actual bug.
Thanks for the bug report. I'll ping you when we roll out the new builds.
nehaljwani
commented
6 years ago @stefanomattia New builds with fix from @mingwandroid have been uploaded to repo.continuum.io . Could you try them out and let us know if your issue is resolved?
stefanomattia
commented
6 years ago @nehaljwani I can't see the new build on https://repo.continuum.io/pkgs/main/osx-64/ and conda update netcdf4 reports everything is up-to-date. Could you point me to the exact location of the new build?
nehaljwani
commented
6 years ago @stefanomattia Try updating libnetcdf
stefanomattia
commented
6 years ago @nehaljwani I'm sorry but I can't seem to be able to find the updated package. Within my netcdf4-1.3.1 virtual environment, trying to update the netCDF4 package returns "all requested packages already installed":
$ source activate netcdf4_1.3.1
netcdf4_1.3.1$ conda list netcdf4
# packages in environment at /Users/stefano/anaconda3/envs/netcdf4_1.3.1:
#
# Name Version Build Channel
netcdf4 1.3.1 py36hdbd7e57_2
netcdf4_1.3.1 $ conda update netcdf4
Solving environment: done
# All requested packages already installed.Trying to search for different versions, I also see the latest package as the one that's already installed on my system.
netcdf4_1.3.1$ conda search netcdf4
Loading channels: done
[...]
netcdf4 1.2.4 np113py36_1 defaults
netcdf4 1.3.1 py27h6186eb7_2 defaults
netcdf4 1.3.1 py27h8161f18_2 defaults
netcdf4 1.3.1 py35h668d88a_2 defaults
netcdf4 1.3.1 py35ha769655_2 defaults
netcdf4 1.3.1 py36h8ec8d5d_2 defaults
netcdf4 1.3.1 py36hdbd7e57_2 defaults These are also the very same packages on https://repo.continuum.io/pkgs/main/osx-64/ which appear not to have been updated since November 2017:
netcdf4-1.3.1-py27h6186eb7_2.tar.bz2 | 660 KB | 2017-11-28 16:32:33 UTC | 7f088446a237b87a06f14179884c2b6f
-- | -- | -- | --
netcdf4-1.3.1-py27h8161f18_2.tar.bz2 | 660 KB | 2017-11-28 16:35:35 UTC | 6f876126c41f0d3ec130146475c7210e
netcdf4-1.3.1-py35h668d88a_2.tar.bz2 | 638 KB | 2017-11-28 16:32:45 UTC | db06a2ca19ea60e5bf773a787aa1a797
netcdf4-1.3.1-py35ha769655_2.tar.bz2 | 636 KB | 2017-11-28 16:35:34 UTC | be581f31d1a00aae4178cb1dd1128e62
netcdf4-1.3.1-py36h8ec8d5d_2.tar.bz2 | 668 KB | 2017-11-28 16:32:34 UTC | 5f660adc55147b54bb6217e74b581a5f
netcdf4-1.3.1-py36hdbd7e57_2.tar.bz2 | 669 KB | 2017-11-28 16:35:41 UTC | 39527841c534f1f7201501d00baa7f15Is there any manual process involved, such as configuring experimental channels, or downloading packages from a specific location and installing them by hand?
nehaljwani
commented
6 years ago @stefanomattia I asked you to update the package libnetcdf but you keep trying to update netcdf.
Type the following:
conda update libnetcdfThese was updated in January and netcdf4 depends on it:
libnetcdf-4.5.0-h42fd751_7.tar.bz2 | 1.1 MB | 2018-01-03 17:52:14 UTC | 39b50ff9f27dd5f6a8f4ef884a91ced0@nehaljwani I was under the impression that a correspondent netcdf4 package update would also be released. Indeed, if I just try to update the libnetcdf (which is a dependencies of netcdf4), nothing happens:
$ conda update libnetcdf
Solving environment: done
# All requested packages already installed.If were to force the install of version 4.5.0, conda would remove the netcdf4 package:
$ conda install libnetcdf=4.5.0
Solving environment: done
## Package Plan ##
environment location: /Users/stefano/anaconda3/envs/netcdf_latest
added / updated specs:
- libnetcdf=4.5.0
The following packages will be downloaded:
package | build
---------------------------|-----------------
bzip2-1.0.6 | hd86a083_4 148 KB
libnetcdf-4.5.0 | h42fd751_7 1.1 MB
------------------------------------------------------------
Total: 1.3 MB
The following NEW packages will be INSTALLED:
bzip2: 1.0.6-hd86a083_4
The following packages will be REMOVED:
netcdf4: 1.3.1-py36hdbd7e57_2 anaconda
The following packages will be UPDATED:
libnetcdf: 4.4.1.1-hdf03fd4_8 --> 4.5.0-h42fd751_7
Proceed ([y]/n)? nI suppose an updated version of netcdf4 is also needed?
nehaljwani
commented
6 years ago Oh, my bad. I thought the pin for netcdf4 was libnetcdf 4.x, but it is libnetcdf 4.4.x
We'll need a new release of netcdf4 it seems. Thanks for the update @stefanomattia
nehaljwani
commented
6 years ago @stefanomattia Newer builds of netcdf4 are available now!
stefanomattia
commented
6 years ago Thanks for the new build. I could install it by using the following command:
conda install netcdf4=1.3.1=py36he3ffdca_2A simple conda update netcdf4 would return a All requested packages already installed message.
I don't have any file for testing the closure of the original issue at hand, will do it on Monday.
One thing I've noticed though, is that the package version is still 1.2.4 as shown by the __version__ attribute:
$ conda list netCDF4
# packages in environment at /Users/stefano/anaconda3/envs/netcdf_latest:
#
# Name Version Build Channel
netcdf4 1.3.1 py36he3ffdca_2
$ ipython
Python 3.6.1 |Anaconda custom (64-bit)| (default, May 11 2017, 13:04:09)
Type "copyright", "credits" or "license" for more information.
IPython 5.3.0 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: import netCDF4 as nc
In [2]: nc.__version__
Out[2]: '1.2.4'Is that a detail that needs to be fixed?
nehaljwani
commented
6 years ago You could've instead done conda update libnetcdf4 which would have update it to 4.5.0 and thereby switched the build of netcdf4. Anyway, I am not able to reproduce the version issue:
(test7845) mac:~ nwani$ conda list netCDF4
# packages in environment at /Users/nwani/m3/envs/test7845:
#
# Name Version Build Channel
netcdf4 1.3.1 py36he3ffdca_2
(test7845) mac:~ nwani$ ipython
Python 3.6.4 |Anaconda, Inc.| (default, Jan 16 2018, 12:04:33)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.2.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import netCDF4 as nc
In [2]: nc.__version__
Out[2]: '1.3.1'
In [3]: exit()This is super weird: when importing netCDF4 in with ipython I get 1.2.4, whereas importing it in the standard python shell I get 1.3.1:
$ ipython
Python 3.6.1 |Anaconda custom (64-bit)| (default, May 11 2017, 13:04:09)
Type "copyright", "credits" or "license" for more information.
IPython 5.3.0 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: import netCDF4 as nc
In [2]: nc.__version__
Out[2]: '1.2.4'
In [3]:
Do you really want to exit ([y]/n)? y
$ python
Python 3.6.3 |Anaconda, Inc.| (default, Dec 5 2017, 17:30:25)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import netCDF4 as nc
>>> nc.__version__
'1.3.1'I then upgraded ipython to the latest version (6.2.1) and now I get 1.3.1 on ipython too.
$ ipython
Python 3.6.3 |Anaconda, Inc.| (default, Dec 5 2017, 17:30:25)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.2.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import netCDF4 as nc
In [2]: nc.__version__
Out[2]: '1.3.1'I suspect the old ipython was somehow not properly linked the activated virtual environment?
Anyway, I will let you know on Monday the status w.r.t the original issue. Thanks!
stefanomattia
commented
6 years ago I can confirm the original issue has been solved with libnetcdf 4.5.0 and netCDF4 1.3.1 (py36he3ffdca_2).
$ ipython
Python 3.6.3 |Anaconda, Inc.| (default, Dec 5 2017, 17:30:25)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.2.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import netCDF4 as nc
In [2]: nc.__version__
Out[2]: '1.3.1'
In [3]: nc.Dataset("S5P_OFFL_L2__NO2____20171129T010127_20171129T024256_00662_01_001107_20171202T195637.nc")
Out[3]:
<class 'netCDF4._netCDF4.Dataset'>
root group (NETCDF4 data model, file format HDF5):
[...]
I have been happily using netCDF4 1.2.4 (built on libnetcdf 4.4.1) on OS X to work on my products:
I recently upgraded the netCDF4 conda package to 1.3.1 (built on libnetcdf 4.4.1.1) and I can't open my products anymore:
The same happens with conda-forge's netCDF4 1.3.1 (built on libnetcdf 4.5.0) package:
Opening the same file on Linux, with any of those netCDF4 versions, does not raise any error: the file can be properly opened and read.