Issue passing python3 filenames to netcdf4

[weather01089]

netcdf4 errors out passing filename from python script. variable is filename4, 16 members, fails on all. a command line ncdump is fine this is the code: print(os.path.isfile(filename4[0])) print(os.path.isfile(os.path.realpath(filename4[0]))) print(type(filename4[0])) print (filename4[0]) netCDF4.Dataset(filename4[0], 'r') result: True True <class 'str'> /home/awips/data/OR_ABI-L1b-RadC-M6C01_G16_s20200521831155_e20200521833528_c20200521833586.nc

OSError: [Errno -101] NetCDF: HDF error: b'/home/awips/data/OR_ABI-L1b-RadC-M6C01_G16_s20200521831155_e20200521833528_c20200521833586.nc'

ncdump on the file from within the program works.

Suggestions? Thanks, Ray

[akrherz]

Are you able to attach the file here for others to inspect?

[weather01089]

the netcdf file is fine, its being read by another program fine. ncdump also displays the appropriate data.

[akrherz]

If you hope to get help here, you need to share the file, so that further debugging can happen. There is likely some subtle quirk with the file that needs review.

[weather01089]

I can download one, that one is now gone, deleted. I ran the program again and Ill grab a file

[weather01089]

the netcdf wont copy/paste you want the python code?

[weather01089]

from satpy.utils import debug_on debug_on() def humanize_time(secs): mins, secs = divmod(secs, 60) hours, mins = divmod(mins, 60) return '%02d:%02d:%02d' % (hours, mins, secs) import dask from multiprocessing.pool import ThreadPool dask.config.set(pool=ThreadPool(8)) import os import sys os.environ['PYTROLL_CHUNK_SIZE']='1312' os.environ['DASK_NUM_WORKERS']='8' os.environ['OMP_NUM_THREADS']='4' os.chdir('/home/awips/data') os.environ['PPP_CONFIG_DIR']='/root/pytroll/lib/python2.7/site-packages/satpy' os.environ['GDAL_CACHEMAX']='15%' import time start_time = time.time() from dask.diagnostics import ProgressBar from numpy import loadtxt if name == 'main': from satpy import Scene from glob import glob from satpy import MultiScene import subprocess sector=sys.argv[1] product=sys.argv[2] sectordef=sys.argv[3] sectorfile='/home/awips/commands/sectors/sectordef'+sectordef sectorcmd='/home/awips/commands/satpycommands/satcat '+sectordef print ('getting filename') scriptname='/home/awips/commands/satpycommands/testls'+sector scriptname=scriptname+'1' latlon = loadtxt(sectorfile, delimiter=" ", unpack=False) print (type(latlon)) print(latlon) latlon=list(latlon) print(latlon) print (type(latlon)) ll_bbox=latlon tempfile='/tmp/Conus' result6 = subprocess.check_output(scriptname) fp=open(tempfile,"r") filename4=fp.readline() fp.close() print ('File output=',filename4) print (type (filename4)) print (product) filename4 = filename4.replace("'", "").split(",")

filename4 = filename4[.replace(" ", "").split(",")

filename4 = filename4.split(',')

print ('File length=') print (len(filename4)) print(type(filename4)) print (filename4[0]) print(os.path.isfile(filename4[0])) print(os.path.isfile(os.path.realpath(filename4[0]))) import netCDF4 print(netCDF4._netCDF4) import subprocess print(type(filename4[0])) print (filename4[0]) netCDF4.Dataset(filename4[0], 'r') mscn = Scene(filename4, reader='abi_l1b')

output: Traceback (most recent call last): File "/home/awips/commands/satpycommands/sector5.py", line 74, in netCDF4.Dataset(filename4[0], 'r') File "netCDF4/_netCDF4.pyx", line 2321, in netCDF4._netCDF4.Dataset.init File "netCDF4/_netCDF4.pyx", line 1885, in netCDF4._netCDF4._ensure_nc_success OSError: [Errno -101] NetCDF: HDF error: b'/home/awips/data/OR_ABI-L1b-RadC-M6C01_G16_s20200522036154_e20200522038527_c20200522039007.nc'

[jswhit2]

If the file is small, you can zip it and attach it to the issue.

[jswhit2]

To rule out some kind of a conflict in the C extension modules being imported in your script, can you try opening the file in a separate, standalone script that only imports netCDF4?

[weather01089]

Ill do that in the meantime, here is the zip OR_ABI-L1b-RadC-M6C01_G16_s20200521911155_e20200521913528_c20200521913591.zip

[weather01089]

With only os and netCDF4 imported (had to open a file to read the filename), it did NOT error. Ray

[jswhit2]

No problem opening it here. There must be an import conflict. I would suggest importing netCDF4 first in your script. A similar issue was reported earlier - the conflict was caused by different modules linking different versions of the netCDF4/HDF5 libraries.

The other option is to rebuilding the offending modules locally making sure they link the same libraries.

[weather01089]

Indeed there was in issue, with debug!

from satpy.utils import debug_on

debug_on()

with it off, netcdf is fine. Thanks!

[weather01089]

The import alone made netcdf4 fail