search

barronh / pseudonetcdf

PseudoNetCDF like NetCDF except for many scientific format backends
GNU Lesser General Public License v3.0
77 stars 35 forks source link

How to open icartt file from the python script? #24

Closed jaegunjung closed 7 years ago

jaegunjung commented 7 years ago

Hi Barron,

I am using python3 installed on anaconda3. I open the module as follows,

from PseudoNetCDF.icarttfiles.ffi1001 import ffi1001

And tried to open the icartt file as follows,

file_plane = ffi1001(inpath_plane)

I got the following error messages,

File ".//diag_plane.step0.py", line 129, in diag_plane() File ".//diag_plane.step0.py", line 71, in diag_plane file_plane = ffi1001(inpath_plane) File "/disk41/jjung_linux/util/python/anaconda3/lib/python3.5/site-packages/PseudoNetCDF/icarttfiles/ffi1001.py", line 40, in init l = f.readline() File "/disk41/jjung_linux/util/python/anaconda3/lib/python3.5/codecs.py", line 321, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte

I can convert to netcdf file from icartt but wonder whether I can do this through the python script.

Thanks - Jaegun

barronh commented 7 years ago

The utf-8 encoding is supposed to be the ICARTT standard[1], but that is not a strict standard and sometimes people use other character sets (often unknowingly). If the file contains special characters from another encoding (e.g., latin1, utf-16, etc), then you will need to specify the encoding. This can be done using the ffi1001 encoding option. If you're not sure what it is, try latin1 or cp1252, which are supersets. Below is an example using latin1.

file_plane = ffi1001(inpath_plane, encoding = 'latin1')

Use help on any of the readers to learn more about their options.

[1] https://cdn.earthdata.nasa.gov/conduit/upload/6158/ESDS-RFC-029v2.pdf

jaegunjung commented 7 years ago

Hi Barron,

My problem is due to trying to read the file that is already converted to netcdf files from pncgen instead of the text file. Sorry for not checking this.

Thanks - Jaegun