search

pytroll / satpy

Python package for earth-observing satellite data processing
http://satpy.readthedocs.org/en/latest/
GNU General Public License v3.0
1.06k stars 292 forks source link

Satpy Writer 'geotiff' exists but could not be loaded #928

Closed anikfal closed 4 years ago

anikfal commented 5 years ago

Code: from satpy import available_writers available_writers()

Description: Output of the above code: Could not import writer config from: ['/home/.../satpy/etc/writers/geotiff.yaml'] Could not import writer config from: ['/home/.../satpy/etc/writers/ninjotiff.yaml'] ['mitiff', 'simple_image', 'scmi', 'cf']

Environment Info:

djhoese commented 5 years ago

You are most likely missing the rasterio library which is a dependency of the geotiff writer. There is also the check_satpy function mentioned in the Troubleshooting section of the Quickstart page of the satpy documentation that could help determine the problem.

@mraspaud Maybe we should make the "hidden" debug messages that likely exist more obvious in cases like this.

anikfal commented 5 years ago

Code:

from satpy.config import check_satpy
check_satpy()

Code Output:

Readers
=======
abi_l1b:  ok
abi_l1b_scmi:  ok
acspo:  ok
ahi_hrit:  ok
ahi_hsd:  ok
amsr2_l1b:  ok
avhrr_l1b_aapp:  ok
avhrr_l1b_eps:  ok
avhrr_l1b_gaclac:  cannot find module 'satpy.readers.avhrr_l1b_gaclac' (No module named 'pygac')
avhrr_l1b_hrpt:  cannot find module 'satpy.readers.hrpt' (No module named 'pygac')
caliop_l2_cloud:  ok
clavrx:  ok
electrol_hrit:  ok
fci_l1c_fdhsi:  ok
generic_image:  ok
geocat:  ok
ghrsst_l3c_sst:  ok
goes-imager_hrit:  ok
goes-imager_nc:  ok
grib:  cannot find module 'satpy.readers.grib' (No module named 'pygrib')
hsaf_grib:  cannot find module 'satpy.readers.hsaf_grib' (No module named 'pygrib')
iasi_l2:  ok
jami_hrit:  ok
li_l2:  cannot find module 'satpy.readers.li_l2' (No module named 'h5netcdf')
maia:  ok
mersi2_l1b:  ok
modis_l1b:  ok
modis_l2:  ok
msi_safe:  cannot find module 'satpy.readers.msi_safe' (No module named 'glymur')
mtsat2-imager_hrit:  ok
nc_seviri_l1b:  cannot find module 'satpy.readers.nc_seviri_l1b' (No module named 'satpy.readers.msg_base')
nucaps:  ok
nwcsaf-geo:  ok
nwcsaf-pps_nc:  ok
olci_l1b:  ok
olci_l2:  ok
omps_edr:  ok
safe_sar_l2_ocn:  ok
sar-c_safe:  ok
scatsat1_l2b:  ok
seviri_l1b_hrit:  ok
seviri_l1b_native:  ok
seviri_l1b_nc:  ok
slstr_l1b:  ok
tropomi_l2:  ok
vaisala_gld360:  ok
viirs_compact:  ok
viirs_edr_active_fires:  cannot find module 'satpy.readers.viirs_edr_active_fires' (fsspec is required to use any file-system functionality. Please install using
conda install -c conda-forge 'fsspec>=0.3.3'
or
pip install 'fsspec>=0.3.3'

Dask dataframe requirements are not installed.

Please either conda or pip install as follows:

  conda install dask                     # either conda install
  pip install dask[dataframe] --upgrade  # or pip install)
viirs_edr_flood:  ok
viirs_l1b:  ok
viirs_sdr:  ok
virr_l1b:  ok

Writers
=======
cf:  ok
geotiff:  ok
mitiff:  ok
ninjotiff:  cannot find module 'satpy.writers.ninjotiff' (No module named 'pyninjotiff')
scmi:  ok
simple_image:  ok

Extras
======
cartopy:  ok
geoviews:  No module named 'geoviews'

I don't know what the cause of the missing modules is. As far as I remember, I used conda for installing most of the packages. However, the Writer geotiff seems to be OK, but could not be loaded.

djhoese commented 5 years ago

Try running from satpy.writers import geotiff. That should show us an import error directly. Also double check that you are running with the same python environment (no changes to PYTHONPATH, etc) for each thing that we try for this issue.

anikfal commented 5 years ago

It's interesting that from satpy.writers import geotiff didn't show any error, and this time Writer geotiff has been loaded. Code:

from satpy import available_writers
available_writers()

Code output:

Could not import writer config from: ['/home/ah/.conda/envs/mypyngl/lib/python3.7/site-packages/satpy/etc/writers/ninjotiff.yaml']
['scmi', 'cf', 'simple_image', 'mitiff', 'geotiff']

ninjotiff.yaml still has some problems. Anyway, sorry for the confusion. Maybe I am doing something wrong. I use conda virtual environment for python.

djhoese commented 5 years ago

Are you running this code from a notebook or interactive environment like Spyder? Maybe you installed rasterio or updated Satpy and needed to restart python?

anikfal commented 5 years ago

I run the codes in Linux Terminal. I will check the issue in another system which is identical to my system.

djhoese commented 4 years ago

@anikfal Any luck in getting this to work?

anikfal commented 4 years ago

@djhoese After installing the missing rasterio library, geotiff appears in the available libraries:

from satpy import available_writers
available_writers()
Could not import writer config from: ['/home/ah/.conda/envs/newpyth/lib/python3.7/site-packages/satpy/etc/writers/ninjotiff.yaml']
['geotiff', 'cf', 'mitiff', 'scmi', 'simple_image']

But using this writer to convert the data to Geotiff seems not much helpful:

from satpy.scene import Scene
from satpy import find_files_and_readers
from datetime import datetime
files = find_files_and_readers(base_dir='/home/ah/MSG10-sdscase/datamsg8',
                               #start_time=datetime(2019,4,1,9,00),
                               #end_time=datetime(2019,4,1,9,15),
                               reader='seviri_l1b_nc')
scn = Scene(filenames=files)
scn.load([10.8])
scn.save_datasets(writer='geotiff')

With the errors which seem to be related to the type of the projection:

/home/ah/.conda/envs/newpyth/lib/python3.7/site-packages/trollimage/xrimage.py:311: FutureWarning: This DataArray contains multi-dimensional coordinates. In the future, these coordinates will be transposed as well unless you specify transpose_coords=False. data = data.transpose('bands', 'y', 'x') Traceback (most recent call last): File "1geotiff.py", line 22, in scn.save_datasets(writer='geotiff') File "/home/ah/.conda/envs/newpyth/lib/python3.7/site-packages/satpy/scene.py", line 1344, in save_datasets return writer.save_datasets(datasets, compute=compute, save_kwargs) File "/home/ah/.conda/envs/newpyth/lib/python3.7/site-packages/satpy/writers/init.py", line 676, in save_datasets results.append(self.save_dataset(ds, compute=False, kwargs)) File "/home/ah/.conda/envs/newpyth/lib/python3.7/site-packages/satpy/writers/init.py", line 807, in save_dataset return self.save_image(img, filename=filename, compute=compute, fill_value=fill_value, kwargs) File "/home/ah/.conda/envs/newpyth/lib/python3.7/site-packages/satpy/writers/geotiff.py", line 190, in save_image gdal_options) File "/home/ah/.conda/envs/newpyth/lib/python3.7/site-packages/trollimage/xrimage.py", line 283, in save format_kwargs) File "/home/ah/.conda/envs/newpyth/lib/python3.7/site-packages/trollimage/xrimage.py", line 370, in rio_save r_file.open() File "/home/ah/.conda/envs/newpyth/lib/python3.7/site-packages/trollimage/xrimage.py", line 104, in open self.rfile = rasterio.open(self.path, mode, self.kwargs) File "/home/ah/.conda/envs/newpyth/lib/python3.7/site-packages/rasterio/env.py", line 445, in wrapper return f(*args, kwds) File "/home/ah/.conda/envs/newpyth/lib/python3.7/site-packages/rasterio/init.py", line 226, in open kwargs) File "rasterio/_io.pyx", line 1202, in rasterio._io.DatasetWriterBase.init File "rasterio/_io.pyx", line 1245, in rasterio._io.DatasetWriterBase._set_crs File "rasterio/_err.pyx", line 182, in rasterio._err.exc_wrap_int rasterio._err.CPLE_AppDefinedError: Only OGC WKT Projections supported for writing to GeoTIFF. PROJCRS["unknown",BASEGEOGCRS["unknown",DATUM["unknown",ELLIPSOID["unknown",6378169,295.488065897014,LENGTHUNIT["metre",1,ID["EPSG",9001]]]],PRIMEM["Greenwich",0,ANGLEUNIT["degree",0.0174532925199433],ID["EPSG",8901]]],CONVERSION["unknown",METHOD["Geostationary Satellite (Sweep Y)"],PARAMETER["Longitude of natural origin",41.5,ANGLEUNIT["degree",0.0174532925199433],ID["EPSG",8802]],PARAMETER["Satellite Height",35785831,LENGTHUNIT["metre",1,ID["EPSG",9001]]],PARAMETER["False easting",0,LENGTHUNIT["metre",1],ID["EPSG",8806]],PARAMETER["False northing",0,LENGTHUNIT["metre",1],ID["EPSG",8807]]],CS["Cartesian",2],AXIS["(E)",east,ORDER[1],LENGTHUNIT["metre",1,ID["EPSG",9001]]],AXIS["(N)",north,ORDER[2],LENGTHUNIT["metre",1,ID["EPSG",9001]]]] not supported.

For my case, using gdal (e.g. gdal_translate) directly, could handle the issue more effectively.

pnuu commented 4 years ago

@anikfal what is the output of print(scn[10.8].attrs['area'])? Although the log above shows the WKT version, I don't directly understand what it is saying. But my guess is that something in the area definition isn't supported.

djhoese commented 4 years ago

@anikfal What version of the GDAL C library do you have installed in your environment? What version of rasterio? pyproj? PROJ?

This error is from rasterio and is stating that the projection is not valid which is odd since it is coming from the reader.

anikfal commented 4 years ago

@pnuu Output of print(scn[10.8].attrs['area']):

Area ID: some_area_name Description: On-the-fly area Projection ID: geosmsg Projection: {'a': '6378169', 'b': '6356583.8', 'h': '35785831', 'lon_0': '41.5', 'no_defs': 'None', 'proj': 'geos', 'type': 'crs', 'units': 'm', 'x_0': '0', 'y_0': '0'} Number of columns: 1084 Number of rows: 1009 Area extent: (-664589.3012, 1648721.5396, 2584847.3274, 4676128.3339)

@djhoese Versions of the requested libraries:

gdal_translate --version
GDAL 2.4.2, released 2019/06/28

conda list | grep -i 'rasterio\|proj\|gdal'
anaconda-project          0.8.3                      py_0  
libgdal                   2.4.2                h04a3d72_6    conda-forge
proj4                     6.1.0                he751ad9_2    conda-forge
pyproj                    2.2.1            py37hc44880f_0    conda-forge
rasterio                  1.0.28           py37hdff7cfa_1    conda-forge
djhoese commented 4 years ago

Sorry, one more, what about the version of trollimage?

anikfal commented 4 years ago

@djhoese OS: Fedora Scientific 29

conda list | grep troll
trollimage                1.10.0                    <pip>
trollsift                 0.3.2                     <pip>

It is interesting that on my laptop with the regular Fedora 31, due to the lack of a library, rasterio could not be imported.

djhoese commented 4 years ago

@anikfal There is a newer version of trollimage (1.11.0) that should fix this issue or at least handle it better. Could you try updating trollimage and see if your code works?

anikfal commented 4 years ago

@djhoese Problem solved. Thank you. After updating trollimage to the version 1.11.0, the previous code generates the geotiff data with just a warning:

/home/ah/.conda/envs/newpyth/lib/python3.7/site-packages/trollimage/xrimage.py:427: FutureWarning: This DataArray contains multi-dimensional coordinates. In the future, these coordinates will be transposed as well unless you specify transpose_coords=False. data = data.transpose('bands', 'y', 'x')

djhoese commented 4 years ago

Glad it works for you now. Thanks for the update.