problem generating products with python

[pat1]

/usr/share/doc/meteosatlib-gdal/products -t 202002231200 -s . -d /tmp/products/ --warp='-t_srs "+proj=latlong" -te -10 30 30 60' --cachedir=/tmp/products/cache/ Traceback (most recent call last): File "/usr/share/doc/meteosatlib-gdal/products", line 842, in sat.make_product(prod, opts.format, opts.shp, destdir=opts.destdir) File "/usr/share/doc/meteosatlib-gdal/products", line 226, in make_product res = maker.compute() File "/usr/share/doc/meteosatlib-gdal/products", line 279, in compute ir16r = self.sat.IR_016.reflectance File "/usr/share/doc/meteosatlib-gdal/products", line 102, in reflectance data = super(MaskedChannelMixin, self).reflectance File "/usr/share/doc/meteosatlib-gdal/products", line 74, in reflectance self.cached_refl = self.sat.get_array(self.name + "r") File "/usr/share/doc/meteosatlib-gdal/products", line 162, in get_array ds = self.gdal_dataset(channel) File "/usr/share/doc/meteosatlib-gdal/products", line 149, in gdal_dataset for line in stderr.split("\n"): TypeError: a bytes-like object is required, not 'str'

[edigiacomo]

@pat1 can you try with examples/products from commit 1e91291 ?

[pat1]

Sample data can be found at: https://github.com/ARPA-SIMC/qc_sample_data/tree/master/satellite/hrit

I have to change "MSG3" to "MSG4" in the script but now I get:

./products -t 202002231200 -s . -d /tmp/products/ --warp='-t_srs "+proj=latlong" ' --cachedir=/tmp/products/cache/
E:ERROR 1: Reprojection failed, err = -20, further errors will be suppressed on the transform object.
E:ERROR 1: Reprojection failed, err = -20, further errors will be suppressed on the transform object.
E:ERROR 4: Attempt to create new tiff file `/tmp/products/cache/H-MSG4-IR_016r-202002231200-latlon.tif' failed: No such file or directory
gdalwarp returned 1:
Traceback (most recent call last):
  File "./products", line 842, in <module>
    sat.make_product(prod, opts.format, opts.shp, destdir=opts.destdir)
  File "./products", line 226, in make_product
    res = maker.compute()
  File "./products", line 279, in compute
    ir16r = self.sat.IR_016.reflectance
  File "./products", line 102, in reflectance
    data = super(MaskedChannelMixin, self).reflectance
  File "./products", line 74, in reflectance
    self.cached_refl = self.sat.get_array(self.name + "r")
  File "./products", line 162, in get_array
    ds = self.gdal_dataset(channel)
  File "./products", line 154, in gdal_dataset
    os.unlink(cache_fname)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/products/cache/H-MSG4-IR_016r-202002231200-latlon.tif'

without gdal warp:

./products -t 202002231200 -s . -d /tmp/products/  --cachedir=/tmp/products/cache/
ERROR 4: Attempt to create new tiff file `/tmp/products/day-natural.tif' failed: No such file or directory
Traceback (most recent call last):
  File "./products", line 842, in <module>
    sat.make_product(prod, opts.format, opts.shp, destdir=opts.destdir)
  File "./products", line 240, in make_product
    outds = self.new_dataset(cls.BASE_CHANNEL, fname, driver=d, bands=bs, gdtype=gdal.GDT_Byte)
  File "./products", line 188, in new_dataset
    target_ds.SetGeoTransform((
AttributeError: 'NoneType' object has no attribute 'SetGeoTransform'

[brancomat]

I tried to reproduce the bug but I stumbled into #23

Meanwhile, some questions:

• what platform and gdal version are you using?
• did you clean the cache dir before launching the products script?

[brancomat]

Update: I can't reproduce the issue with the latest products script (version 1.13, released just now). Can you try it and see if it works for you?

If it fails, please provide some info on the platform / gdal version you're using

[pat1]

If the problem is specific for fFedora 31 we can close this without problems

Fedora 31

rpm -qf /usr/share/doc/meteosatlib-gdal/products meteosatlib-gdal-1.12-1.x86_64

rpm -q gdal gdal-2.3.2-10.fc31.x86_64

/usr/share/doc/meteosatlib-gdal/products -t 202002231200 -s . -d /tmp/products/ --warp='-t_srs "+proj=latlong" -te -10 30 30 60' --cachedir=/tmp/products/  --msat=MSG4
Traceback (most recent call last):
  File "/usr/share/doc/meteosatlib-gdal/products", line 883, in <module>
    sat.make_product(prod, args.format, args.shp, destdir=args.destdir)
  File "/usr/share/doc/meteosatlib-gdal/products", line 252, in make_product
    res = maker.compute()
  File "/usr/share/doc/meteosatlib-gdal/products", line 307, in compute
    vis8r = self.sat.VIS008.reflectance
  File "/usr/share/doc/meteosatlib-gdal/products", line 123, in reflectance
    mask = numpy.ma.mask_or(data.mask, self.solar_mask)
  File "/usr/share/doc/meteosatlib-gdal/products", line 98, in solar_mask
    self.cached_solar_mask = self.sat.get_solar_mask(self.name)
  File "/usr/share/doc/meteosatlib-gdal/products", line 203, in get_solar_mask
    cos_sol_za = self.get_array(channel + "a")
  File "/usr/share/doc/meteosatlib-gdal/products", line 188, in get_array
    ds = self.gdal_dataset(channel)
  File "/usr/share/doc/meteosatlib-gdal/products", line 174, in gdal_dataset
    for line in stderr.split("\n"):
TypeError: a bytes-like object is required, not 'str'

/usr/share/doc/meteosatlib-gdal/products -t 202002231200 -s . -d /tmp/products/  --msat MSG5 --cachedir=/tmp/products/cache/
terminate called after throwing an instance of 'std::runtime_error'
  what():  No such file(s)
Annullato (core dump creato)

[brancomat]

I suspect it's a problem with gdal 2 rather than fedora 31 specific. Unfortunately we don't have a f31 test server. We have a CentOS7 server but it has gdal 1.11 and I can't launch the products script there since python3-gdal is not available.

Given our current production and development operating systems I'm afraid we can't offer meteosatlib support for gdal < 3 anymore.