Fedora 36 build errored: field `sources` has incomplete type `std::array<GDALRasterBand*, 12>`

[brancomat]

This happens only in fedora 36 builds (gdal 3.4.3)

Full log: https://download.copr.fedorainfracloud.org/results/simc/stable/fedora-36-x86_64/04438391-meteosatlib/builder-live.log.gz or https://simc.arpae.it/moncic-ci/meteosatlib/202205240728/master/fedora36/build.log

Relevant bit:

make[2]: Entering directory '/builddir/build/BUILD/meteosatlib-1.14-3/gdal'
/bin/sh ../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H    -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1     -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection  -Wall -Wextra -Wno-unused-parameter -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1      -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -c -o msatgdalplugin.lo msatgdalplugin.cpp
libtool: compile:  g++ -DHAVE_CONFIG_H -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -Wall -Wextra -Wno-unused-parameter -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -c msatgdalplugin.cpp  -fPIC -DPIC -o .libs/msatgdalplugin.o
In file included from msatgdalplugin.cpp:27:
reflectance/reflectance.h:26:37: error: field 'sources' has incomplete type 'std::array<GDALRasterBand*, 12>'
   26 |     std::array<GDALRasterBand*, 12> sources{};
      |                                     ^~~~~~~
In file included from /usr/include/c++/12/bits/unique_ptr.h:36,
                 from /usr/include/c++/12/memory:76,
                 from /usr/include/gdal/cpl_minixml.h:192,
                 from /usr/include/gdal/gdal.h:49,
                 from /usr/include/gdal/gdal_priv.h:60,
                 from reflectance/base.h:4,
                 from reflectance/reflectance.h:4:
/usr/include/c++/12/tuple:1595:45: note: declaration of 'struct std::array<GDALRasterBand*, 12>'
 1595 |   template<typename _Tp, size_t _Nm> struct array;
      |                                             ^~~~~
make[2]: *** [Makefile:686: msatgdalplugin.lo] Error 1
make[2]: Leaving directory '/builddir/build/BUILD/meteosatlib-1.14-3/gdal'

[spanezz]

This is the current output of make check that I observe

Running  /tmp/meteosatlib/tests/msat_test 
msat_facts: ...
msat_fileaccess: ....
msat_dataaccess: ......
gdal_georef_grib: ...
gdal_georef_netcdf: ...
gdal_georef_xrit: ...
gdal_georef_xrit_hrv: ...
gdal_import_grib: .............
gdal_import_netcdf: .............
gdal_import_xrit: .............
gdal_import_xrit_rss: .............
gdal_import_xrit_nonlinear: .............
gdal_import_xrit_hrv: .............
gdal_import_xrit_rss_hrv: .............
gdal_xrit_reflectance: ✘.✘.✘.✘✘.✘.
gdal_xrit_solar_za: ✘.✘.

 * Test failures

gdal_xrit_reflectance.vis06: N4msat5tests4gdal13gdalexceptionE: GDAL failure error 1: Point outside of projection domain
  gdal/test-xrit-reflectance.cpp:50:actual(rb->RasterIO(GF_Read, 2000, 3400, 1, 1, &valr, 1, 1, GDT_Float32, 0, 0)) == CE_None

gdal_xrit_reflectance.ir039: N4msat5tests4gdal13gdalexceptionE: GDAL failure error 1: Point outside of projection domain
  gdal/test-xrit-reflectance.cpp:87:actual(rb->RasterIO(GF_Read, 2000, 350, 1, 1, &valr, 1, 1, GDT_Float32, 0, 0)) == CE_None

gdal_xrit_reflectance.new_vis06: N4msat5tests4gdal13gdalexceptionE: GDAL failure error 1: Point outside of projection domain
  gdal/test-xrit-reflectance.cpp:130:actual(rb->RasterIO(GF_Read, 2000, 3400, 1, 1, &valr, 1, 1, GDT_Float32, 0, 0)) == CE_None

gdal_xrit_reflectance.new_ir039_sat_za: N4msat5tests4gdal13gdalexceptionE: GDAL failure error 1: Point outside of projection domain
  gdal/test-xrit-reflectance.cpp:167:actual(rb->RasterIO(GF_Read, 2000, 3400, 1, 1, &valr, 1, 1, GDT_Float64, 0, 0)) == CE_None

gdal_xrit_reflectance.new_ir039_cos_sol_za: N4msat5tests4gdal13gdalexceptionE: GDAL failure error 1: Point outside of projection domain
  gdal/test-xrit-reflectance.cpp:185:actual(rb->RasterIO(GF_Read, 2000, 3400, 1, 1, &valr, 1, 1, GDT_Float64, 0, 0)) == CE_None

gdal_xrit_reflectance.new_ir039_vrt_reflectance: 100.000 is different than the expected 22.324
  gdal/test-xrit-reflectance.cpp:211:actual((double)valr).almost_equal(22.3242, 3)

gdal_xrit_solar_za.vis06:[N4msat5tests4gdal13gdalexceptionE] GDAL failure error 1: Point outside of projection domain

gdal_xrit_solar_za.ir039:[N4msat5tests4gdal13gdalexceptionE] GDAL failure error 1: Point outside of projection domain

 * Result summary

8/131 tests failed
123 tests succeeded

[spanezz]

Questo problema nei test non si riproduce su debian bullseye

[spanezz]

Ho risolto i vari Point outside of projection, e ok. Rimane 100.000 is different than the expected 22.324.

Nel GDAL di Fedora36, l'elemento OpenOptions del file .vrt (vedi tests/data/reflectance_ir039.vrt) non viene piú passato nei dati del dataset da aprire.

L'unico workaround che mi viene in mente è supportare le varie postprocessazioni di dataset di meteosatlib non solo tramite opzioni di apertura, ma anche tramite una sintassi nel nome del file (tipo aprendo file.grib::sat_za per postprocessare calcolando il satellize zenith angle).

Ho cercato di guardare i sorgenti di gdal per capire come mai si fossero perse quelle informazioni, ma ho avuto poco successo. Semmai provo a chiedere in IRC

[spanezz]

In alternativa, non si specificano i postprocessing nel file .vrt ma solo i dataset di input, e poi si fa il setup dei postprocessing internamente.

In entrambe le proposte di workaround cambierebbe il comportamento dei file .vrt: quanto e come sono usati in produzione?

[spanezz]

Come non detto, il setup del postprocessing internamente non si può fare perché alla pixel function che calcola la riflettanza arrivano non i dataset o rasterband sorgenti, ma gìa delle matrici di dati

[spanezz]

Intanto confermo che il bug persiste anche in gdal 3.5

[spanezz]

Sembra che al momento non sia usata la funzione di usare vrt per calcolare riflettanze a partire da dataset non necessariamente in formato HRIT, per cui il resto di questo issue si può risolvere semplicemente rimuovendo la funzionalità da meteosatlib.

Prima di farlo, cerco di creare un reproducer per la mancanza del passaggio delle OpenOptions e di aprire un bug in GDAL, cosí se poi ci servirà reintrodurre questa funzionalità, magari avremo lumi lato GDAL

[spanezz]

Ho fatto un sorgente minimo per riprodurre il problema e ho aperto un issue a GDAL