NO2 S5P PAL Maps

[AlessandroScremin]

Ciao @gmilcinski

We toegether with EOX and Anca we noticed that the NO2 Maps from S5P Pal are not displaing the colors corerctly. What we noticed is that there should be a kind of conversion from 16 bit to 8 bit while ingesting the source data via WMS. Is it a limitation of EDC or there is something from our side that need to be corrected? Let say they are the same products prepared for the Hackaton and same workflow...but when we compare the S5P Pal site and what is in EDC the colors but in some cases also values seems different. What can we do to make it very close to S5P Pal?

thanks for the support

[AlessandroScremin]

Dear @gmilcinski I also checked the original netcdf file from S5p PAL site (converted in geotiff with your EDC commands) with the ocnverted COG before ingestion and the images are the same....

from netcdf to geotiff the min max values varies from - 30 and 192 whereas after the conversion in geotiff we have -37 + 348. Eventhough the histogramm has the same shape behaviour except for the higher extreme value varying on singular pixel range

So something should happend via WMS....

let me kwno your opinion

k

Alessandro

[tomislijepcevic]

Hi Alessandro,

I will try to help you. Can you provide me WMS instance id, layer id, files or where can I get them, and commands used for conversion.

[AlessandroScremin]

@tomislijepcevic FYI @mdelgadoblasco We used the command suggested in the documentation

gdal_translate -a_nodata none NETCDF:"${filename}.nc":${band_name} intermediate.tif -a_srs '+proj=longlat +datum=WGS84'

gdaladdo -r average --config GDAL_TIFF_OVR_BLOCKSIZE 1024 intermediate.tif 2 4 8 16 32

gdal_translate -co TILED=YES -co COPY_SRC_OVERVIEWS=YES --config GDAL_TIFF_OVR_BLOCKSIZE 1024 -co BLOCKXSIZE=1024 -co BLOCKYSIZE=1024 -co COMPRESS=DEFLATE intermediate.tif ${filename}.tif

the only exception is the gdal_translate -a_nodata none because the nan in netcdf was a float.

the wms samle is this one https://shservices.mundiwebservices.com/ogc/wms/163db53d-e8b9-4513-b1ec-453ea722fcf3?version=1.1.1&service=WMS&request=GetMap&format=image/tiff;depth=32f&crs=EPSG:4326&layers=S5P-TW&RESX=1627.315m&RESY=1627.315m&BBOX=4.616667,52.050000000000004,5.216666999999999,52.65&TIME=2018-06-11/2018-06-11

layer id: ID: NO2 (visualisation)

let me know if you need something else

kr Alessandro

[tomislijepcevic]

I don't understand these two statements

"What we noticed is that there should be a kind of conversion from 16 bit to 8 bit while ingesting the source data via WMS."

"So something should happend via WMS...."

Are you saying that you think data gets converted from 16 bits to 8 bits when being accessed though WMS?

From what I see, your data is saved in 32-bit floating points and the evalscript of the layer NO2-visualisation outputs 8 bits.

[gobaRules]

HI @AlessandroScremin,

your gdal commands seems fine.

In your first post you have a print screen from qgis, where "diff_cog_pal" layer (probably the differences between netCDF file and COG) seems to have only 0. Is this not what you would expect?

How did you come to these values: "from netcdf to geotiff the min max values varies from - 30 and 192 whereas after the conversion in geotiff we have -37 + 348." ?

Kind regards, Anja

[AlessandroScremin]

@tomislijepcevic "Are you saying that you think data gets converted from 16 bits to 8 bits when being accessed though WMS?"

Yes because we used this //VERSION=3

function setup() { return { input: ["tropno2","weight", "dataMask"], output: { bands: 4, sampleType: "FLOAT32" } }; }

function evaluatePixel(sample) { var valt = sample.tropno2; var valw = sample.weight; // if (val>180) return [0,0,0,0]; // no data values return [valt,valt,valw,255];

}

and we got an error on Sentinel hub:

An error has occurred while fetching images: Format image/png does not support sample type FLOAT32.

And in any case the corrected script (the one in NO2-Visualisation layer) is the one you Provide us for the Space APP.... So we only use evalscript you provide us.

[AlessandroScremin]

@gobaRules Dear Anjia answer in line

HI @AlessandroScremin,

your gdal commands seems fine.

In your first post you have a print screen from qgis, where "diff_cog_pal" layer (probably the differences between netCDF file and COG) seems to have only 0. Is this not what you would expect?

The difference was done from NetCDF converted to Geotiff and the Cog (adding overviews and nodata value)

How did you come to these values: "from netcdf to geotiff the min max values varies from - 30 and 192 whereas after the conversion in geotiff we have -37 + 348." ?

These data come from the original NetCDF ebfore conversion to Geotiff and the values ion the Geotiff

(gdalinfo outputs)

Kind regards, Anja

[gobaRules]

So, if I understand correctly:

• netCDF -> geotiff conversion: values before and after the conversion were different,
• geotiff -> COG conversion : values before and after the conversion were the same.

If this is true, how did you do the netCDF -> geotiff conversion ?

[AlessandroScremin]

So, if I understand correctly:

• netCDF -> geotiff conversion: values before and after the conversion were different,
• geotiff -> COG conversion : values before and after the conversion were the same.

If this is true, how did you do the netCDF -> geotiff conversion ?

gdal_translate -a_nodata none NETCDF:"${filename}.nc":${band_name} intermediate.tif -a_srs '+proj=longlat +datum=WGS84'

gdaladdo -r average --config GDAL_TIFF_OVR_BLOCKSIZE 1024 intermediate.tif 2 4 8 16 32

gdal_translate -co TILED=YES -co COPY_SRC_OVERVIEWS=YES --config GDAL_TIFF_OVR_BLOCKSIZE 1024 -co BLOCKXSIZE=1024 -co BLOCKYSIZE=1024 -co COMPRESS=DEFLATE intermediate.tif ${filename}.tif

[gobaRules]

Ah, I see you posted that commands previously already. Sorry.

So, the difference seems to be the result of the first step netCDF -> geotiff. We do not know these data, we can not help much with the investigation. I suggest you follow the same steps as you did for the comparison of geotiff and COG:

• Drag the original .nc and itermediate.tif in Qgis and calculate differences to see where they appear.
• What datum does the original file use?

[AlessandroScremin]

@gobaRules could we have a meeting to discuss this issue...I don't think we can mange it via chat... let me know your availability FYI @mdelgadoblasco

[gobaRules]

Hi, yes, sure! Today after 1pm, tomorrow 9am-11am or Friday 9am-12am would work for me.

[AlessandroScremin]

@gobaRules @mdelgadoblasco sent invitation at 15 is it fine to you?

discussion will be focused on tis script and its refinement //VERSION=3

function setup() { return { input: ["tropno2", "dataMask"], output: { bands: 4, sampleType: "UINT8" } }; }

function evaluatePixel(sample) { var val = sample.tropno2; if (val>500) return [0,0,0,0]; // no data values var result= colorBlend(val, [0,45,90,135,180,254 ], //[[253,243,237], // 0 [[253,253,253], [248,210,192], // 45 [236,115,82], // 90
[213,43,34], // 135
[93,19,15],// 180 [30,15,15], // 254 ]); result.push(180); //transparence level return result; }

[gobaRules]

The time slot is ok for me.

If we are to discuss netCDF vs GeoTIFF differences, please have the examples of both files ready, so that we can do some analysis.