ChangeType.update stac_dist/ERA5_Land_monthly/ERA5_Land_monthly.json

[fairicube-data]

{"filename": "ERA5_Land_monthly/ERA5_Land_monthly.json", "item_type": "stac_dist", "change_type": "Update", "user": "FAiRICUBE", "data_owner": true}

[misev]

@eox-cs1 @KathiSchleidt (not sure who is the right person) I have a problem adding this datacube to the catalog, it seems to ignore the time unit I set (I tried day and d) and the automatic validation fails. When I go to edit the entry, the time unit is empty so it wasn't saved at all.

[baloola]

Hi @misev, thank you for reporting this issue, this looks like a bug, I will fix it.

[misev]

@baloola ok, just let me know when it's fixed so I can update it.

Also, I'm not sure if this is configured in the catalog editor: is it possible to change the default reviewer for rasdaman data requests from @Mohinem to @misev?

[baloola]

I manually fixed the time unit issue now, I did fix the issue in the GUI frontend code but before I create the release I would llike to ask, do you want to replace Mohit as the reviewer for all rasdaman products ? waitnig for your answer :) PS: band evaporation_from_the_top_of_canopy has no nodata value, fixing this will pass the validation test

[misev]

Yes please replace Mohit with misev for all rasdaman cases. The nodata should be same as the other bands -32767, maybe you could add that as well? Thanks.

[baloola]

done

[KathiSchleidt]

When we discussed providing information on ERA5 with the UC, we had agreed to put the various variable names in the keywords section to enable simpler search. Could you please add these?

Otherwise the JSON looks good :)

[KathiSchleidt]

@misev Question to the new temporal encoding, at the ERA5 source I see that this dataset is monthly, to my understanding what's being provided in ERA5_Land_monthly is hourly (resolution="0.041666666666666664" days). How does this work?

[misev]

I updated the keywords; if this is a requirement maybe the GUI code can automatically append the variable names to the keywords @baloola @KathiSchleidt? Also I'd suggest to add a hint on the text field that it expects comma-separated values (I just guessed this, wasn't sure if it's what's really expected).

@KathiSchleidt the resolution was a copy-paste error. I'm not sure how to set it to 1 month yet though, because the time CRS we use AnsiDate has UoM of day. Adding a new CRS at OGC is pretty involved as I understand (@pebau). As an interim workaround we could convert the time axis to an irregular axis, in which case a resolution will not be needed.

The information in the netcdf data is as following btw:

    int time(time) ;
        time:units = "hours since 1900-01-01 00:00:00.0" ;
        time:long_name = "time" ;
        time:calendar = "gregorian" ;

[KathiSchleidt]

@misev I like the idea of automatically adding the names of the bands to the keywords, @baloola can you add this to the implementation tasks for the WebGUI? Described in the issue Automatically add band names to Keywords

On temporal information, are "hours since 1900-01-01 00:00:00.0" a thing? Never encountered before, are there functions converting this to normal dates as with UNIX or POSIX Time? Btw - from what I've picked up in the OGC, the decision there is to go for POSIX time

[misev]

In rasdaman I changed the time axis to irregular, but with monthly areas of validity. This technically makes it regular with 1 month resolution, so in the catalog entry I've marked it as such.

With that I think this data request can be merged into the catalog.

[misev]

On temporal information, are "hours since 1900-01-01 00:00:00.0" a thing? Never encountered before, are there functions converting this to normal dates as with UNIX or POSIX Time? Btw - from what I've picked up in the OGC, the decision there is to go for POSIX time

Yes it seems common in netcdf data. We convert them to standard date with some Python code in our import config:

"axes": {
  "time": {
    "min": "(datetime(1900,1,1,0,0,0) + timedelta(hours=${netcdf:variable:time:min})).strftime(\"%Y-%m-%dT%H:%M\")",
    "max": "(datetime(1900,1,1,0,0,0) + timedelta(hours=${netcdf:variable:time:max})).strftime(\"%Y-%m-%dT%H:%M\")",
    "statements": "from datetime import datetime, timedelta",
    "directPositions": "[(datetime(1900,1,1,0,0,0) + timedelta(hours=i)).strftime(\"%Y-%m-%dT%H:%M\") for i in ${netcdf:variable:time}]",
    "gridOrder": 0,
    "irregular": true,
    "areasOfValidity": [ { "start": "(datetime(1900,1,1,0,0,0) + timedelta(hours=${netcdf:variable:time:min})).strftime(\"%Y-%m\")",
                           "end": "(datetime(1900,1,1,0,0,0) + timedelta(hours=${netcdf:variable:time:min})).strftime(\"%Y-%m\")" } ]
  },
  ...

[baloola]

Some remarks:

• Bbox typo ? minx = maxx (180, -90, 180, 90)
• the thumbnail request is returning 500 https://catalog:JdpsUHpPoqXtbM3@fairicube.rasdaman.com/rasdaman/ows?service=WMS&version=1.3.0&request=GetMap&layers=ERA5_Land_monthly&bbox=-90,-180,90,180&time=1950-01-01T00:00Z&width=800&height=600&crs=EPSG:4326&format=image/png&transparent=true&styles=

[KathiSchleidt]

@misev I just tried to look at the DescribeCoverage of this dataset, but seems there is an issue. Get the following response:

<ows:Exception exceptionCode="IOConnectionError" >
        <ows:ExceptionText>Cannot retrieve serialized object in JSON string from remote petascope endpoint 'https://ds.earthserver.xyz/rasdaman/rasfed/coverage/description?coverageId=ds.earthserver.xyz:7000:ERA5_Land_monthly&version=10.0.9'. Reason: Cannot get result from URL 'https://ds.earthserver.xyz/rasdaman/rasfed/coverage/description?coverageId=ds.earthserver.xyz:7000:ERA5_Land_monthly&version=10.0.9'. Reason: Read timed out</ows:ExceptionText>
</ows:Exception>

On the time encoding, I wonder how many users will be able to work with this. If you decide to go this way, please provide the required functions as python code under the Common Code Repo

[misev]

We have an issue currently on our server so DescribeCoverage and other requests will fail; we are working to remedy that.

The time conversion is only relevant for the import of netcdf into rasdaman. Users of the WCS/WCPS service see and work with standard ISO datetimes.

[KathiSchleidt]

@misev do I understand correctly that the FAIRiCUBE rasdaman VM is also a development machine?

[misev]

The VM hosts a rasdaman installation and a JupyterLab. What do you mean by development machine?

[KathiSchleidt]

We've had such issues with the rasdaman server being unavailable several times, thus I get the impression that new untested SW is being deployed. Back when I was in development, we had 3 tiers

• development: usually on developers desktop
• integration: a dedicated machine where the code from all developers comes together, is tested together
• production: a functional machine used by end-users, only tested SW deployed

[misev]

It's a production server, nobody is developing or testing on the server. However, some issues become apparent only in production due to differences in data size, and we hit such a case now.

[misev]

@baloola thanks for checking, I've fixed the bbox and the thumbnail request works now as well. @KathiSchleidt the DescribeCoverage works fine as well now.

[KathiSchleidt]

@misev much better. Still a few issues:

• for several of the bands, you provide UoM=1. Under ERA5, these are listed as m2 m-2. While this logically cancels out, the m²/m² has a meaning for users. I think UCUM would expect m2m-2 or m2/m2. Same applies with exponent 3
• I understand the issues with temporal CRS, it's not possible to work with offsets, thus we have to list all individual times. But, I'm wondering why we have to provide this loooong list twice, once under the DomainSet, the IrregularAxis time, a second time under Metadata in undocumented XML. Must we provide these ~900 entries twice?

[KathiSchleidt]

@misev question: in the OWS Spec, while I see that Longitude values are NOT limited to +/– 180 degrees., I do not find a corresponding entry for Latitude. Is lowerBound="-90.05" upperBound="90.05" correct?

Longitude is provided as lowerBound="-180.05" upperBound="179.95", seems inconsistent

[misev]

• The UCUM version for those units is 1 as far as I understood. I can change them back to m2/m2 etc. though.
• The metadata contains the areas of validity (or footprints) of the temporal coordinates, which ensure monthly resolution in rasdaman. It's documented in the rasdaman documentation.
• The Lat/Lon effect is because of pixel in corner semantics in the netcdf data (I presume, it's not stated anywhere) vs. pixel in center in rasdaman. I'll correct our import config to match the data

[KathiSchleidt]

Thanks for correcting the UoM.

On the various elements that rasdaman provides under the ras namespace in the metadata element, is there any ambition to provide a schema for these components? The documentation provided is a bit superficial

[pebau]

On the various elements that rasdaman provides under the ras namespace in the metadata element, is there any ambition to provide a schema for these components? The documentation provided is a bit superficial

That is currently under work, including setting up a schema repo.