Trilateral GeoDB data additions

[lubojr]

As a part of trilateral dashboard migration to STAC, we have noticed a few datasets, which had part of their POIs only defined in the client. We would need these to be migrated to GeoDB to unify the used access.

1) E13c - Ship detections Planet there are 4 POIs defined here and defined here: which should be added to E13c_tri collection. No need to migrate the detections geometries or the planet scenes, just these POI entries and all the individual dates. 2) E13b - Plane detections Planet - same situation as 1), just 14 POIs are missing - defined here - same operation as in 1) Please add to E13b_tri collection.

Could you please integrate these data @dmoglioni ? Thank you

cc @santilland

[dmoglioni]

@AlessandroScremin

[AlessandroScremin]

@lubojr cc @santilland @dmoglioni

I think in order to add them we need the last csv used. During the Migration form csv to GeoDB you didn't point us to these csv, and now I'm not able to find them in github. same for ship and planes

I found one commit in the hiustory (app/public/data/trilateral/E13c.csv)

but would be beteter if you can link or point us to the latests version you used in the Dashboard.

If I remember well the planet data were one of the first, and then moved to archived becaused ESA decided not to use commenrcial; data anymore nad moved to S2. Then those csv did't provide the detections, so only the number of airplanes or ship were included.

Do we really need to include also these data that were avaialbe only for 2020? @aapopescu

[lubojr]

@AlessandroScremin These entries from the config I pointed you to were never in CSV or JSON format, no point in looking to history. In this issue description I pointed to parts of the .js file where the definitions are located. These are not commercial/noncommercial and S2, this is concerning contributing data from NASA and JAXA located on their buckets and using their tiles API.

This is still used in operational eodashboard.org so I was guessing we should migrate it. We are talking about these indicators. https://eodashboard.org/explore?indicator=E13c and https://eodashboard.org/explore?indicator=E13b in both cases we are talking about POIs outside of Europe (USA/Asia). @aapopescu please could you confirm if we should keep listing these data?

If yes, I can just add it myself to these collection (as they are not updated regularly anymore).

[aapopescu]

Thank you, @lubojr . I confirm we keep them. I also notice the data is not loading for some locations. Can we fix this ? thanks!

[lubojr]

@AlessandroScremin In order not to duplicate work, could you please confirm if your team will take care of loading the missing data to E13c_tri and E13b_tri mentioned above to the GeoDB or should we do it? Thank you.

[lubojr]

@dmoglioni @AlessandroScremin Has there been any progress on adding the mentioned NASA+JAXA POIs to the corresponding GeoDB collections? Thank you?

[lubojr]

Agreed on a meeting to also ingest the referenced detections GeoJSONs into corresponding _detections tables.

[dmoglioni]

@lubojr could you point me to the detections url for the missing E13b/c POIs? Thanks.

[lubojr]

@dmoglioni Appologies for a late reply. The features URL if not defined directly in the display key will be placed in the layerNameMapping object - under these two configs "ports" and "airports" https://github.com/eurodatacube/eodash/blob/7ba96c1d1/app/src/config/trilateral.js#L871-L923 as features.url - e.g, https://8ib71h0627.execute-api.us-east-1.amazonaws.com/v1/detections/ship/{site}/{featuresTime}.geojson, where {site} should be substituted by the siteMapping based on AOI_ID - /detections/ship/ URL as https://github.com/eurodatacube/eodash/blob/7ba96c1d1/app/src/config/trilateral.js#L876-L882

and/detections/plane/ URL https://github.com/eurodatacube/eodash/blob/7ba96c1d1/app/src/config/trilateral.js#L920 by https://github.com/eurodatacube/eodash/blob/7ba96c1d1/app/src/config/trilateral.js#L898-L913

[dmoglioni]

As a part of trilateral dashboard migration to STAC, we have noticed a few datasets, which had part of their POIs only defined in the client. We would need these to be migrated to GeoDB to unify the used access.

1. E13c - Ship detections Planet there are 4 POIs defined here and defined here: which should be added to E13c_tri collection. No need to migrate the detections geometries or the planet scenes, just these POI entries and all the individual dates.
2. E13b - Plane detections Planet - same situation as 1), just 14 POIs are missing - defined here - same operation as in 1) Please add to E13b_tri collection.

Could you please integrate these data @dmoglioni ? Thank you

cc @santilland

@lubojr Update: I've ingested the timeseries for the missing POIs for both E13b_tri and E13c_tri. I'll proceed with the generation of the detection data.

[dmoglioni]

@dmoglioni Appologies for a late reply. The features URL if not defined directly in the display key will be placed in the layerNameMapping object - under these two configs "ports" and "airports" https://github.com/eurodatacube/eodash/blob/7ba96c1d1/app/src/config/trilateral.js#L871-L923 as features.url - e.g, https://8ib71h0627.execute-api.us-east-1.amazonaws.com/v1/detections/ship/{site}/{featuresTime}.geojson, where {site} should be substituted by the siteMapping based on AOI_ID - /detections/ship/ URL as https://github.com/eurodatacube/eodash/blob/7ba96c1d1/app/src/config/trilateral.js#L876-L882

and/detections/plane/ URL https://github.com/eurodatacube/eodash/blob/7ba96c1d1/app/src/config/trilateral.js#L920 by https://github.com/eurodatacube/eodash/blob/7ba96c1d1/app/src/config/trilateral.js#L898-L913

@lubojr I've generated the detection files for both ships and airplanes and ingested them in the corresponding data collections (E13c_tri-detections, E13b_tri-detections). Could you please fetch the data and cross-check? Thank you!

[lubojr]

Thank you, I will have a look in the next days and try to integrate.

[lubojr]

@dmoglioni Sorry for late feedback, could you please fill the inputdata field (e.g. ports, airports) into the added POIs in E13b tri and E13c_tri collections so that we match it back to the rendering configuration? Thank you.

[dmoglioni]

@lubojr no problem, I've updated the input_data entries for both data collections.

[lubojr]

Thanks. I have double checked the integration and after minor config update it works fine on our end. I have just one random comment. I have spotted that for example for poi=US01-E13c there are detections missing for the latest day even though they are available for other dates and are present on the source endpoint. Could you please check? I have not gone through other pois carefully but some detections seem missing.

time=2020_05_21 aoi_id=US01 source geojson works: https://8ib71h0627.execute-api.us-east-1.amazonaws.com/v1/detections/ship/ny/2020_05_21.geojson

geodb returns empty: https://xcube-geodb.brockmann-consult.de/eodash/6bf15325-f6a0-4b6a-bf80-a2491753f8f2/eodash_E13c_tri-detections?time=eq.2020-05-21T00:00:00&aoi_id=eq.US01&select=geometry,time

and confirmed by getting only times of detections for E13c-US01:

https://xcube-geodb.brockmann-consult.de/eodash/6bf15325-f6a0-4b6a-bf80-a2491753f8f2/eodash_E13c_tri-detections?&aoi_id=eq.US01&select=time

[
    {
        "time": "2020-01-02T00:00:00"
    },
    {
        "time": "2020-01-09T00:00:00"
    },
    {
        "time": "2020-01-11T00:00:00"
    },
    {
        "time": "2020-01-16T00:00:00"
    },
    {
        "time": "2020-01-17T00:00:00"
    },
    {
        "time": "2020-01-19T00:00:00"
    },
    {
        "time": "2020-01-23T00:00:00"
    },
    {
        "time": "2020-01-24T00:00:00"
    },
    {
        "time": "2020-01-30T00:00:00"
    },
    {
        "time": "2020-05-02T00:00:00"
    },
    {
        "time": "2020-05-05T00:00:00"
    }
]

[lubojr]

@dmoglioni Sorry for early confirmation.

It seems that now the input data for ALL rows of the tables E13b_tri and E13c_tri, are changed to airports and ports, not only the newly added POIs for US and Japan. We still need to differentiate (based on input_data field) the POIs from Europe which are having layers in SH and those from US + Japan which have layers rendered via VEDA API. Could you please fix these? Previously it was I think using the S2L2A input_data field for the EU ones.

[dmoglioni]

@lubojr I restored the previous input_data values associated to the corresponding data_provider: PLES -> Sentinel 2 L2A. Hope now everything is correct.

[dmoglioni]

Thanks. I have double checked the integration and after minor config update it works fine on our end. I have just one random comment. I have spotted that for example for poi=US01-E13c there are detections missing for the latest day even though they are available for other dates and are present on the source endpoint. Could you please check? I have not gone through other pois carefully but some detections seem missing.

time=2020_05_21 aoi_id=US01 source geojson works: https://8ib71h0627.execute-api.us-east-1.amazonaws.com/v1/detections/ship/ny/2020_05_21.geojson

geodb returns empty: https://xcube-geodb.brockmann-consult.de/eodash/6bf15325-f6a0-4b6a-bf80-a2491753f8f2/eodash_E13c_tri-detections?time=eq.2020-05-21T00:00:00&aoi_id=eq.US01&select=geometry,time

and confirmed by getting only times of detections for E13c-US01:

https://xcube-geodb.brockmann-consult.de/eodash/6bf15325-f6a0-4b6a-bf80-a2491753f8f2/eodash_E13c_tri-detections?&aoi_id=eq.US01&select=time

[
    {
        "time": "2020-01-02T00:00:00"
    },
    {
        "time": "2020-01-09T00:00:00"
    },
    {
        "time": "2020-01-11T00:00:00"
    },
    {
        "time": "2020-01-16T00:00:00"
    },
    {
        "time": "2020-01-17T00:00:00"
    },
    {
        "time": "2020-01-19T00:00:00"
    },
    {
        "time": "2020-01-23T00:00:00"
    },
    {
        "time": "2020-01-24T00:00:00"
    },
    {
        "time": "2020-01-30T00:00:00"
    },
    {
        "time": "2020-05-02T00:00:00"
    },
    {
        "time": "2020-05-05T00:00:00"
    }
]

@lubojr I checked your example and this is related to the mixed use of the 'time' formatting used for the ships .geojson files naming. In particular there are both '%Y-%m-%d' and '%Y%m%d' for file names while my parsing script was expecting only the former. For the airplanes case I used the latter, assuming no mixing took place. Could you confirm that?

[dmoglioni]

Thanks. I have double checked the integration and after minor config update it works fine on our end. I have just one random comment. I have spotted that for example for poi=US01-E13c there are detections missing for the latest day even though they are available for other dates and are present on the source endpoint. Could you please check? I have not gone through other pois carefully but some detections seem missing. time=2020_05_21 aoi_id=US01 source geojson works: https://8ib71h0627.execute-api.us-east-1.amazonaws.com/v1/detections/ship/ny/2020_05_21.geojson geodb returns empty: https://xcube-geodb.brockmann-consult.de/eodash/6bf15325-f6a0-4b6a-bf80-a2491753f8f2/eodash_E13c_tri-detections?time=eq.2020-05-21T00:00:00&aoi_id=eq.US01&select=geometry,time and confirmed by getting only times of detections for E13c-US01: https://xcube-geodb.brockmann-consult.de/eodash/6bf15325-f6a0-4b6a-bf80-a2491753f8f2/eodash_E13c_tri-detections?&aoi_id=eq.US01&select=time

[
    {
        "time": "2020-01-02T00:00:00"
    },
    {
        "time": "2020-01-09T00:00:00"
    },
    {
        "time": "2020-01-11T00:00:00"
    },
    {
        "time": "2020-01-16T00:00:00"
    },
    {
        "time": "2020-01-17T00:00:00"
    },
    {
        "time": "2020-01-19T00:00:00"
    },
    {
        "time": "2020-01-23T00:00:00"
    },
    {
        "time": "2020-01-24T00:00:00"
    },
    {
        "time": "2020-01-30T00:00:00"
    },
    {
        "time": "2020-05-02T00:00:00"
    },
    {
        "time": "2020-05-05T00:00:00"
    }
]

@lubojr I checked your example and this is related to the mixed use of the 'time' formatting used for the ships .geojson files naming. In particular there are both '%Y-%m-%d' and '%Y%m%d' for file names while my parsing script was expecting only the former. For the airplanes case I used the latter, assuming no mixing took place. Could you confirm that?

@lubojr following up on what I noticed in my previous comment, I modified the script taking into account both time formatting and reparsed ships detection data. You should be able to fetch all the dates now.

[lubojr]

@dmoglioni Thank you for the investigation! That means also our original implementation was not considering the mixed separators in the file names. Nice catch. I have triggered a catalog rebuild to get the new POIs into the operational dashboard catalog. Thank you for the support. Closing the issue.