Unable to ingest private metadata

[drnextgis]

Although the items table contains private column, the pypgstac loader is unable to ingest private metadata.

[zacharyDez]

@drnextgis, could you provide some information on how to reproduce the issue?

[drnextgis]

Just examining the loader code reveals that the private property isn't handled at all. You can replicate the issue easily:

1. Follow the instructions here to create a collection named noaa-emergency-response.
2. Create a items.json file and include the private object:

{"type":"Feature","private":{"a":1},"stac_version":"1.0.0","id":"20200307aC0852700w360900","properties":{"event":"Nashville Tornado","datetime":"2020-03-07T00:00:00Z"},"geometry":{"type":"Polygon","coordinates":[[[-85.4501,36.1501],[-85.4501,36.1249],[-85.4249,36.1249],[-85.4249,36.1501],[-85.4501,36.1501]]]},"links":[{"rel":"collection","href":"noaa-emergency-response","type":"application/json"}],"assets":{"cog":{"href":"s3://noaa-eri-pds/2020_Nashville_Tornado/20200307a_RGB/20200307aC0852700w360900n.tif","type":"image/tiff; application=geotiff; profile=cloud-optimized"}},"bbox":[-85.4501,36.1249,-85.4249,36.1501],"stac_extensions":[],"collection":"noaa-emergency-response"}
{"type":"Feature","private":{"b":2},"stac_version":"1.0.0","id":"20200307aC0852700w361030","properties":{"event":"Nashville Tornado","datetime":"2020-03-07T00:00:00Z"},"geometry":{"type":"Polygon","coordinates":[[[-85.4501,36.1751],[-85.4501,36.1499],[-85.4249,36.1499],[-85.4249,36.1751],[-85.4501,36.1751]]]},"links":[{"rel":"collection","href":"noaa-emergency-response","type":"application/json"}],"assets":{"cog":{"href":"s3://noaa-eri-pds/2020_Nashville_Tornado/20200307a_RGB/20200307aC0852700w361030n.tif","type":"image/tiff; application=geotiff; profile=cloud-optimized"}},"bbox":[-85.4501,36.1499,-85.4249,36.1751],"stac_extensions":[],"collection":"noaa-emergency-response"}

3. Proceed to load it into the database:

$ pypgstac load items items.njson \
    --dsn postgresql://username:password@localhost:5439/postgis \
    --method insert

4. Connect to the database and ensure the private column is empty:

postgis=# select private from items;
 private 
---------

(2 rows)

[vincentsarago]

@drnextgis are you using hydratation/dehydratation?

are you sure that private key are allowed in a stac Item? I don't think it's allowed and they might just be ignored

[drnextgis]

But this is the case. Despite the presence of the private column in the items table, there appears to be no way to populate it using pypgstac. The only method I can see is resorting to raw SQL queries. Am I missing something?

[mmcfarland]

My understanding for the private column in items (and also maybe on collections) was in anticipation of future work, and the column is not currently used by pgstac sql or pypgstac (e.g., I'm not sure you could do anything with it through existing pgstac interfaces even if you were able to load it in). The rationale was there was a breaking version jump (0.7 -> 0.8) occurring and @bitner wanted to get the schema change in for it, with follow up work to be done within the 0.8 series.

[bitner]

Correct, currently there is no plumbing to do anything with the private column or for the loader to populate it. It is there for future development to allow for ad hoc storage of anything that does not belong in the public stac metadata (ie user name who created the record).

[drnextgis]

Thank you all for the comprehensive response. We currently require a way to store private metadata alongside STAC items. Therefore, I would like to implement the ability to populate this column using the pypgstac loader. What do you think?

[bitner]

@drnextgis if you'd like to take a stab at it, please do! If you'd like to pair on anything to get going, let me know!