woodRock / psychic-invention

NZODN Data Ingestion Project
0 stars 0 forks source link

Postgres Schema #5

Closed woodRock closed 3 years ago

woodRock commented 3 years ago

Goal

Create a schema for the biological data-set on the wellimos server.

Tasklist

Success Criteria

There is a postgres schema for the biologoical data-set. This includes a table which refers to the file path for downloads.

woodRock commented 3 years ago

We have replicated a the table from the co2 schema. This is a very similar dataset that offers the same functionality. We replicate this table in order to reproduce the same functionality for the Biological dataset.

woodRock commented 3 years ago

However, we could not add an entry to store the filepath and url for the zipfile, because we don't have permission to write to the relevant directory. The zipfiles available for download are published in the /data/niwa/publish directory. We raised this in a separate issue #8.

woodRock commented 3 years ago

Update this table to use an auto-incrementing sequence for its ID. This is slightly overkilled for this dataset. But it is good practice to ensure unique primary keys. It may be useful to replicate this table for ingestion in the future. The steps do to this were documented on StackOverflow.

woodRock commented 3 years ago

Added a record to this schema for the zip file #8. This should be all the relevant setup in order to make the zip file both accessible through the web service and to the NZODN. Although I have emailed Glenn to seek some clarification on this.

woodRock commented 3 years ago

Glenn replied to the email. He said it should be the relevant setup in order to get the Biological dataset working.

woodRock commented 3 years ago

We created a script which recreates this schema from scratch each time it is run. This ensures that our results are reproducible and consistent.