Add iNaturalist ETL to Rahti

[mikkohei13]

As authority/researcher, I want to get uncoarsened iNat data, so that I can use it for my work.

As FinBIF admin, I want the iNat ETL process be automatic and frequent, so that fresh data is available.

https://trello.com/c/KJgc66Gt/17-inaturalist-etl-mikko-mikko-will-m

---

22.8.2024:

Next: Set up on Rahti 2. Keep data on Allas. Will will check if we need more resources.

19.8.2024 We need more resources! Need to have a discussion, what we need.

• [x] Add Finland EEZ area
• [ ] Decide and implement fully automatic ETL system
• [x] Integrate sensitive data dump to process. For each obs that is obscured check if there is detailed data
• [ ] Document process of adding sensitive data to Rahti/Allas?
• [ ] Future dev: sync non-Finnish observations of users who have linked their iNat user account with laji.fi
• [ ] Future dev: Handle deletions based on bi-yearly data dumps

[mikkohei13]

UK data ingestion & verification: https://irecord.org.uk/sites/default/files/docs_pdf/iRecord%20guidance_verifying%20iNaturalist%20records%20v1.pdf