search

iobis / pyobis

OBIS Python client
https://iobis.github.io/pyobis
MIT License
14 stars 10 forks source link

Could this package provide the url to the "Full OBIS export"? #128

Open MathewBiddle opened 1 year ago

MathewBiddle commented 1 year ago

I was fiddling around with my notebook and got to thinking "What if I wanted to do a similar activity for all the data in OBIS?" (ie. use the "Full OBIS export" from https://obis.org/data/access/).

After some limited sleuthing, I found this api endpoint which provides information about the full OBIS exports: https://api.obis.org/export?complete=true

Looking at the most recent export in parquet, you can see this subset of information about the parquet file download:

{"id":"0a04e2d6-d55a-445a-b8f2-f1bd0fa4bb9c",
"created":"2023-02-11T19:09:18.989Z",
"creator":"Ocean Biodiversity Information System (OBIS)",
"title":"Full OBIS export 2023-02-08",
"modified":null,
"doi":null,
"abstract":null,
"occurrence_records":null,
"complete":true,
"s3path":"exports/obis_20230208.parquet",
"type":"parquet"}

I'm curious if it would be useful to add a small feature to this package to simply provide the url to where a user can download the "Full OBIS export". In essence, use the export endpoint to compile what the download url should be?

I think the url for the export above would look like: https://obis-datasets.ams3.digitaloceanspaces.com/exports/obis_20230208.parquet

ayushanand18 commented 1 year ago

This would probably be a great addition to the package. But I am not sure which module to dedicate this to.