Searchable database for CODA

[ChrisC28]

Can we produce a searchable database for CODA?

Issues to think about:

• File format? Parquet?
• Do we need to change the format or structure of the underlying CODA netcdf files (ie. is grouped yearly, by data originator and platform optimal for a searchable database? Would day-by-day or profile-by-profile work better?)
• How do we interogate? Polars? https://docs.pola.rs/

[Thomas-Moore-Creative]

Yesterday @ChrisC28 & @BecCowley we discussed this issue and talked about the following use case:

for all the ARGO (or CTD, or ... ) data type in a given region, over a time period, over a depth range, with given variable . .

• quickly and easily make plots of where this data is on a map
• grab all the data, record by record in a tabular format
• grab all the data as analysis-ready profiles

[Thomas-Moore-Creative]

starting with 2012 as an example: /g/data/es60/users/thomas_moore/CODA/2012

(base) tm4888@gadi-login-04 /g/data/es60/users/thomas_moore/CODA/2012 ls -l
total 119157796
-rw-rw-r-- 1 tm4888 es60     2621287 May 28 16:13 AIMS_CODA_2012_ctd.nc
-rw-rw-r-- 1 tm4888 es60    39738448 May 21 10:40 MNF_CODA_2012_ctd.nc
-rw-rw-r-- 1 tm4888 es60    49086082 May 21 10:39 RAN_CODA_2012_ctd.nc
-rw-rw-r-- 1 tm4888 es60 15309446859 May 15 21:31 WOD2018_CODA_2012_ctd.nc
-rw-rw-r-- 1 tm4888 es60 33029760782 May 15 23:46 WOD2018_CODA_2012_gld.nc
-rw-rw-r-- 1 tm4888 es60    85704294 May 15 23:05 WOD2018_CODA_2012_mrb.nc
-rw-rw-r-- 1 tm4888 es60   129700151 May 15 21:21 WOD2018_CODA_2012_osd.nc
-rw-rw-r-- 1 tm4888 es60 71305005747 May 15 22:54 WOD2018_CODA_2012_pfl.nc
-rw-rw-r-- 1 tm4888 es60  2066312325 May 15 21:35 WOD2018_CODA_2012_xbt.nc

[Thomas-Moore-Creative]

Heya @ChrisC28 do variables like Temperature_WODflag need to be float64?