search

npolar / marine-db

https://doi.org/10.21334/marine-db
0 stars 0 forks source link

IOPAN legacy protist inventory #56

Closed cnrdh closed 2 years ago

cnrdh commented 2 years ago

We need to make sure that we include all known expeditions and input files and cover the relevant sample types (here: phytoplankton, microplankton,handnet).

cnrdh commented 2 years ago 

conrad@nordfjellet:~/npolar/marine-db$ cat data/input/iopan/protist-biodiversity/log/line-counts.txt 
1469 data/deposit/iopan/protist-biodiversity/alkekonge-2009.tsv
1468 data/input/iopan/protist-biodiversity/ndjson/alkekonge-2009.ndjson
1469 data/input/iopan/protist-biodiversity/tsv/alkekonge-2009.tsv

1110 data/deposit/iopan/protist-biodiversity/merclim-2009.tsv
1109 data/input/iopan/protist-biodiversity/ndjson/merclim-2009.ndjson
1110 data/input/iopan/protist-biodiversity/tsv/merclim-2009.tsv

962 data/deposit/iopan/protist-biodiversity/alkekonge2010.tsv
961 data/input/iopan/protist-biodiversity/ndjson/alkekonge2010.ndjson
962 data/input/iopan/protist-biodiversity/tsv/alkekonge2010.tsv

3444 data/deposit/iopan/protist-biodiversity/ice2010_konghau_database_complete_phytoplankton_niskin.tsv
3443 data/input/iopan/protist-biodiversity/ndjson/ice2010_konghau_database_complete_phytoplankton_niskin.ndjson
3444 data/input/iopan/protist-biodiversity/tsv/ice2010_konghau_database_complete_phytoplankton_niskin.tsv

153 data/deposit/iopan/protist-biodiversity/ice2010_konghau_database_complete_handnet.tsv
152 data/input/iopan/protist-biodiversity/ndjson/ice2010_konghau_database_complete_handnet.ndjson
153 data/input/iopan/protist-biodiversity/tsv/ice2010_konghau_database_complete_handnet.tsv

1538 data/deposit/iopan/protist-biodiversity/mosj_2011.tsv
1537 data/input/iopan/protist-biodiversity/ndjson/mosj_2011.ndjson
1538 data/input/iopan/protist-biodiversity/tsv/mosj_2011.tsv

1639 data/deposit/iopan/protist-biodiversity/mosj2012.tsv
1638 data/input/iopan/protist-biodiversity/ndjson/mosj2012.ndjson
1639 data/input/iopan/protist-biodiversity/tsv/mosj2012.tsv

2671 data/deposit/iopan/protist-biodiversity/ice2012wcolumndatabase.tsv
2665 data/input/iopan/protist-biodiversity/ndjson/ice2012wcolumndatabase.ndjson
2666 data/input/iopan/protist-biodiversity/tsv/ice2012wcolumndatabase.tsv

41 data/deposit/iopan/protist-biodiversity/pnc2012-protist.tsv
40 data/input/iopan/protist-biodiversity/ndjson/pnc2012-protist.ndjson
41 data/input/iopan/protist-biodiversity/tsv/pnc2012-protist.tsv

2269 data/deposit/iopan/protist-biodiversity/mosj2013.tsv
2268 data/input/iopan/protist-biodiversity/ndjson/mosj2013.ndjson
2269 data/input/iopan/protist-biodiversity/tsv/mosj2013.tsv

1390 data/deposit/iopan/protist-biodiversity/mosj2014_pht.tsv
1388 data/input/iopan/protist-biodiversity/ndjson/mosj2014_pht.ndjson
1389 data/input/iopan/protist-biodiversity/tsv/mosj2014_pht.tsv

922 data/deposit/iopan/protist-biodiversity/ice2014_pht.tsv
919 data/input/iopan/protist-biodiversity/ndjson/ice2014_pht.ndjson
920 data/input/iopan/protist-biodiversity/tsv/ice2014_pht.tsv

353 data/deposit/iopan/protist-biodiversity/mosj2015.tsv
350 data/input/iopan/protist-biodiversity/ndjson/mosj2015.ndjson
351 data/input/iopan/protist-biodiversity/tsv/mosj2015.tsv

4843 data/deposit/iopan/protist-biodiversity/n-ice2015_pht.tsv
4842 data/input/iopan/protist-biodiversity/ndjson/n-ice2015_pht.ndjson
4843 data/input/iopan/protist-biodiversity/tsv/n-ice2015_pht.tsv

7863 data/deposit/iopan/protist-biodiversity/n-ice2015_iat.tsv
7860 data/input/iopan/protist-biodiversity/ndjson/n-ice2015_iat.ndjson
7861 data/input/iopan/protist-biodiversity/tsv/n-ice2015_iat.tsv

3258 data/deposit/iopan/protist-biodiversity/mosj2016_pht.tsv
3257 data/input/iopan/protist-biodiversity/ndjson/mosj2016_pht.ndjson
3258 data/input/iopan/protist-biodiversity/tsv/mosj2016_pht.tsv

769 data/deposit/iopan/protist-biodiversity/mosj2016_mit.tsv
768 data/input/iopan/protist-biodiversity/ndjson/mosj2016_mit.ndjson
769 data/input/iopan/protist-biodiversity/tsv/mosj2016_mit.tsv

315 data/deposit/iopan/protist-biodiversity/mosj2016_han.tsv
314 data/input/iopan/protist-biodiversity/ndjson/mosj2016_han.ndjson
315 data/input/iopan/protist-biodiversity/tsv/mosj2016_han.tsv

1580 data/deposit/iopan/protist-biodiversity/arcex2016_pht.tsv
1579 data/input/iopan/protist-biodiversity/ndjson/arcex2016_pht.ndjson
1580 data/input/iopan/protist-biodiversity/tsv/arcex2016_pht.tsv

347 data/deposit/iopan/protist-biodiversity/glacierfront_tw-ice2017_pht.tsv
346 data/input/iopan/protist-biodiversity/ndjson/glacierfront_tw-ice2017_pht.ndjson
347 data/input/iopan/protist-biodiversity/tsv/glacierfront_tw-ice2017_pht.tsv

75 data/deposit/iopan/protist-biodiversity/glacierfront_tw-ice2017_han.tsv
74 data/input/iopan/protist-biodiversity/ndjson/glacierfront_tw-ice2017_han.ndjson
75 data/input/iopan/protist-biodiversity/tsv/glacierfront_tw-ice2017_han.tsv

Deposited TSV lines:
37011
NDJSON lines:
36978
DwC TSV (with events) lines:
35829 data/input/iopan/protist-biodiversity/tsv/iopan_legacy_protist_dwc-v2021-11-09.tsv
cnrdh commented 2 years ago

Rejected: conrad@nordfjellet:~/npolar/marine-db$ cat data/input/iopan/protist-biodiversity/log/error-schema-rejected.ndjson | ndjson-map '[d.expedition,d.fieldNumber,d.scientificName]'


["ICE2012","ICE12-39","„pierscionek”"]
["ICE2012","ICE12-41","„pierscionek”"]
["ICE2012","ICE12-37","„pierscionek”"]
["ICE2012","ICE12-775","„pierscionek”"]
["ICE2012","no","takson"]
["MOSJ2014","PHT-012","detrytus"]
["MOSJ2015","PHT-29","„Bodo”"]
["MOSJ2015","PHT-12","sed"]
["N-ICE2015","IAT-302","cella"]
["N-ICE2015","IAT-303","cella"]