search

google-code-export / stoqs

Automatically exported from code.google.com/p/stoqs
GNU General Public License v3.0
1 stars 1 forks source link

Create procedure for loading plankton net tow Samples #60

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
Water samples collected by Niskin bottle trips or Dorado Gulper gulps are 
automatically loaded into STOQS databases using records from files created when 
these samples are taken. There is no such automated process for recording when 
and where plankton nets are towed.

All of the SIMZ cruises have plankton net tow data that need to be brought into 
their respective databases. By convention net tows were done following each CTD 
cast, at a consistent depth, for a consistent duration.

The Sample records for net tows could be loaded with a command-line script that 
accepts as arguments:

--database
--platform
--startTimeFollowingCast
--duration
--depth
--volume
--filterDiameter
--filterPoreSize
--laboratory
--researcher
[--startCast
--endCast]

In a way, a plankton net tow is similar to a subsample where water is passed 
through a filter. We would like to record all of the relevant metadata for what 
ends up in the cod end of the net. 

Once these initial Samples are in the database we will be able to load the net 
tow laboratory analysis data using the same method we currently use for loading 
subsample data for Niskin and Gulper Samples.

Original issue reported on code.google.com by MBARIm...@gmail.com on 2 Feb 2015 at 4:55

GoogleCodeExporter commented 9 years ago 
"following each CTD cast, at a consistent depth," - Rather, prior to each CTD 
cast, and varying depths.

We have start times for most net tows.  Where we don't have a time, we can 
subtract 30 minutes from the CTD cast times and use those numbers as 
approximate start times.

Original comment by jbjhar...@gmail.com on 2 Feb 2015 at 5:19

GoogleCodeExporter commented 9 years ago 
Are there multiple net tows at different depths at a CTD cast station? Or are 
there *always* just one net tow per CTD station?

Original comment by MBARIm...@gmail.com on 2 Feb 2015 at 6:34

GoogleCodeExporter commented 9 years ago 
Rather than execute a script with a set of arguments to load the parent Sample 
records, it would be better load them from a .csv file. Most of the information 
needed is already in the subsample excel files being provided, e.g.:

Cruise,Bottle Number,Sample Type,Depth [m],Sample Volume [mL],Filter Diameter 
[mm],Filter Pore Size [uM],Parameter Name,Parameter Value,Parameter 
Units,Laboratory,Researcher,Analysis Method,Comment Name,Comment Value
simz2013c01,NetTow,vertical net tow,15,,,200 µm,B1006 barnacles,0.115,OD A450 
nm,Vrijenhoek,Harvey,Sandwich Hybridization Assay,,
simz2013c01,NetTow,vertical net tow,15,,,200 µm,M2B mussels,0.072,OD A450 
nm,Vrijenhoek,Harvey,Sandwich Hybridization Assay,,
simz2013c01,NetTow,vertical net tow,15,,,200 µm,GCRAB Carcinus,0.065,OD A450 
nm,Vrijenhoek,Harvey,Sandwich Hybridization Assay,,
simz2013c01,NetTow,vertical net tow,15,,,200 µm,CAL903 calanoida,0.12,OD A450 
nm,Vrijenhoek,Harvey,Sandwich Hybridization Assay,,
simz2013c01,NetTow,vertical net tow,15,,,200 µm,CAL1939 calanoida,4,OD A450 
nm,Vrijenhoek,Harvey,Sandwich Hybridization Assay,,
simz2013c01,NetTow,vertical net tow,15,,,200 µm,POD1951 podoplea,0.144,OD A450 
nm,Vrijenhoek,Harvey,Sandwich Hybridization Assay,,
simz2013c01,NetTow,vertical net tow,15,,,200 µm,SAB1182 sabellariidae,0.066,OD 
A450 nm,Vrijenhoek,Harvey,Sandwich Hybridization Assay,,
simz2013c01,NetTow,vertical net tow,15,,,200 µm,CRAB903 brachyura,0.077,OD 
A450 nm,Vrijenhoek,Harvey,Sandwich Hybridization Assay,,
simz2013c01,NetTow,vertical net tow,15,,,200 µm,OS1022 polychaeta,0.237,OD 
A450 nm,Vrijenhoek,Harvey,Sandwich Hybridization Assay,,
simz2013c01,NetTow,vertical net tow,15,,,200 µm,SPI1181 spionidae,0.05,OD A450 
nm,Vrijenhoek,Harvey,Sandwich Hybridization Assay,,
simz2013c02,NetTow,vertical net tow,30,,,200 µm,B1006 barnacles,0.087,OD A450 
nm,Vrijenhoek,Harvey,Sandwich Hybridization Assay,,
simz2013c02,NetTow,vertical net tow,30,,,200 µm,M2B mussels,0.079,OD A450 
nm,Vrijenhoek,Harvey,Sandwich Hybridization Assay,,
...

The rest of the information needed is already in the database associated with 
the CTD cast Activity names (simz2013c01, simz2013c02, ...).

I propose a 2 step process:

1. Run a script to combine the subsample information (which has important 
information such as tow depth) with database information to produce a parent 
Sample .csv file. This file can then be hand edited, if needed.

2. Execute a script to load data from this .csv file into the database

The subsample data can then be loaded - linking to the parent Samples.

Original comment by MBARIm...@gmail.com on 3 Feb 2015 at 4:39

GoogleCodeExporter commented 9 years ago

Original comment by MBARIm...@gmail.com on 10 Feb 2015 at 12:35

GoogleCodeExporter commented 9 years ago 
The "nettow.py" script has been written to perform these 2 steps. Here is the 
usage note:

(venv-stoqs)[mccann@localhost stoqshg]$ loaders/nettow.py -h
usage: nettow.py [-h] -d DATABASE [-s SUBSAMPLEFILE] [-c CSVFILE]
                 [-m SUBTRACTMINUTES] [--laboratory LABORATORY]
                 [--researcher RESEARCHER] [--purpose PURPOSE] [-l LOADFILE]
                 [-v [{1,2,3}]]

Script to load parent Samples for Tow Net data

optional arguments:
  -h, --help            show this help message and exit
  -d DATABASE, --database DATABASE
                        Database alias
  -s SUBSAMPLEFILE, --subsampleFile SUBSAMPLEFILE
                        File name containing analysis data from net tows in STOQS subsample format
  -c CSVFILE, --csvFile CSVFILE
                        Output comma separated value file containing parent Sample data
  -m SUBTRACTMINUTES, --subtractMinutes SUBTRACTMINUTES
                        Subtract these number of minutes from start of CTD cast for net tow time
  --laboratory LABORATORY
                        Laboratory responsible for the Samples
  --researcher RESEARCHER
                        Researcher responsible for the Samples
  --purpose PURPOSE     Purpose of the Sample
  -l LOADFILE, --loadFile LOADFILE
                        Load parent Sample data into database
  -v [{1,2,3}], --verbose [{1,2,3}]
                        Turn on verbose output. Higher number = more output.

Example:

  Step 1 - Create .cvs file of parent Sample information:
    loaders/nettow.py --database stoqs_simz_aug2013_t --subsampleFile 2013_SIMZ_TowNets_STOQS.csv --csvFile 2013_SIMZ_TowNet_ParentSamples.csv

  Step 2 - Load parent Sample information:
    loaders/nettow.py --database stoqs_simz_aug2013_t --loadFile 2013_SIMZ_TowNet_ParentSamples.csv

If running from cde-package replace ".py" with ".py.cde".

Original comment by MBARIm...@gmail.com on 11 Feb 2015 at 12:15

GoogleCodeExporter commented 9 years ago

Original comment by MBARIm...@gmail.com on 11 Feb 2015 at 3:46

GoogleCodeExporter commented 9 years ago 
Loaded Net Tow parent Samples into MBARI's production database using nettow.py 
script:

(venv-stoqs)[stoqsadm@kraken stoqshg]$ loaders/nettow.py --database 
stoqs_simz_aug2013 --subsampleFile 
loaders/MolecularEcology/SIMZAug2013/2013_SIMZ_TowNets_STOQS.csv --csvFile 
loaders/MolecularEcology/SIMZAug2013/2013_SIMZ_TowNet_ParentSamples.csv
INFO 2015-02-12 16:39:39,927 nettow nettow.py _db_join():85 Joining subsample 
information with cast data from the database using subtractMinutes = 30

(venv-stoqs)[stoqsadm@kraken stoqshg]$ cat 
loaders/MolecularEcology/SIMZAug2013/2013_SIMZ_TowNet_ParentSamples.csv
Cast,name,depth,sampletype,volume,filterdiameter,filterporesize,longitude,latitu
de,datetime_gmt
simz2013c01,,15,vertical net 
tow,,,200.0,-121.9363333,36.9458333,2013-08-12T16:34:16
simz2013c02,,30,vertical net 
tow,,,200.0,-121.9961667,36.8948333,2013-08-12T19:05:05
simz2013c03,,30,vertical net 
tow,,,200.0,-121.9656667,36.9246667,2013-08-12T20:59:50
simz2013c04,,15,vertical net 
tow,,,200.0,-121.9331667,36.9465,2013-08-13T15:22:41
simz2013c05,,30,vertical net tow,,,200.0,-121.989,36.9018333,2013-08-13T18:00:28
simz2013c06,,30,vertical net 
tow,,,200.0,-121.9621667,36.9255,2013-08-13T20:07:26
simz2013c07,,15,vertical net 
tow,,,200.0,-121.9276667,36.9493333,2013-08-14T15:06:48
simz2013c08,,15,vertical net tow,,,200.0,-121.967,36.9256667,2013-08-14T16:41:44
simz2013c09,,30,vertical net 
tow,,,200.0,-122.0041667,36.9001667,2013-08-14T18:38:55
simz2013c10,,30,vertical net tow,,,200.0,-121.916,36.8848333,2013-08-14T20:37:48
simz2013c11,,10,vertical net 
tow,,,200.0,-121.9331667,36.9486667,2013-08-15T14:42:51
simz2013c12,,20,vertical net tow,,,200.0,-121.9675,36.9275,2013-08-15T16:13:47
simz2013c13,,30,vertical net 
tow,,,200.0,-121.9985,36.9021667,2013-08-15T18:01:54
simz2013c14,,25,vertical net 
tow,,,200.0,-121.9163333,36.8858333,2013-08-15T20:12:26
simz2013c15,,10,vertical net 
tow,,,200.0,-121.9316667,36.9511667,2013-08-16T14:53:58
simz2013c16,,20,vertical net 
tow,,,200.0,-121.9676667,36.9225,2013-08-16T16:22:02
simz2013c17,,30,vertical net 
tow,,,200.0,-121.9983333,36.9046667,2013-08-16T18:26:34
simz2013c18,,25,vertical net 
tow,,,200.0,-121.9191667,36.8871667,2013-08-16T19:51:56

(venv-stoqs)[stoqsadm@kraken stoqshg]$ loaders/nettow.py --database 
stoqs_simz_aug2013 --loadFile 
loaders/MolecularEcology/SIMZAug2013/2013_SIMZ_TowNet_ParentSamples.csv
INFO 2015-02-12 16:40:38,661 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,677 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,692 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,707 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,722 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,738 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,753 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,768 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,783 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,799 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,814 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,829 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,844 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,859 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,874 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,889 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,904 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,920 nettow nettow.py load_samples():218 Loaded Sample 1

Confirm that new NetTow Activities are in the database:
http://odss.mbari.org/canon/stoqs_simz_aug2013/activity.html?name__contains=Net
http://odss.mbari.org/canon/stoqs_simz_aug2013/sample.html?sampletype__name__con
tains=Net

And examine the data in the STOQS UI: 
http://odss.mbari.org/canon/stoqs_simz_aug2013/

Subsamples can now be loaded from our regular load scripts as they now have 
parent Samples to link to.

More work needs to be done in the UI to properly represent integrative NetTow 
sampling, yet this Issue is being marked as fixed.

Original comment by MBARIm...@gmail.com on 12 Feb 2015 at 5:03