Closed GoogleCodeExporter closed 9 years ago
"following each CTD cast, at a consistent depth," - Rather, prior to each CTD
cast, and varying depths.
We have start times for most net tows. Where we don't have a time, we can
subtract 30 minutes from the CTD cast times and use those numbers as
approximate start times.
Original comment by jbjhar...@gmail.com
on 2 Feb 2015 at 5:19
Are there multiple net tows at different depths at a CTD cast station? Or are
there *always* just one net tow per CTD station?
Original comment by MBARIm...@gmail.com
on 2 Feb 2015 at 6:34
Rather than execute a script with a set of arguments to load the parent Sample
records, it would be better load them from a .csv file. Most of the information
needed is already in the subsample excel files being provided, e.g.:
Cruise,Bottle Number,Sample Type,Depth [m],Sample Volume [mL],Filter Diameter
[mm],Filter Pore Size [uM],Parameter Name,Parameter Value,Parameter
Units,Laboratory,Researcher,Analysis Method,Comment Name,Comment Value
simz2013c01,NetTow,vertical net tow,15,,,200 µm,B1006 barnacles,0.115,OD A450
nm,Vrijenhoek,Harvey,Sandwich Hybridization Assay,,
simz2013c01,NetTow,vertical net tow,15,,,200 µm,M2B mussels,0.072,OD A450
nm,Vrijenhoek,Harvey,Sandwich Hybridization Assay,,
simz2013c01,NetTow,vertical net tow,15,,,200 µm,GCRAB Carcinus,0.065,OD A450
nm,Vrijenhoek,Harvey,Sandwich Hybridization Assay,,
simz2013c01,NetTow,vertical net tow,15,,,200 µm,CAL903 calanoida,0.12,OD A450
nm,Vrijenhoek,Harvey,Sandwich Hybridization Assay,,
simz2013c01,NetTow,vertical net tow,15,,,200 µm,CAL1939 calanoida,4,OD A450
nm,Vrijenhoek,Harvey,Sandwich Hybridization Assay,,
simz2013c01,NetTow,vertical net tow,15,,,200 µm,POD1951 podoplea,0.144,OD A450
nm,Vrijenhoek,Harvey,Sandwich Hybridization Assay,,
simz2013c01,NetTow,vertical net tow,15,,,200 µm,SAB1182 sabellariidae,0.066,OD
A450 nm,Vrijenhoek,Harvey,Sandwich Hybridization Assay,,
simz2013c01,NetTow,vertical net tow,15,,,200 µm,CRAB903 brachyura,0.077,OD
A450 nm,Vrijenhoek,Harvey,Sandwich Hybridization Assay,,
simz2013c01,NetTow,vertical net tow,15,,,200 µm,OS1022 polychaeta,0.237,OD
A450 nm,Vrijenhoek,Harvey,Sandwich Hybridization Assay,,
simz2013c01,NetTow,vertical net tow,15,,,200 µm,SPI1181 spionidae,0.05,OD A450
nm,Vrijenhoek,Harvey,Sandwich Hybridization Assay,,
simz2013c02,NetTow,vertical net tow,30,,,200 µm,B1006 barnacles,0.087,OD A450
nm,Vrijenhoek,Harvey,Sandwich Hybridization Assay,,
simz2013c02,NetTow,vertical net tow,30,,,200 µm,M2B mussels,0.079,OD A450
nm,Vrijenhoek,Harvey,Sandwich Hybridization Assay,,
...
The rest of the information needed is already in the database associated with
the CTD cast Activity names (simz2013c01, simz2013c02, ...).
I propose a 2 step process:
1. Run a script to combine the subsample information (which has important
information such as tow depth) with database information to produce a parent
Sample .csv file. This file can then be hand edited, if needed.
2. Execute a script to load data from this .csv file into the database
The subsample data can then be loaded - linking to the parent Samples.
Original comment by MBARIm...@gmail.com
on 3 Feb 2015 at 4:39
Original comment by MBARIm...@gmail.com
on 10 Feb 2015 at 12:35
The "nettow.py" script has been written to perform these 2 steps. Here is the
usage note:
(venv-stoqs)[mccann@localhost stoqshg]$ loaders/nettow.py -h
usage: nettow.py [-h] -d DATABASE [-s SUBSAMPLEFILE] [-c CSVFILE]
[-m SUBTRACTMINUTES] [--laboratory LABORATORY]
[--researcher RESEARCHER] [--purpose PURPOSE] [-l LOADFILE]
[-v [{1,2,3}]]
Script to load parent Samples for Tow Net data
optional arguments:
-h, --help show this help message and exit
-d DATABASE, --database DATABASE
Database alias
-s SUBSAMPLEFILE, --subsampleFile SUBSAMPLEFILE
File name containing analysis data from net tows in STOQS subsample format
-c CSVFILE, --csvFile CSVFILE
Output comma separated value file containing parent Sample data
-m SUBTRACTMINUTES, --subtractMinutes SUBTRACTMINUTES
Subtract these number of minutes from start of CTD cast for net tow time
--laboratory LABORATORY
Laboratory responsible for the Samples
--researcher RESEARCHER
Researcher responsible for the Samples
--purpose PURPOSE Purpose of the Sample
-l LOADFILE, --loadFile LOADFILE
Load parent Sample data into database
-v [{1,2,3}], --verbose [{1,2,3}]
Turn on verbose output. Higher number = more output.
Example:
Step 1 - Create .cvs file of parent Sample information:
loaders/nettow.py --database stoqs_simz_aug2013_t --subsampleFile 2013_SIMZ_TowNets_STOQS.csv --csvFile 2013_SIMZ_TowNet_ParentSamples.csv
Step 2 - Load parent Sample information:
loaders/nettow.py --database stoqs_simz_aug2013_t --loadFile 2013_SIMZ_TowNet_ParentSamples.csv
If running from cde-package replace ".py" with ".py.cde".
Original comment by MBARIm...@gmail.com
on 11 Feb 2015 at 12:15
Original comment by MBARIm...@gmail.com
on 11 Feb 2015 at 3:46
Loaded Net Tow parent Samples into MBARI's production database using nettow.py
script:
(venv-stoqs)[stoqsadm@kraken stoqshg]$ loaders/nettow.py --database
stoqs_simz_aug2013 --subsampleFile
loaders/MolecularEcology/SIMZAug2013/2013_SIMZ_TowNets_STOQS.csv --csvFile
loaders/MolecularEcology/SIMZAug2013/2013_SIMZ_TowNet_ParentSamples.csv
INFO 2015-02-12 16:39:39,927 nettow nettow.py _db_join():85 Joining subsample
information with cast data from the database using subtractMinutes = 30
(venv-stoqs)[stoqsadm@kraken stoqshg]$ cat
loaders/MolecularEcology/SIMZAug2013/2013_SIMZ_TowNet_ParentSamples.csv
Cast,name,depth,sampletype,volume,filterdiameter,filterporesize,longitude,latitu
de,datetime_gmt
simz2013c01,,15,vertical net
tow,,,200.0,-121.9363333,36.9458333,2013-08-12T16:34:16
simz2013c02,,30,vertical net
tow,,,200.0,-121.9961667,36.8948333,2013-08-12T19:05:05
simz2013c03,,30,vertical net
tow,,,200.0,-121.9656667,36.9246667,2013-08-12T20:59:50
simz2013c04,,15,vertical net
tow,,,200.0,-121.9331667,36.9465,2013-08-13T15:22:41
simz2013c05,,30,vertical net tow,,,200.0,-121.989,36.9018333,2013-08-13T18:00:28
simz2013c06,,30,vertical net
tow,,,200.0,-121.9621667,36.9255,2013-08-13T20:07:26
simz2013c07,,15,vertical net
tow,,,200.0,-121.9276667,36.9493333,2013-08-14T15:06:48
simz2013c08,,15,vertical net tow,,,200.0,-121.967,36.9256667,2013-08-14T16:41:44
simz2013c09,,30,vertical net
tow,,,200.0,-122.0041667,36.9001667,2013-08-14T18:38:55
simz2013c10,,30,vertical net tow,,,200.0,-121.916,36.8848333,2013-08-14T20:37:48
simz2013c11,,10,vertical net
tow,,,200.0,-121.9331667,36.9486667,2013-08-15T14:42:51
simz2013c12,,20,vertical net tow,,,200.0,-121.9675,36.9275,2013-08-15T16:13:47
simz2013c13,,30,vertical net
tow,,,200.0,-121.9985,36.9021667,2013-08-15T18:01:54
simz2013c14,,25,vertical net
tow,,,200.0,-121.9163333,36.8858333,2013-08-15T20:12:26
simz2013c15,,10,vertical net
tow,,,200.0,-121.9316667,36.9511667,2013-08-16T14:53:58
simz2013c16,,20,vertical net
tow,,,200.0,-121.9676667,36.9225,2013-08-16T16:22:02
simz2013c17,,30,vertical net
tow,,,200.0,-121.9983333,36.9046667,2013-08-16T18:26:34
simz2013c18,,25,vertical net
tow,,,200.0,-121.9191667,36.8871667,2013-08-16T19:51:56
(venv-stoqs)[stoqsadm@kraken stoqshg]$ loaders/nettow.py --database
stoqs_simz_aug2013 --loadFile
loaders/MolecularEcology/SIMZAug2013/2013_SIMZ_TowNet_ParentSamples.csv
INFO 2015-02-12 16:40:38,661 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,677 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,692 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,707 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,722 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,738 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,753 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,768 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,783 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,799 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,814 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,829 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,844 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,859 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,874 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,889 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,904 nettow nettow.py load_samples():218 Loaded Sample 1
INFO 2015-02-12 16:40:38,920 nettow nettow.py load_samples():218 Loaded Sample 1
Confirm that new NetTow Activities are in the database:
http://odss.mbari.org/canon/stoqs_simz_aug2013/activity.html?name__contains=Net
http://odss.mbari.org/canon/stoqs_simz_aug2013/sample.html?sampletype__name__con
tains=Net
And examine the data in the STOQS UI:
http://odss.mbari.org/canon/stoqs_simz_aug2013/
Subsamples can now be loaded from our regular load scripts as they now have
parent Samples to link to.
More work needs to be done in the UI to properly represent integrative NetTow
sampling, yet this Issue is being marked as fixed.
Original comment by MBARIm...@gmail.com
on 12 Feb 2015 at 5:03
Original issue reported on code.google.com by
MBARIm...@gmail.com
on 2 Feb 2015 at 4:55