upload leaf width / height for seasons 4 and 6

[dlebauer]

From @rleontie

We would like to update the results on Betty DB for the leaf length and width extractors for season 4 and 6 data. You can find season 4 and season 6 linked here.

• [x] remove existing records
• [x] round to 3 digits (?)

[rleontie]

Confirming please delete existing records. The measurements are in millimeters so not sure that having more than 3 digits make sense.

[dlebauer]

Flagging old records

update traits set method_id = 6000000040 where method_id = 6000000029;
delete from methods where id = 6000000029;

update traits set checked = -1, 
notes = 'old version of leaf length / width extractor replaced 2020-02-14', access_level = 1
WHERE
    site_id IN ( SELECT ID FROM sites WHERE sitename LIKE '%Season 4%' ) 
    AND variable_id IN ( SELECT ID FROM variables WHERE name IN ( 'leaf_width', 'leaf_length' ) ) 
    AND method_id IN ( SELECT ID FROM methods WHERE name = '3D scanner to leaf length and width' );

update traits set checked = -1, 
notes = 'old version of leaf length / width extractor replaced 2020-02-14', access_level = 1
WHERE
    site_id IN ( SELECT ID FROM sites WHERE sitename LIKE '%Season 6%' ) 
    AND variable_id IN ( SELECT ID FROM variables WHERE name IN ( 'leaf_width', 'leaf_length' ) ) 
    AND method_id IN ( SELECT ID FROM methods where name = '3D scanner to leaf length and width' )

Uploading new data (in progress)

fix errors

cat betydb_s4_leaf_length_width.csv | \
  sed 's/Scanner\ 3d\ ply\ data\ to\ leaf\ length/3D\ scanner\ to\ leaf\ length\ and\ width/g' | \
  sed 's/\ width/\ width\,2/g' | sed 's/method/method\,access_level/g' > s4.csv

cat betydb_s6_leaf_length_width.csv | \
  sed 's/Scanner\ 3d\ ply\ data\ to\ leaf\ length/3D\ scanner\ to\ leaf\ length\ and\ width/g' | \
  sed 's/\ width/\ width\,2/g' | sed 's/method/method\,access_level/g' > s6.csv

Upload

curl -H "Content-Type: text/csv" -X POST --data-binary @s4.csv \
    https://terraref.ncsa.illinois.edu/bety/api/v1/traits.csv?key=SECRET
curl -H "Content-Type: text/csv" -X POST --data-binary @s6.csv \ 
    https://terraref.ncsa.illinois.edu/bety/api/v1/traits.csv?key=SECRET

[dlebauer]

try dividing into chunks ...

sed -n -e 1p -e 2,5000p s4.csv > s4.1.csv
sed -n -e 1p -e 5001,10000p s4.csv > s4.2.csv
sed -n -e 1p -e 10001,12759p s4.csv > s4.3.csv

for i in {1..3}; do echo curl -H "Content-Type: text/csv" -X POST --data-binary @s4.$i.csv https://terraref.ncsa.illinois.edu/bety/api/v1/traits.csv?key=SECRET; done;

# actually not necessary to divide these up; a timeout is given but the data is still entered
curl -H "Content-Type: text/csv" -X POST --data-binary @s6.csv https://terraref.ncsa.illinois.edu/bety/api/v1/traits.csv?key=