S9 & S10

[meredithspalmer]

@aliburchard @mkosmala

I think MSI is down for maintenance at the moment, but as soon as I can get back online, I am going to upload the cleaned S9 and S10 captures.

In addition, I will be uploading new scripts onto both MSI and GitHub:

• I ended up rewriting all of Ali's directory restructuring scripts into a single shell script. It's a little less complicated and it doesn't drop, delete, or misplace any files (I ran file counting scripts before and after and everything looks good).
• I also had to update the R cleaning scripts to deal with two-digit season numbers (10 vs. 9 messes up some of the substring commands) and the fact that our current data enters into 2015. Previously, any data over 2015 was flagged as "future" data, so I've made the scripts so that you can select a year over which data needs to be marked as suspicious.

I am also uploading to both files detailing any manual changes I made to the directory structure, moving files, etc. These include combining A & B rolls, moving movies and corrupt images to other directories (file recounting scripts were rerun after all files that were not to be included we moved out), and dealing with any errors caught by the extract_date.py and set_up_captures.py scripts.

General notes on cleaning these seasons:

• Excel was not touched! Hopefully datetimes remain intact.
• There were lots of timechanges noted in the field data that I did not include in the action list. For these rolls, the cameras misfired constantly and filled up in less than a week; I suspect the timechanges occurred later, so these rolls are not flagged as all data in them in consistent (no lags, etc.) and times make sense with time of day, etc. Comments on other timechanges are in the action lists.
• I went through the data specifically looking for stuck capture events that were not caught by the "badcaps" script. There are a few, which are dealt with in the action list. I either separated the captures if there was value information in them into separate CEs and renamed all images/subsequent CEs OR I invalidated all by the first three images (first real capture event).

Overall, there didn't seem to be that many issues to deal with. @mkosmala -- I reran all the python scripts on MSI on S8 earlier, but didn't download the S8_captures.csv file and without this right now, I can't clean up S8 again. As soon as I can get back on MSI, I can quickly fix that up so that you can keep working on S8.

[meredithspalmer]

Okay, @mkosmala @aliburchard -- the cleaned S9 and S10 files (+ scripts, action lists, etc.) are online at MSI now. I'll finish redoing S8 today/tomorrow.

[meredithspalmer]

Also, comments on missing rolls are also uploaded. There are two rolls missing (one from each season) that I have emailed Norbert about.