Reprocess stereoTop data for 2016-2017

[craig-willis]

Re-run full stereoTop pipeline extractors on all data though September 20th, 2017. This includes: bin2tif, fieldmosaic, and canopycover extractors.

Re-processing steps:

• [x] Query to remove raw metadata from MongoDB and generate list of datasetIds and names
• [x] Use list to reload LemnaTec metadata using new cleaning process
• [x] Delete Level 1 RGB data on disk and in Clowder (rgb_geotiff, stereoTop_geotiff, demosaic, fullfield for RGB
• [x] Purge RabbitMQ message queues
• [x] Purge geostreams
• [x] Purge canopycover traits from BETYdb
• [x] Queue raw datasets for bin2tif
• [x] Run rule_checker (3), bin2tif (~25), fieldmosaic (1) and canopycover (1) extractors

[max-zilla]

@robkooper FYI To simplify this pipeline I am introducing a small new extractor: https://github.com/terraref/extractors-metadata/tree/master/cleaner terraref/ext-metadata-cleaner

This extractor binds to nothing - that is, it does not trigger automatically on any kind of file or dataset. It can only be triggered manually.

When triggered on a dataset, it will (optionally) delete all metadata currently attached to the dataset, then use terrautils + dataset name to determine where on ROGER the raw original metadata.json file is located. Once found it runs that file through our latest terrautils cleaning code and optionally triggers a callback extractor if given.

With this setup, the process becomes:

• [ ] purge Level_1 rgb_geotiff, stereoTop_geotiff, demosaic directory on ROGER
• [ ] purge or set aside current Level_1 fullfield directory on ROGER
• [ ] purge RabbitMQ message queues
• [ ] purge geostreams of all sensorposition streams
• [x] deploy cleaner extractor(s) v1.0
• [x] deploy sensorposition extractor(s) v1.0
• [x] deploy bin2tif extractor(s) v1.0
• [x] deploy rulechecker extractor(s) v1.0
• [x] deploy fieldmosaic extractor(s) v1.0
• [ ] submit root stereoTop collection for cleaner extractor configured to trigger bin2tif when done
• [ ] delete old Level_1 RGB GeoTIFFs from Clowder (will be in separate space from new stuff)

I am running final tests of our new code we added this week but will start this process today. This will require a lot of care as there is more at stake than the EnvironmentLogger processing.

[max-zilla]

Basically here's the sketch I came up with last night:

[max-zilla]

Continued in #393