start up big_bend classification

[7yl4r]

Picking up from #8

Moving from staging status (external (2)) into to_load (3) so airflow can start processing.

MariaDB [imars_product_metadata]> UPDATE file SET status_id=3 WHERE status_id=2 AND date_time NOT IN ("2014-04-02 16:51:15","2014-03-01 16:29:30","2014-07-17 16:41:46","2014-12-13 16:39:54","2014-12-13 16:40:07");
Query OK, 384 rows affected (0.05 sec)
Rows matched: 384  Changed: 384  Warnings: 0

Note that is most of big_bend 2014, but the few dates are excluded because of duplicate dates & missing files.

[7yl4r]

Hmm... something didn't work right here:

MariaDB [imars_product_metadata]> SELECT short_name,id FROM product ORDER BY id;
+------------------------+----+
| short_name             | id |
+------------------------+----+
| ntf_wv2_m1bs           | 11 |

MariaDB [imars_product_metadata]> SELECT status_id,COUNT(*) FROM file WHERE product_id=11 GROUP BY status_id ;
+-----------+----------+
| status_id | COUNT(*) |
+-----------+----------+
|         1 |      176 |
|         2 |        9 |
|         4 |      193 |
|         5 |      116 |
+-----------+----------+
4 rows in set (0.00 sec)

MariaDB [imars_product_metadata]> SELECT * FROM status;
+----+------------+-----------------------------------------------------------------------------------------------+
| id | short_name | description                                                                                   |
+----+------------+-----------------------------------------------------------------------------------------------+
|  1 | std        | default status normal. the file exists, is up-to-date, and is accessible in the imars system. |
|  2 | external   | file not ingested into the imars system, but is available from an external source.            |
|  3 | to_load    | file loaded into the data warehouse, but not yet processed. to_load is a holdover misnomer.   |
|  4 | error      | product is associated with an error in IMaRS automated processing system                      |
|  5 | deleted    | file was deleted and is no longer served by IMaRS                                             |
+----+------------+-----------------------------------------------------------------------------------------------+

Looks like a lot split into std & error, but very few dags (none?) were triggered... Oh! Maybe because there is no wv2_classfication_big_bend DAG to trigger.

[7yl4r]

Fixed missing dag and trying again:

MariaDB [imars_product_metadata]> UPDATE file SET status_id=3 WHERE product_id=11 AND filepath LIKE "/srv/imars-objects/big_bend/wv2%";
Query OK, 201 rows affected (0.05 sec)
Rows matched: 201  Changed: 201  Warnings: 0

To confirm:

MariaDB [imars_product_metadata]> SELECT status_id,COUNT(*) FROM file WHERE product_id=11 GROUP BY status_id ;
+-----------+----------+
| status_id | COUNT(*) |
+-----------+----------+
|         1 |      176 |
|         3 |      201 |
|         4 |        1 |
|         5 |      116 |
+-----------+----------+