Airflow DAGs failing for FKNMS for sat data only

[dotis]

With a recent manual push of csv files to the Gbucket by @dotis, the DAGs for sat time series files for the FKNMS dashboard have failed. All other DAGs (rivers, buoys) seem fine w/csv files from the same manual push. All DAGs for SEUS and FGBNMS seem fine.

[dotis]

Related - Although the DAGs for GRNMS dash worked, data is not updated on the dash. The bucket has the updated file and the DAG worked, but the time series is not updated.

[7yl4r]

I have resolved the issues introduced with my should_convert_time changes to the mbon_data_uploader but the sat dags are still failing due to the -62167305600000000000 first-row value issue in these data files. Here is a summary of where we are seeing the issue:

• [ ] SEUS sat ingest (bouys and rivers are working)
• [ ] FKNMS sat ingest (bouys and rivers are working)
• [ ] FGBNMS sat ingest (rivers working)

I can add something to the mbon_data_uploader that drops bad rows but ideally we would identify the source of the bad first row (shown below) and remove it before uploading the data to the gcloud bucket.

(base) tylar@tylar-gram:~/Downloads$ head datafile.csv 
Time,mean,climatology,anomaly
-62167305600,0.000000,NaN,NaN
1025740800,NaN,0.000656,NaN
1025827200,NaN,0.000647,NaN

[dotis]

I assume that negative posixtime is ok for the ingest. This simply indicates times prior to 1/1/1970. This issue appears fixed in the river and met files and the DAGs are passing. This is still an issue for the sat extractions, but I identified the issue and that should be fixed today. We can check DAGs tomorrow.

[dotis]

All DAGs are passing. csv files from @dotis no longer have a bad first line.

[dotis]

Reopening this. The GR met buoy files are updated in the gbucket and the DAG passed. However, the dash is not updated. Perhaps an issue w/this ingest file.

[dotis]

Updates and DAGs were successful this am.