search

marinebon / mbon-dashboard-server

server software for MBON early alert dashboard using Docker
1 stars 2 forks source link

Ingest of .csv from GH #48

Open dotis opened 1 year ago

dotis commented 1 year ago

The .csv ingest from GH seems to be working for the most part. There was an issue with the VIIRS sstn, which I fixed. Most other airflow jobs seem to be good.

However, the buoy and river data jobs are failing. The .csv files are now on GH with the same filenames as before. I can't tell if the paths in ts_ingest.py need to be updated (see below). 

BOUY_FPATH = "{REGION}-_-SAL_TS_NDBC-_-{roi}_NDBC_{product}_FKdb.csv"

RIVER_FPATH = "{REGION}-_-DISCH_CSV_USGS-_-USGS_disch_{river}.csv"
dotis commented 1 year ago

The filenames of the buoy and river data are somewhat standardized, but could be better. @7yl4r let me know if you think I should further standardize them. For example, filenames now contain an ID for the particular dashboard ("FWCdb", "FKdb", or "FGBdb"). I'm not sure that the particular DB needs to be in the filename.

We could also make the format river files match the format of the buoy files.

For example, "WHIPRAY_NDBC_sal_FKdb.csv" could be changed to "NDBC_sal_WHIPRAY_FKdb.csv", which would more closely match the river discharge files.

dotis commented 1 year ago

VIIRS SST is now working. MODA Kd_490 has an issue, but that should be fixed. The only remaining issue seems to be the buoys and rivers. Seems to be a 404 error from the log.

dotis commented 1 year ago

Upon further examination, there are no VIIRS ingest jobs at all in the Airflow "ts_ingest" dag. There are also no locations from FGB. I'm not sure why those disappeared. So, I only see MODA ingest jobs for FKNMS locations. Somehow the all the VSNPP and FGB locations have gone away. Is airflow running on all branches? It seems the only branch that I see on the DAG list is the client-fknms.

dotis commented 1 year ago

The rivers and buoys are now up. The only thing not working is the MODA Kd_490. Not sure what the issue is. The last attempted Airflow runs for both FK and FGB were on 6/24. The FGB dashboard time series are current through 6/19. FKNMS dashboard is current through 6/20. Getting there....

dotis commented 1 year ago

Looking more closely, I see the MODA Kd_490 files have not been pushed to GH. The last push of .csv files to GH was four days ago, so the rsync has stopped.