search

marinebon / mbon-dashboard-server

server software for MBON early alert dashboard using Docker
1 stars 2 forks source link

Data from Sofar buoy in FGBNMS #49

Open dotis opened 1 year ago

dotis commented 1 year ago

The new Sofar buoy at FGBNMS is collecting data at two depths. The data can be found here:

https://spotters.sofarocean.com/historical/SPOT-30987C?spotter-filter=SPOT-30987C

I can click on the download link and get a .csv file with the data, but I can't see the URL. I'm not sure if it's hidden, but to add the data to the FGB dash, I assume we will need a link. Once we have the link, I would like to push the .csv to GH and display the temperature data on the grafana dash.

7yl4r commented 1 year ago

curl command that grabs the data:

curl 'https://api.sofarocean.com/fetch/download-sensor-data/?spotterId=SPOT-30987C&startDate=2023-06-29T04:00:00Z&endDate=2023-07-01T04:00:00Z&processingSources=all' \
  -X GET \
  -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.0' \
  -H 'Accept: application/json, text/plain, */*' \
  -H 'Accept-Language: en-US,en;q=0.5' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -H 'view_token: 1bc9848d3e524c34a1eb220e121d9a9e' \
  -H 'Sec-Fetch-Dest: empty' \
  -H 'Sec-Fetch-Mode: cors' \
  -H 'Sec-Fetch-Site: same-site' \
  -H 'Pragma: no-cache' \
  -H 'Cache-Control: no-cache' \
  -H 'referrer: https://spotters.sofarocean.com/' \
  -H 'credentials: omit' \
  -H 'mode: cors'

This can be implemented as an airflow job similar to the ts_ingest DAG. The DAG can be set to run daily and only grab the last day's data. There are two (easy) ways to do this:

  1. allow the DAG to backfill back to the start of the bouy (2023-06-20)
    • if this is done, a dashboard reset will hit the server with a lot of requests all at once, possibly leading to us being blacklisted.
  2. set the DAG to only_latest, so that only the latest week of data is grabbed by airflow
    • this means that a manual step will need to be added to grab+upload older historical data every time the database is reset.

The more computationally-efficient but harder to implement way would be to execute the query based on the dates in the InfluxDB.

dotis commented 1 year ago

From Duncan at Sofar:

The API allows you to request historical data or latest data from the system. Be sure to click the ">" arrows to expand the documentation.

https://docs.sofarocean.com/spotter-and-smart-mooring

https://docs.sofarocean.com/spotter-and-smart-mooring/smart-mooring-sensor-data

Also there is the option of using the POST forwarding feature in order to forward all data generated by the system to an endpoint/server of your choice.

https://docs.sofarocean.com/spotter-and-smart-mooring/post-forwarding

7yl4r commented 11 months ago
7yl4r commented 11 months ago

I think the DAG is now working but is getting no data because the source has no data for {today}T00:00 through {yesterday}T00:00. I am going to let it run a day or two; if it still isn't working I will need to:

  1. guess what the data publication lag time is & use execution_date.strftime.
  2. expect the first run to fail and use DAG retries
7yl4r commented 11 months ago

f4555cbf82fd121a20712033a977f5e654ffccbd

dotis commented 11 months ago

Is there a way to include data from both surface and depth on the same plot? I don't think we need a dropdown to select the buoy, but can put them on the same plot.