search

marinebon / mbon-dashboard-server

server software for MBON early alert dashboard using Docker
1 stars 2 forks source link

ts_ingest push to multiple endpoints? #28

Open 7yl4r opened 3 years ago

7yl4r commented 3 years ago

I'm setting up an instance of this on IMaRS servers to hopefully solve #26. Two ways to do this:

  1. airflow ts_ingest pushes data from IMaRS server to influxDB on FWC google cloud
  2. airflow ts_ingest pushes data from IMaRS server to influxDB at IMaRS. Then grafana running on FWC google cloud connects to IMaRS influxDB.

Either is fine. 1 is easier so I will do that. It is probably more performant too. But this did give me an idea:

perhaps ts_ingest should push data to a list of endpoints? That should be an easy feature addition and could be useful. For example: in the future we may want to push this same data from IMaRS into the FWC and FKNMS dashboards. This will require either:

  1. two airflow tasks. ie ts_ingest_fwc and ts_ingest_fknms with some duplication in filepaths
  2. ts_ingest tasks organized based on our data structures that push to multiple influxdb endpoints