search

bcgov / nr-rfc-grib-copy

Demo of using github actions to copy / process raw data
Apache License 2.0
2 stars 2 forks source link

Implement Event Listener - Trigger #8

Open franTarkenton opened 1 year ago

franTarkenton commented 1 year ago

In order to speed up the processing of the cmc grib data, this task will do the following:

Focus of this work is to speed up the downloading / processing of the cmc grib data to enable hourly processing.

Task relates to: https://github.com/bcgov/nr-rfc-admin/issues/21 and https://github.com/bcgov/nr-rfc-grib-copy/issues/7

franTarkenton commented 1 year ago

working on this however is time consuming and difficult to debug due to infrequency that events are emitted for the data we are monitoring for.

Have resolves some timeout issues, but still haven't managed to observe an event from the stream we are attempting to monitor.

Branch with the work for this ticket: feat/event_listener

franTarkenton commented 1 year ago

link to doc on how to setup the other side (github action) for remote trigger: https://dev.to/rikurouvila/how-to-trigger-a-github-action-with-an-htt-request-545

franTarkenton commented 1 year ago

Created the PR: https://github.com/bcgov/nr-rfc-grib-copy/pull/14 which contains code showing how to setup an event listener process.

Currently all the script does is listen for events and log them to the log that is emitted to STDOUT. Breaking this task up into smaller pieces described below

Next steps is to create tickets for the following tasks.

  1. containerize code - create dockerfile and github action that will build the dockerimage and cache either in dockerhub or github container registry.
  2. Create helm chart that will do the actual deployment to the openshift namespace
  3. Figure out best way to call github action remotely. See link in comment above.
  4. If time permitting create PR to metpx (https://github.com/MetPX/sarracenia) repo with documentation fixes