MetisLinker - adjustable time interval

[dtm2451]

Defines a MetisLinkerDefine wrapper function, Edit: which assumes the original MetisLinker function name, to allow adjustment of the dag settings when defining the MetisLinker DAG.

[dtm2451]

Outdated!

This will also require updating the code in the airflow code editor from current:

dag = MetisLinker()

To:

dag = MetisLinkerDefine(interval_minutes=5)

Current: No change to the production ETL code will be needed. The time interval update will 'just happen' =)

[dtm2451]

Just FYI, the code comparison here isn't representing the changes well... so a quick description:

• top of section:
  • adds a wrapper function around the @dag portion and original MetisLinker function definition
  • so adds indentation to the whole @dag portion + original MetisLinker definition. (This is what probably confuses the algorithm.)
• bottom of section:
  • only real addition outside of indentations is the return MetisLinker() line.

[dtm2451]

Full instructions for testing were laid out pretty well by @graft a while back: https://github.com/mountetna/monoetna/pull/1175#issuecomment-1563250651

Here, you could probably skip a lot of the steps as the main thing we want to see is the time interval being 5 minutes instead of 1 hour, with toggle-ability via giving any number of minutes in the code editor definition.

E.g. using this should set the 'MetisLinker' DAG to run every 25 minutes instead of the new default of every 5 minutes, while nothing else about the DAG should change:

from etna.etls.metis_linker import MetisLinker

dag = MetisLinker(25)

[dtm2451]

Bleh, noticed that with the internal function being called 'MetisLinkerDefine', that got inherited as an update to the dag id/name. And then it becomes disconnected from old runs/logs.

Tried a few other ways, but seems the best way to keep the dag appearing the exact same as it currently does (and still linked with past runs/logs) is to scratch the Define part on the internal function name.

Having both the wrapper and internal function have the same name makes the code a bit more confusing to read, but it works... :shrug:

[dtm2451]

More specific reviewer testing goals:

1. as soon as you checkout this branch and re-build airflow, your MetisLinker DAG run interval should update to 5 minutes without you needing to do anything.
2. a full run of the dag, with some polyphemus config to target, should be successful.