in the log, just before the SAS aspects start appearing
I want to access these, in the SAS code that gets executed, with sysget, however, they don't seem to exist, presumable as they have passed down into the SAS execution context.
I know there's support for passing in arbitrary env vars, but I would expect these "Airflow Context" ones to be "standard" or by default.
This would allow the instance of the code to properly contextualize itself, knowing:
that's executing "in batch" under Airflow (as opposed to interactively, in Studio)
use the discrete value of AIRFLOW_CTX_DAG_RUN_ID to do things like update a log table of execution timings, or even establish a shared discrete "context" between all Flows in the DAG, allowing rich execution separation when there are multiple instances, and allow better integration with the Airflow API for things like DAG triggering and housekeeping and monitoring
It looks like the standard (?) Python operator supports this with a parameter:
_providecontext=True.
Adding this to the SAS Operator would really open up some rich possibilities.
Apologies if I've missed how this already works/how to do it.
When I look at the log of a Flow that has executed in a DAG, I can see the following environment variables:
AIRFLOW_CTX_DAG_OWNER=*** AIRFLOW_CTX_DAG_ID=ABCD AIRFLOW_CTX_TASK_ID=ABCD_1.1.flw AIRFLOW_CTX_EXECUTION_DATE=2023-05-18T00:49:12.460154+00:00 AIRFLOW_CTX_TRY_NUMBER=1 AIRFLOW_CTX_DAG_RUN_ID=manual__2023-05-18T00:49:12.460154+00:00
in the log, just before the SAS aspects start appearing
I want to access these, in the SAS code that gets executed, with sysget, however, they don't seem to exist, presumable as they have passed down into the SAS execution context.
I know there's support for passing in arbitrary env vars, but I would expect these "Airflow Context" ones to be "standard" or by default.
This would allow the instance of the code to properly contextualize itself, knowing:
It looks like the standard (?) Python operator supports this with a parameter: _providecontext=True.
Adding this to the SAS Operator would really open up some rich possibilities.