CamFromStar
commented
2 weeks ago An interesting implementation would be setting up the macros to exist outside of codespaces. This would allow easier construction of maintenance jobs that modify/interact with multiple codespaces. This idea would also further isolate macros from the concept of assets and jobs.
What's the use case?
It would be an excellent feature to allow us to define maintenance macros that can be executed either with the press of a button from the UI, or with a schedule. These are fundamentally different from jobs/assets, as their purpose is not to transform external assets, but alter the Dagster environment itself for maintenance purposes.
The UI should have a macro tab that lists the macros available, and that can be run with the press of a button. The layout of macros within the macro tab should be somewhat customizable.
A good example use case would be implementing a shut-down and start-up sequence. Imagine an environment with a litany of sensors, schedules, auto-materialized assets, etc. In such a scenario their may be a desired order of shut down, along with some checks to be performed, before a code-space update is pulled in. Once the update is complete, a start-up sequence could be executed that resets sensor cursors, executes particular jobs, then re-enables certain sensors and schedules.
Right now the above example could be achieved with sending graphql queries via the API. However, it would be nice for this to be formalized and made easier.
Ideas of implementation
Two types of macros come to mind, python ones and graphql ones. Simpler macros could be a series of GraphQL queries. More complicated ones could be written in python.
Python macros can be created similar to assets, with a function decorator.
GraphQL macros could consist of a series of GraqhQL queries, along with an XML/JSON file that describes how the queries should be executed.
Additional information
No response
Message from the maintainers
Impacted by this issue? Give it a 👍! We factor engagement into prioritization.