run-level resource cleanup / lifecycle hooks

[sryza]

It would be useful to be able to run some cleanup code at the end of a run. Examples:

• Tear down a cluster
• Delete a temporary directory

I think it would likely make sense to tie these to resources.

Relevant requests:

• Marco - https://dagster.slack.com/archives/C01U954MEER/p1677864919402759
  • Team, what is the best way to avoid persisting to storage (disk, DB, etc) materialisations that are only required until a given job has finished running? I was thinking to use the mem_io_manager but it does not work with multiprocess execution
• Danny Steffy - https://dagster.slack.com/archives/C01U954MEER/p1677879840168849
  • Could use some help with a temporary pickled file manager. We have an op that reads in data from a MSSQL server, that we need to pass to a downstream op, and then we want to scrap that data. We have the job running with the multiprocess_executor, so we can't just use the mem_io_manager , so I figure the best way to solve this is to write a temp io manager. How would I go about deleting the file after the downstream op has finished processing it?
• https://www.notion.so/223b21238a4e473faacf7c64a4a4f1e1
  • Spinning up a Ray cluster takes 12 minutes - would like to share this across many assets. We usually use one cluster for each job. Shut it down at the end.
• I have a question about graph backed assets. I have a use case where the workflow is roughly (all ops). Unfortunately, dagster out of the box doesn't seem to support having that third step be part of the asset graph, since it doesn't produce any output. Is there a way to run that op as part of the asset graph?:
  • spin up a ray cluster
  • run a script that materializes a table
  • spin down the ray cluster

[MSigno]

Hi - any progress on this issue?

[eloreaux]

Any progress on this? Would like to delete a tmp folder after run is complete

[judahrand]

Is there anyway to do this currently? Any workarounds?

[sryza]

@judahrand the closest that we have is op-level hooks, but they don't provide the full functionality discussed here. https://docs.dagster.io/concepts/ops-jobs-graphs/op-hooks

Here's a way that one integration has hacked around this: https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-mlflow/dagster_mlflow/hooks.py#L33

[lintonye]

Isn't this what's requested? https://docs.dagster.io/concepts/resources#lifecycle-hooks

When a resource is initialized during a Dagster run, the setup_for_execution method is called. This method is passed an InitResourceContext object, which contains the resource's config and other run information. The resource can use this context to initialize any state it needs for the duration of the run.

Once a resource is no longer needed, the teardown_after_execution method is called. This method is passed the same context object as setup_for_execution. This method can be useful for cleaning up any state that was initialized in setup_for_execution.

setup_for_execution and teardown_after_execution are each called once per run, per process.