sematic-ai / sematic

An open-source ML pipeline development platform
Other
972 stars 59 forks source link

Allow registration of pipeline-level cleanup funcs #1125

Open augray opened 3 months ago

augray commented 3 months ago

To allow cleanup of resources even if the pipeline crashes. A possible API:


@sematic.func
def do_foo(x: float, bar: str) -> Output:
    some_resource = create_resource(x, bar)  # create_resource is NOT a func
    sematic.on_cleanup(clean_some_resource(resource.id))  # arguments to cleanup funcs must be resolved at call time
    # ...
    return output

@sematic.func
def clean_some_resource(resource_id: str) -> None:  # cleanup funcs must return None
    # ...
augray commented 3 months ago

A more general and holistic solution would be to allow future-based try/excepts, but that would be a bigger project.