Add a retry delay to the dagsterDaemon.runRetries Helm chart values

[DustyShap]

What's the use case?

In my case, if an Airbyte sync fails, usually due to a sync on that connection already running, it would be nice to wait a provided amount of time before retrying. It looks like it retries immediately, and in my case, doesn't provide any values to the retries.

Ideas of implementation

Add a retry_delay variable to allow an amount of time to be passed to the dagsterDaemon.runRetries object on the Helm template values.

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[smackesey]

cc @OwenKephart

[yuhan]

@benpankow do you have context on this? sounds like a reasonable behavior to enable?

[dragos-pop]

This can be done using a RetryPolicy with delay in seconds as an argument, while defining your Airbyte sync job. Something similar to: jobs.py (asset job)

from dagster import (
    define_asset_job,
    AssetSelection,
    load_assets_from_modules,
    RetryPolicy
)

from . import assets

all_assets = load_assets_from_modules([assets])
job_airbyte_sync = define_asset_job(
    name="job_airbyte_sync”
    selection=AssetSelection.all(),
    op_retry_policy=RetryPolicy(
        max_retries=5,
        delay=300  # seconds
    )
)

OR: jobs.py (op job)

from dagster import (
    job,
    RetryPolicy
)

from .ops import (
    op_airbyte_sync
)

@job(
    op_retry_policy=RetryPolicy(
        max_retries=5,
        delay=300  # seconds
    )
)
def job_airbyte_sync():
    op_airbyte_sync()