search

catalyst-cooperative / pudl-usage-metrics

A dagster ETL for collecting and cleaning PUDL usage metrics.
MIT License
1 stars 0 forks source link

Update dagster requirement from <1.8.14,>=1.7.15 to >=1.7.15,<1.9.1 #213

Closed dependabot[bot] closed 3 weeks ago

dependabot[bot] commented 3 weeks ago

Updates the requirements on dagster to permit the latest version.

Changelog

Sourced from dagster's changelog.

1.9.0 (core) / 0.25.0 (libraries)

Major changes since 1.8.0 (core) / 0.24.0 (libraries)

Automation

  • Declarative Automation, the system which enables setting per-asset AutomationConditions, is no longer experimental. We now recommend using this system in all cases where asset-centric orchestration is desired. A suite of built-in static constructors have been added for common usecases, such as AutomationCondition.on_missing() (which can fill in missing partitions of assets as soon as upstream data is available), and AutomationCondition.all_deps_blocking_checks_passed() (which can prevent materialization of assets until all upstream blocking checks have passed).
  • You can now assign AutomationConditions to asset checks, via the automation_condition parameter on @asset_check or AssetCheckSpec.
  • You can now assign AutomationConditions to observable source assets, via the automation_condition parameter on @observable_source_asset.
  • [experimental] You can now define custom subclasses of AutomationCondition to execute arbitrary Python code in the context of a broader expression. This allows you to compose built-in conditions with custom business logic.
  • The target arguments on schedules and sensors are now marked stable, allowing a stable way for schedules and sensors to target asset selections without needing to define a job.

Integrations

  • Introduced a slate of integrations with business intelligence (BI) tools, enabling dashboards, views, and reports to be represented in the Dagster asset graph.
  • A rich set of metadata is now automatically collected by our suite of ELT integrations.
    • The dagster/table_name metadata tag, containing the fully-qualified name of the destination model, has been added for Airbyte, dlt, Fivetran and Sling assets.
    • The dagster/row_count metadata tag, containing the number of records loaded in the corresponding run, has been added for dlt and Sling assets.
    • The dagster/column_schema metadata tag, containing column schema information of the destination tables, has been added for Fivetran assets.
    • Column lineage information is now collected for Sling assets.
  • dagster-pipes are replacing the now deprecated Step Launchers as the new recommended approach for executing remote Spark jobs. Three new Pipes clients for running Spark applications on Amazon Web Services have been added:
    • dagster_aws.pipes.PipesGlueClient
    • dagster_aws.pipes.PipesEMRServerlessClient
    • dagster_aws.pipes.PipesEMRClient

UI

  • Several changes have been made to the information architecture to make it easier to find what you’re looking for:
    • Backfills have been moved from their own tab underneath the Overview page to entries within the table on the Runs page. This reflects the fact that backfills and runs are similar entities that share most properties. You can continue to use the legacy Runs page with the “Revert to legacy Runs page” user setting. (GitHub Discussion)
    • “Jobs” is now a page reachable from the top-level navigation pane. It replaces the Jobs tab within the Overview page.
    • “Automations” is now a page reachable from the top-level navigation pane. It replaces the schedule and sensor tabs within the Overview page.
  • @asset and AssetSpec now have a kinds attribute that enables specifying labels that show up on asset nodes in the asset graph in the UI. This supersedes the compute_kind attribute.

Changes since 1.8.13 (core) / 0.24.13 (libraries)

New

  • The tags parameter to @asset and AssetSpec is no longer marked as experimental.
  • The @observable_source_asset decorator now supports an automation_condition argument.
  • AutomationCondition and associated APIs are no longer marked as experimental.
  • Added a new use_user_code_server parameter to AutomationConditionSensorDefinition. If set, the sensor will be evaluated in the user code server (as traditional sensors are), allowing custom AutomationCondition subclasses to be evaluated.
  • Added a new column to the BulkActions table, a new column to the Runs table, and a new BackfillTags table to improve the performance of the Runs page. To take advantage of these performance improvements, run dagster instance migrate. This migration involves a schema migration to add the new columns and table, and a data migration to populate the new columns for historical backfills and runs.
  • Performance improvements when loading definitions with multi-assets with many asset keys.
  • [ui] The previously-experimental changes to the top nav are now enabled for all users.
  • [ui] Added new code location pages which provide information regarding library versions, metadata, and definitions.
  • [ui] The new version of the Runs page is now enabled by default. To use the legacy version of the Runs page, toggle the "Revert to legacy Runs page" user setting.

... (truncated)

Commits


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)