search

catalyst-cooperative / pudl-usage-metrics

A dagster ETL for collecting and cleaning PUDL usage metrics.
MIT License
1 stars 0 forks source link

Update dagit requirement from ~=0.15.0 to >=0.15,<1.3 #105

Closed dependabot[bot] closed 1 year ago

dependabot[bot] commented 1 year ago

Updates the requirements on dagit to permit the latest version.

Changelog

Sourced from dagit's changelog.

1.2.0 (core) / 0.18.0 (libraries)

Major Changes since 1.1.0 (core) / 0.17.0 (libraries)

Core

  • Added a new dagster dev command that can be used to run both Dagit and the Dagster daemon in the same process during local development. [docs]
  • Config and Resources
  • Repository > Definitions [docs]
  • Declarative scheduling
    • The asset reconciliation sensor is now 100x more performant in many situations, meaning that it can handle more assets and more partitions.
    • You can now set freshness policies on time-partitioned assets.
    • You can now hover over a stale asset to learn why that asset is considered stale.
  • Partitions
    • DynamicPartitionsDefinition allows partitioning assets dynamically - you can add and remove partitions without reloading your definitions (experimental). [docs]
    • The asset graph in the UI now displays the number of materialized, missing, and failed partitions for each partitioned asset.
    • Asset partitions can now depend on earlier time partitions of the same asset. Backfills and the asset reconciliation sensor respect these dependencies when requesting runs [example].
    • TimeWindowPartitionMapping now accepts start_offset and end_offset arguments that allow specifying that time partitions depend on earlier or later time partitions of upstream assets [docs].
  • Backfills
    • Dagster now allows backfills that target assets with different partitions, such as a daily asset which rolls up into a weekly asset, as long as the root assets in the selection are partitioned in the same way.
    • You can now choose to pass a range of asset partitions to a single run rather than launching a backfill with a run per partition [instructions].

Integrations

  • Weights and Biases - A new integration dagster-wandb with Weights & Biases allows you to orchestrate your MLOps pipelines and maintain ML assets with Dagster. [docs]
  • Snowflake + PySpark - A new integration dagster-snowflake-pyspark allows you to store and load PySpark DataFrames as Snowflake tables using the snowflake_pyspark_io_manager. [docs]
  • Google BigQuery - A new BigQuery I/O manager and new integrations dagster-gcp-pandas and dagster-gcp-pyspark allow you to store and load Pandas and PySpark DataFrames as BigQuery tables using the bigquery_pandas_io_manager and bigquery_pyspark_io_manager. [docs]
  • Airflow The dagster-airflow integration library was bumped to 1.x.x, with that major bump the library has been refocused on enabling migration from Airflow to Dagster. Refer to the docs for an in-depth migration guide.
  • Databricks - Changes:
    • Added op factories to create ops for running existing Databricks jobs (create_databricks_run_now_op), as well as submitting one-off Databricks jobs (create_databricks_submit_run_op).
    • Added a new Databricks guide.
    • The previous create_databricks_job_op op factory is now deprecated.

Docs

  • Automating pipelines guide - Check out the best practices for automating your Dagster data pipelines with this new guide. Learn when to use different Dagster tools, such as schedules and sensors, using this guide and its included cheatsheet.
  • Structuring your Dagster project guide - Need some help structuring your Dagster project? Learn about our recommendations for getting started and scaling sustainably.
  • Tutorial revamp - Goodbye cereals and hello HackerNews! We’ve overhauled our intro to assets tutorial to not only focus on a more realistic example, but to touch on more Dagster concepts as you build your first end-to-end pipeline in Dagster. Check it out here.

Stay tuned, as this is only the first part of the overhaul. We’ll be adding more chapters - including automating materializations, using resources, using I/O managers, and more - in the next few weeks.

Since 1.1.21 (core) / 0.17.21 (libraries)

New

  • Freshness policies can now be assigned to assets constructed with @graph_asset and @graph_multi_asset.
  • The project_fully_featured example now uses the built in DuckDB and Snowflake I/O managers.
  • A new “failed” state on asset partitions makes it more clear which partitions did not materialize successfully. The number of failed partitions is shown on the asset graph and a new red state appears on asset health bars and status dots.
  • Hovering over “Stale” asset tags in the Dagster UI now explains why the annotated assets are stale. Reasons can include more recent upstream data, changes to code versions, and more.

... (truncated)

Commits
  • 56eb23c 1.2.0
  • 9dbcd68 Changelog 1.2.0 (#12851)
  • b321e2d [dagster-airflow] remove double encoding of timezones (#12811)
  • a93a665 Add a way to cause partition health data to refetch without passing around re...
  • 5fa2b86 [dagit] Fix yaml editor help context when indented on a new line (#12537) (#1...
  • cbeec1e [dagit] Update the asset partitions / events view after run failures (#12798)
  • 8b5f19f Maintain a separate cursor in the asset partition cache pointing back to the ...
  • b857254 Recommended Project Structure Guide (#12656)
  • 2e058d6 Remove PartitionMetadataEntry (#12726)
  • 681cd4f [docs] - Update RBAC docs for Cloud (#12752)
  • Additional commits viewable in compare view


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)