search

catalyst-cooperative / pudl-usage-metrics

A dagster ETL for collecting and cleaning PUDL usage metrics.
MIT License
1 stars 0 forks source link

Bump dagster from 0.15.0 to 0.15.5 #51

Closed dependabot[bot] closed 1 year ago

dependabot[bot] commented 1 year ago

Bumps dagster from 0.15.0 to 0.15.5.

Release notes

Sourced from dagster's releases.

0.15.5 (2022-07-07)

New

  • Added documentation and helm chart configuration for threaded sensor evaluations.
  • Added documentation and helm chart configuration for tick retention policies.
  • Added descriptions for default config schema. Fields like execution, loggers, ops, and resources are now documented.
  • UnresolvedAssetJob objects can now be passed to run status sensors.
  • [dagit] A new global asset lineage view, linked from the Asset Catalog and Asset Group pages, allows you to view a graph of assets in all loaded asset groups and filter by query selector and repo.
  • [dagit] A new option on Asset Lineage pages allows you to choose how many layers of the upstream / downstream graph to display.
  • [dagit] Dagit's DAG view now collapses large sets of edges between the same ops for improved readability and rendering performance.

Bugfixes

  • Fixed a bug with materialize that would cause required resources to not be applied correctly.
  • Fixed issue that caused repositories to fail to load when build_schedule_from_partitioned_job and define_asset_job were used together.
  • Fixed a bug that caused auto run retries to always use the FROM_FAILURE strategy
  • Previously, it was possible to construct Software-Defined Assets from graphs whose leaf ops were not mapped to assets. This is invalid, as these ops are not required for the production of any assets, and would cause confusing behavior or errors on execution. This will now result in an error at definition time, as intended.
  • Fixed issue where the run monitoring daemon could mark completed runs as failed if they transitioned quickly between STARTING and SUCCESS status.
  • Fixed stability issues with the sensor daemon introduced in 0.15.3 that caused the daemon to fail heartbeat checks if the sensor evaluation took too long.
  • Fixed issues with the thread pool implementation of the sensor daemon where race conditions caused the sensor to fire more frequently than the minimum interval.
  • Fixed an issue with storage implementations using MySQL server version 5.6 which caused SQL syntax exceptions to surface when rendering the Instance overview pages in Dagit.
  • Fixed a bug with the default_executor_def argument on repository where asset jobs that defined executor config would result in errors.
  • Fixed a bug where an erroneous exception would be raised if an empty list was returned for a list output of an op.
  • [dagit] Clicking the "Materialize" button for assets with configurable resources will now present the asset launchpad.
  • [dagit] If you have an asset group and no jobs, Dagit will display it by default rather than directing you to the asset catalog.
  • [dagit] DAG renderings of software-defined assets now display only the last component of the asset's key for improved readability.
  • [dagit] Fixes a regression where clicking on a source asset would trigger a GraphQL error.
  • [dagit] Fixed issue where the “Unloadable” section on the sensors / schedules pages in Dagit were populated erroneously with loadable sensors and schedules
  • [dagster-dbt] Fixed an issue where an exception would be raised when using the dbt build command with Software-Defined Assets if a test was defined on a source.

Deprecations

  • Removed the deprecated dagster-daemon health-check CLI command

Community Contributions

  • TimeWindow is now exported from the dagster package (Thanks @​nvinhphuc!)
  • Added a fix to allow customization of slack messages (Thanks @​solarisa21!)
  • [dagster-databricks] The databricks_pyspark_step_launcher now allows you to configure the following (Thanks @​Phazure!):
    • the aws_attributes of the cluster that will be spun up for the step.
    • arbitrary environment variables to be copied over to databricks from the host machine, rather than requiring these variables to be stored as secrets.
    • job and cluster permissions, allowing users to view the completed runs through the databricks console, even if they’re kicked off by a service account.

Experimental

  • [dagster-k8s] Added k8s_job_op to launch a Kubernetes Job with an arbitrary image and CLI command. This is in contrast with the k8s_job_executor, which runs each Dagster op in a Dagster job in its own k8s job. This op may be useful when you need to orchestrate a command that isn't a Dagster op (or isn't written in Python). Usage:

    from dagster_k8s import k8s_job_op

    my_k8s_op = k8s_job_op.configured({

... (truncated)

Changelog

Sourced from dagster's changelog.

0.15.5

New

  • Added documentation and helm chart configuration for threaded sensor evaluations.
  • Added documentation and helm chart configuration for tick retention policies.
  • Added descriptions for default config schema. Fields like execution, loggers, ops, and resources are now documented.
  • UnresolvedAssetJob objects can now be passed to run status sensors.
  • [dagit] A new global asset lineage view, linked from the Asset Catalog and Asset Group pages, allows you to view a graph of assets in all loaded asset groups and filter by query selector and repo.
  • [dagit] A new option on Asset Lineage pages allows you to choose how many layers of the upstream / downstream graph to display.
  • [dagit] Dagit's DAG view now collapses large sets of edges between the same ops for improved readability and rendering performance.

Bugfixes

  • Fixed a bug with materialize that would cause required resources to not be applied correctly.
  • Fixed issue that caused repositories to fail to load when build_schedule_from_partitioned_job and define_asset_job were used together.
  • Fixed a bug that caused auto run retries to always use the FROM_FAILURE strategy
  • Previously, it was possible to construct Software-Defined Assets from graphs whose leaf ops were not mapped to assets. This is invalid, as these ops are not required for the production of any assets, and would cause confusing behavior or errors on execution. This will now result in an error at definition time, as intended.
  • Fixed issue where the run monitoring daemon could mark completed runs as failed if they transitioned quickly between STARTING and SUCCESS status.
  • Fixed stability issues with the sensor daemon introduced in 0.15.3 that caused the daemon to fail heartbeat checks if the sensor evaluation took too long.
  • Fixed issues with the thread pool implementation of the sensor daemon where race conditions caused the sensor to fire more frequently than the minimum interval.
  • Fixed an issue with storage implementations using MySQL server version 5.6 which caused SQL syntax exceptions to surface when rendering the Instance overview pages in Dagit.
  • Fixed a bug with the default_executor_def argument on repository where asset jobs that defined executor config would result in errors.
  • Fixed a bug where an erroneous exception would be raised if an empty list was returned for a list output of an op.
  • [dagit] Clicking the "Materialize" button for assets with configurable resources will now present the asset launchpad.
  • [dagit] If you have an asset group and no jobs, Dagit will display it by default rather than directing you to the asset catalog.
  • [dagit] DAG renderings of software-defined assets now display only the last component of the asset's key for improved readability.
  • [dagit] Fixes a regression where clicking on a source asset would trigger a GraphQL error.
  • [dagit] Fixed issue where the “Unloadable” section on the sensors / schedules pages in Dagit were populated erroneously with loadable sensors and schedules
  • [dagster-dbt] Fixed an issue where an exception would be raised when using the dbt build command with Software-Defined Assets if a test was defined on a source.

Deprecations

  • Removed the deprecated dagster-daemon health-check CLI command

Community Contributions

  • TimeWindow is now exported from the dagster package (Thanks @​nvinhphuc!)
  • Added a fix to allow customization of slack messages (Thanks @​solarisa21!)
  • [dagster-databricks] The databricks_pyspark_step_launcher now allows you to configure the following (Thanks @​Phazure!):
    • the aws_attributes of the cluster that will be spun up for the step.
    • arbitrary environment variables to be copied over to databricks from the host machine, rather than requiring these variables to be stored as secrets.
    • job and cluster permissions, allowing users to view the completed runs through the databricks console, even if they’re kicked off by a service account.

Experimental

  • [dagster-k8s] Added k8s_job_op to launch a Kubernetes Job with an arbitrary image and CLI command. This is in contrast with the k8s_job_executor, which runs each Dagster op in a Dagster job in its own k8s job. This op may be useful when you need to orchestrate a command that isn't a Dagster op (or isn't written in Python). Usage:

    from dagster_k8s import k8s_job_op

... (truncated)

Commits
  • 8f9f0d0 0.15.5
  • 39fecd4 Fix bug with how resources are applied in materialize (#8790)
  • 798e9c1 0.15.5 Changelog (#8781)
  • a42146f fix issue with repos and partitioned scheduled asset jobs (#8779)
  • 2fe13df improve error for build_schedule_from_partitioned_job with non-partitioned as...
  • e40122e fix source asset regression (#8784)
  • eeeb82c [dagit] Bring back the global asset graph as an “all asset groups” view (#8709)
  • 1544643 Ensure graph inputs/outputs are included in all_dagster_types (#8736)
  • 87e291e Fix config case for default executor (#8777)
  • 4058bb5 Fold asset_defs submodule into main dagster structure (#8446)
  • Additional commits viewable in compare view


Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
dependabot[bot] commented 1 year ago

Looks like dagster is no longer updatable, so this is no longer needed.