search

ssec-jhu / dplutils

Distributed(Data) Pipeline Uitilities
BSD 3-Clause "New" or "Revised" License
1 stars 0 forks source link

Bump ray[default] from 2.9.3 to 2.10.0 #69

Closed dependabot[bot] closed 4 months ago

dependabot[bot] commented 4 months ago

Bumps ray[default] from 2.9.3 to 2.10.0.

Release notes

Sourced from ray[default]'s releases.

Ray-2.10.0

Release Highlights

Ray 2.10 release brings important stability improvements and enhancements to Ray Data, with Ray Data becoming generally available (GA).

  • [Data] Ray Data becomes generally available with stability improvements in streaming execution, reading and writing data, better tasks concurrency control, and debuggability improvement with dashboard, logging and metrics visualization.
  • [RLlib] “New API Stack” officially announced as alpha for PPO and SAC.
  • [Serve] Added a default autoscaling policy set via num_replicas=”auto” (#42613).
  • [Serve] Added support for active load shedding via max_queued_requests (#42950).
  • [Serve] Added replica queue length caching to the DeploymentHandle scheduler (#42943).
    • This should improve overhead in the Serve proxy and handles.
    • max_ongoing_requests (max_concurrent_queries) is also now strictly enforced (#42947).
    • If you see any issues, please report them on GitHub and you can disable this behavior by setting: RAY_SERVE_ENABLE_QUEUE_LENGTH_CACHE=0.
  • [Serve] Renamed the following parameters. Each of the old names will be supported for another release before removal.
    • max_concurrent_queries -> max_ongoing_requests
    • target_num_ongoing_requests_per_replica -> target_ongoing_requests
    • downscale_smoothing_factor -> downscaling_factor
    • upscale_smoothing_factor -> upscaling_factor
  • [Serve] WARNING: the following default values will change in Ray 2.11:
    • Default for max_ongoing_requests will change from 100 to 5.
    • Default for target_ongoing_requests will change from 1 to 2.
  • [Core] Autoscaler v2 is in alpha and can be tried out with Kuberay. It has improved observability and stability compared to v1.
  • [Train] Added support for accelerator types via ScalingConfig(accelerator_type).
  • [Train] Revamped the XGBoostTrainer and LightGBMTrainer to no longer depend on xgboost_ray and lightgbm_ray. A new, more flexible API will be released in a future release.
  • [Train/Tune] Refactored local staging directory to remove the need for local_dir and RAY_AIR_LOCAL_CACHE_DIR.

Ray Libraries

Ray Data

🎉 New Features:

  • Streaming execution stability improvement to avoid memory issue, including per-operator resource reservation, streaming generator output buffer management, and better runtime resource estimation (#43026, #43171, #43298, #43299, #42930, #42504)
  • Metadata read stability improvement to avoid AWS transient error, including retry on application-level exception, spread tasks across multiple nodes, and configure retry interval (#42044, #43216, #42922, #42759).
  • Allow tasks concurrency control for read, map, and write APIs (#42849, #43113, #43177, #42637)
  • Data dashboard and statistics improvement with more runtime metrics for each components (#43790, #43628, #43241, #43477, #43110, #43112)
  • Allow to specify application-level error to retry for actor task (#42492)
  • Add num_rows_per_file parameter to file-based writes (#42694)
  • Add DataIterator.materialize (#43210)
  • Skip schema call in DataIterator.to_tf if tf.TypeSpec is provided (#42917)
  • Add option to append for Dataset.write_bigquery (#42584)
  • Deprecate legacy components and classes (#43575, #43178, #43347, #43349, #43342, #43341, #42936, #43144, #43022, #43023)

💫 Enhancements:

  • Restructure stdout logging for better readability (#43360)
  • Add a more performant way to read large TFRecord datasets (#42277)
  • Modify ImageDatasource to use Image.BILINEAR as the default image resampling filter (#43484)
  • Reduce internal stack trace output by default (#43251)
  • Perform incremental writes to Parquet files (#43563)
  • Warn on excessive driver memory usage during shuffle ops (#42574)
  • Distributed reads for ray.data.from_huggingface (#42599)

... (truncated)

Commits


Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
amitschang commented 4 months ago

I'll look into how we can upgrade

codecov[bot] commented 4 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 100.00%. Comparing base (c09e53b) to head (69c25ac). Report is 2 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #69 +/- ## ========================================= Coverage 100.00% 100.00% ========================================= Files 11 11 Lines 576 576 ========================================= Hits 576 576 ``` | [Flag](https://app.codecov.io/gh/ssec-jhu/dplutils/pull/69/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ssec-jhu) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/ssec-jhu/dplutils/pull/69/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ssec-jhu) | `100.00% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ssec-jhu#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

amitschang commented 4 months ago

@jamienoss, cool thanks. I don't there there are anymore gotchas here - in the end I think we will deprecate the ray.data version of executor anyhow as it is really meant for another kind of workflow