ray-project / ray

Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
https://ray.io
Apache License 2.0
33.05k stars 5.59k forks source link

[Data] Pending object memory store usage in the progress bar should incorporate estimated size of output blocks (obj_store_mem_pending_task_outputs) #47125

Open jeffreyjeffreywang opened 1 month ago

jeffreyjeffreywang commented 1 month ago

What happened + What you expected to happen

Pending resource usage is introduced to the progress bar in #46729 -- number of pending actors scheduled on CPU and GPU are now displayed in progress bar. We should also include obj_store_mem_pending_task_outputs as the pending object store usage. https://github.com/ray-project/ray/blob/82af058f3456dd7088b94a8d29a95450f73d215b/python/ray/data/_internal/execution/interfaces/op_runtime_metrics.py#L338

Versions / Dependencies

ray master

Reproduction script

Use the following script to spin up actors to be executed on GPUs.

import time
import ray

ray.init(num_gpus=1)

class Identity:
    def __call__(self, batch):
        time.sleep(1)
        return batch

ray.data.range(10, override_num_blocks=10).map_batches(
    Identity, num_gpus=1, batch_size=1, concurrency=(1, 2)
).materialize()

Issue Severity

Low: It annoys or frustrates me.

raulchen commented 4 weeks ago

cc @bveeramani