oap-project / raydp

RayDP provides simple APIs for running Spark on Ray and integrating Spark with AI libraries.
Apache License 2.0
308 stars 68 forks source link

xgboost_ray example will fail when xgboost-ray is upgraded to 0.1.14 or above #307

Closed KepingYan closed 1 year ago

KepingYan commented 1 year ago
Traceback (most recent call last):
  File "/home/rainbow/miniconda3/envs/test_spark/lib/python3.7/site-packages/xgboost_ray/main.py", line 1181, in _train
    ray.get(ready)
  File "/home/rainbow/miniconda3/envs/test_spark/lib/python3.7/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
    return func(*args, **kwargs)
  File "/home/rainbow/miniconda3/envs/test_spark/lib/python3.7/site-packages/ray/_private/worker.py", line 2309, in get
PC: @     0x7f9ddb1e3c90  (unknown)  accept4
    @     0x7f9ddb42c3c0  (unknown)  (unknown)
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(TypeError): ray::_RemoteRayXGBoostActor.train() (pid=1649778, ip=10.239.158.60, repr=<xgboost_ray.main._RemoteRayXGBoostActor object at 0x7f8fe6c50dd0>)
  File "/home/rainbow/miniconda3/envs/test_spark/lib/python3.7/site-packages/xgboost_ray/main.py", line 630, in train
    self.load_data(dtrain)
  File "/home/rainbow/miniconda3/envs/test_spark/lib/python3.7/site-packages/xgboost_ray/main.py", line 596, in load_data
    param = data.get_data(self.rank, self.num_actors)
  File "/home/rainbow/miniconda3/envs/test_spark/lib/python3.7/site-packages/xgboost_ray/matrix.py", line 847, in get_data
    self.load_data(num_actors=num_actors, rank=rank)
  File "/home/rainbow/miniconda3/envs/test_spark/lib/python3.7/site-packages/xgboost_ray/matrix.py", line 834, in load_data
    self.num_actors, self.sharding, rank=rank)
  File "/home/rainbow/miniconda3/envs/test_spark/lib/python3.7/site-packages/xgboost_ray/matrix.py", line 558, in load_data
    **self.kwargs)
  File "/home/rainbow/miniconda3/envs/test_spark/lib/python3.7/site-packages/xgboost_ray/data_sources/ray_dataset.py", line 60, in load_data
    local_df = [ds.to_pandas(limit=DATASET_TO_PANDAS_LIMIT) for ds in data]
  File "/home/rainbow/miniconda3/envs/test_spark/lib/python3.7/site-packages/xgboost_ray/data_sources/ray_dataset.py", line 60, in <listcomp>
    local_df = [ds.to_pandas(limit=DATASET_TO_PANDAS_LIMIT) for ds in data]
  File "/home/rainbow/miniconda3/envs/test_spark/lib/python3.7/site-packages/ray/data/dataset.py", line 3391, in to_pandas
    output.add_block(ray.get(block))
  File "/home/rainbow/miniconda3/envs/test_spark/lib/python3.7/site-packages/ray/data/_internal/delegating_block_builder.py", line 56, in add_block
    self._builder.add_block(block)
  File "/home/rainbow/miniconda3/envs/test_spark/lib/python3.7/site-packages/ray/data/_internal/table_block.py", line 74, in add_block
    f"Got a block of type {type(block)}, expected {self._block_type}."
TypeError: Got a block of type <class 'bytes'>, expected <class 'pyarrow.lib.Table'>.If you are mapping a function, ensure it returns an object with the expected type. Block:
b'\xff\xff\xff\xff\xd8\x05\x00\x00\x10\x00\x00\x00\x00\x00\n\x00\x0e\x00\x06\x00\r\x00\x08\x00\n\x00\x00\x00\x00\x00\x04\x00\x10\x00\x00\x00\x00\x01\n\x00\x0c\x00\x0...
kira-lin commented 1 year ago

This has been fixed, it turns out to be a bug in Ray to_pandas