xorbitsai / xorbits

Scalable Python DS & ML, in an API compatible & lightning fast way.
https://xorbits.readthedocs.io
Apache License 2.0
1.11k stars 67 forks source link

BUG: ValueError: memoryview assignment: lvalue and rvalue have different structures #638

Closed codingl2k1 closed 1 year ago

codingl2k1 commented 1 year ago

Describe the bug

A clear and concise description of what the bug is.

To Reproduce

To help us to reproduce this bug, please provide information below:

Put a pyarrow table to store.

  File "/Users/codingl2k1/Work/xorbits/python/xorbits/_mars/services/subtask/worker/processor.py", line 616, in run
    ) = await self._store_data(chunk_graph)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/codingl2k1/Work/xorbits/python/xorbits/_mars/services/subtask/worker/processor.py", line 346, in _store_data
    store_infos = await put_infos
                  ^^^^^^^^^^^^^^^
  File "/Users/codingl2k1/Work/xoscar/python/xoscar/batch.py", line 149, in _async_batch
    return await self.batch_func(args_list, kwargs_list)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/codingl2k1/Work/xorbits/python/xorbits/_mars/services/storage/api/oscar.py", line 130, in batch_put
    return await self._storage_handler_ref.put.batch(*puts)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "xoscar/core.pyx", line 266, in __pyx_actor_method_wrapper
  File "xoscar/core.pyx", line 269, in xoscar.core.__pyx_actor_method_wrapper
  File "/Users/codingl2k1/Work/xoscar/python/xoscar/batch.py", line 149, in _async_batch
    return await self.batch_func(args_list, kwargs_list)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/codingl2k1/Work/xorbits/python/xorbits/_mars/services/storage/handler.py", line 258, in batch_put
    object_info = await self._clients[level].put(obj)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/codingl2k1/Work/xorbits/python/xorbits/_mars/storage/shared_memory.py", line 179, in put
    await f.write(buffer)
  File "/Users/codingl2k1/Work/xoscar/python/xoscar/aio/base.py", line 26, in method
    return await self._loop.run_in_executor(self._executor, func)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/codingl2k1/.pyenv/versions/3.11.4/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/codingl2k1/Work/xorbits/python/xorbits/_mars/storage/core.py", line 127, in write
    self._mv[offset:new_offset] = content
    ~~~~~~~~^^^^^^^^^^^^^^^^^^^
ValueError: [address=127.0.0.1:53344, pid=15386] memoryview assignment: lvalue and rvalue have different structures

memoryview_assignment_failed

  1. Your Python version
  2. The version of Xorbits you use
  3. Versions of crucial packages, such as numpy, scipy and pandas
  4. Full stack of the error.
  5. Minimized code to reproduce the error.

Expected behavior

A clear and concise description of what you expected to happen.

Additional context

Add any other context about the problem here.

related issue: https://bugs.python.org/issue34905