search

mars-project / mars

Mars is a tensor-based unified framework for large-scale data computation which scales numpy, pandas, scikit-learn and Python functions.
https://mars-project.readthedocs.io
Apache License 2.0
2.68k stars 325 forks source link

[BUG] anaconda3/lib/python3.8/uuid.py", line 180, raise ValueError('bytes is not a 16-char string') #3325

Closed kexuedaishu closed 1 year ago

kexuedaishu commented 1 year ago

Code:

from mars.session import new_session new_session().as_default()

Get error: anaconda3/lib/python3.8/uuid.py", line 180, in init raise ValueError('bytes is not a 16-char string') ValueError: bytes is not a 16-char string

Error under Linux version 3.10.0-957.10.1.el7.x86_64.

fyrestone commented 1 year ago

Could you provide the full traceback details?

kexuedaishu commented 1 year ago

Traceback (most recent call last): File "/mnt/nfs/yijing.zhou/venv_38/lib/python3.8/site-packages/mars/services/cluster/uploader.py", line 114, in upload_node_info self._info.env = await asyncio.to_thread(gather_node_env) File "/mnt/nfs/yijing.zhou/venv_38/lib/python3.8/site-packages/mars/lib/aio/_threads.py", line 36, in to_thread return await loop.run_in_executor(None, func_call) File "/mnt/nfs/zhihui/anaconda3/lib/python3.8/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/mnt/nfs/yijing.zhou/venv_38/lib/python3.8/site-packages/mars/services/cluster/gather.py", line 73, in gather_node_env cuda_info = mars_resource.cuda_info() File "/mnt/nfs/yijing.zhou/venv_38/lib/python3.8/site-packages/mars/resource.py", line 352, in cuda_info products=[nvutils.get_device_info(idx).name for idx in range(gpu_count)], File "/mnt/nfs/yijing.zhou/venv_38/lib/python3.8/site-packages/mars/resource.py", line 352, in products=[nvutils.get_device_info(idx).name for idx in range(gpu_count)], File "/mnt/nfs/yijing.zhou/venv_38/lib/python3.8/site-packages/mars/lib/nvutils.py", line 343, in get_device_info uuid=uuid.UUID(bytes=uuid_t.bytes), File "/mnt/nfs/zhihui/anaconda3/lib/python3.8/uuid.py", line 180, in init raise ValueError('bytes is not a 16-char string') ValueError: bytes is not a 16-char string Failed to upload node info: bytes is not a 16-char string

qinxuye commented 1 year ago

Which Mars version did you use?

kexuedaishu commented 1 year ago

0.9.0a1

fyrestone commented 1 year ago

0.9.0a1

Can you try the latest version?

kexuedaishu commented 1 year ago

yes, actually the older version 0.6.1 was used, but have to upgrade to the newer version and encountered such errors...

qinxuye commented 1 year ago

Latest version v0.10.0 may address your issue, could you try it please?

flpractioner commented 1 year ago

I just tried, the latest version v0.10.0 doesn't work as well.

qinxuye commented 1 year ago

Looks like sth went wrong when getting info from cuda device, can you give more info about your cuda setting?

flpractioner commented 1 year ago

CUDA Version 9.0.176 CUDA Patch Version 9.0.176.1 CUDA Patch Version 9.0.176.2 CUDA Patch Version 9.0.176.3 CUDA Patch Version 9.0.176.4

qinxuye commented 1 year ago

I try to run the function on my computer with 2 GPU cards, but no error occurs.

>>> from mars.lib.nvutils import get_device_info
>>> get_device_info(0)
_cu_device_info(index=0, uuid=UUID('096adb53-fc18-1075-1097-5792e93051b7'), name='NVIDIA Graphics Device', multiprocessors=84, cuda_cores=16128, threads=129024)

Could you try this one?

flpractioner commented 1 year ago 
>>> from mars.lib.nvutils import get_device_info
>>> get_device_info(0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/***/pkgs/mars-0.10.0/mars/lib/nvutils.py", line 343, in get_device_info
    uuid=uuid.UUID(bytes=uuid_t.bytes),
  File "/****/anaconda3/lib/python3.8/uuid.py", line 180, in __init__
    raise ValueError('bytes is not a 16-char string')
ValueError: bytes is not a 16-char string