NotImplementedError: Loading a dataset cached in a LocalFileSystem is not supported.

[umuiq]

Hello, author. Thank you for sharing the embodied intelligence GitHub code. I have configured the environment as instructed in the readme and downloaded the 1.3B model. However, I encountered the following issue while running the evaluation code eval_gen_1.3B. Could you please advise on how to resolve this?

eval_gen_1.3B.sh: 7: [: plan_gen_unseen: unexpected operator ===================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

bin /data/yinzhaoran/anaconda3/envs/wmflm/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda117.so /data/yinzhaoran/anaconda3/envs/wmflm/lib/python3.11/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: /data/yinzhaoran/anaconda3/envs/wmflm did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths... warn(msg) /data/yinzhaoran/anaconda3/envs/wmflm/lib/python3.11/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: :/data/yinzhaoran/my_usr/usr/:/data/yinzhaoran/my_usr/usr/lib/x86_64-linux-gnu:/data/yinzhaoran/my_usr/usr/:/data/yinzhaoran/my_usr/usr/lib/x86_64-linux-gnu did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths... warn(msg) /data/yinzhaoran/anaconda3/envs/wmflm/lib/python3.11/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('socks5'), PosixPath('7890'), PosixPath('//127.0.0.1')} warn(msg) /data/yinzhaoran/anaconda3/envs/wmflm/lib/python3.11/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/run/user/1009/vscode-ipc-e7b1549a-e585-48c4-b646-8aa9b9a954a0.sock')} warn(msg) /data/yinzhaoran/anaconda3/envs/wmflm/lib/python3.11/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('http'), PosixPath('7890'), PosixPath('//127.0.0.1')} warn(msg) CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths... /data/yinzhaoran/anaconda3/envs/wmflm/lib/python3.11/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/usr/local/cuda/lib64/libcudart.so.11.0'), PosixPath('/usr/local/cuda/lib64/libcudart.so')}.. We'll flip a coin and try one of these, in order to fail forward. Either way, this might cause trouble in the future: If you get CUDA error: invalid device function errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env. warn(msg) CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so.11.0 CUDA SETUP: Highest compute capability among GPUs detected: 8.0 CUDA SETUP: Detected CUDA version 117 CUDA SETUP: Loading binary /data/yinzhaoran/anaconda3/envs/wmflm/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cuda117.so... Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher. Found cached dataset json (file:///data/yinzhaoran/.cache/huggingface/datasets/json/default-1e0345f1a4034634/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4) Traceback (most recent call last): File "/data/yinzhaoran/wmflm/world-model-for-language-model/eval_gen.py", line 149, in fire.Fire(main) File "/data/yinzhaoran/anaconda3/envs/wmflm/lib/python3.11/site-packages/fire/core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/yinzhaoran/anaconda3/envs/wmflm/lib/python3.11/site-packages/fire/core.py", line 475, in _Fire component, remaining_args = _CallAndUpdateTrace( ^^^^^^^^^^^^^^^^^^^^ File "/data/yinzhaoran/anaconda3/envs/wmflm/lib/python3.11/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^ File "/data/yinzhaoran/wmflm/world-model-for-language-model/eval_gen.py", line 51, in main eval_dataset = load_dataset(val_file.split(".")[-1], data_files={'validation': val_file})['validation'] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/yinzhaoran/anaconda3/envs/wmflm/lib/python3.11/site-packages/datasets/load.py", line 1810, in load_dataset ds = builder_instance.as_dataset(split=split, verification_mode=verification_mode, in_memory=keep_in_memory) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/yinzhaoran/anaconda3/envs/wmflm/lib/python3.11/site-packages/datasets/builder.py", line 1107, in as_dataset raise NotImplementedError(f"Loading a dataset cached in a {type(self._fs).name} is not supported.") NotImplementedError: Loading a dataset cached in a LocalFileSystem is not supported.

[szxiangjn]

Hi,

I am not sure but it looks relevant to https://github.com/huggingface/datasets/issues/6352. Also please make sure you put the dataset at the right path.

[umuiq]

Nice！This successfully resolved my issue！！