I'm using AWS lambda to run a python application. I run the load_dataset function with cache_dir="/tmp" and is still throws the OSError(30, 'Read-only file system') error. Is even updated all the HF envs to point to /tmp dir but the issue still persists. I can confirm that the I can write to /tmp directory.
Steps to reproduce the bug
d = load_dataset(
path=hugging_face_link,
split=split,
token=token,
cache_dir="/tmp/hugging_face_cache",
)
Expected behavior
Everything written to the file system as part of the load_datasets function should be in the /tmp directory.
Describe the bug
I'm using AWS lambda to run a python application. I run the
load_dataset
function with cache_dir="/tmp" and is still throws the OSError(30, 'Read-only file system') error. Is even updated all the HF envs to point to /tmp dir but the issue still persists. I can confirm that the I can write to /tmp directory.Steps to reproduce the bug
Expected behavior
Everything written to the file system as part of the load_datasets function should be in the /tmp directory.
Environment info
datasets version: 2.16.1 Platform: Linux-5.10.216-225.855.amzn2.x86_64-x86_64-with-glibc2.26 Python version: 3.11.9 huggingface_hub version: 0.19.4 PyArrow version: 16.1.0 Pandas version: 2.2.2 fsspec version: 2023.10.0