search

tensorflow / datasets

TFDS is a collection of datasets ready to use with TensorFlow, Jax, ...
https://www.tensorflow.org/datasets
Apache License 2.0
4.28k stars 1.53k forks source link

gs' not implemented #5324

Open zelongjiang123 opened 6 months ago

zelongjiang123 commented 6 months ago

/!\ PLEASE INCLUDE THE FULL STACKTRACE AND CODE SNIPPET

Short description Description of the bug.

Environment information

Reproduction instructions

git clone https://github.com/tensorflow/datasets.git
cd datasets
pip install -e .
python -c "import tensorflow_datasets as tfds; tfds = tfds.load('berkeley_autolab_ur5')"

Link to logs

Traceback (most recent call last):
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\utils\py_utils.py", line 420, in try_reraise
    yield
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\load.py", line 222, in builder
    return cls(**builder_kwargs)  # pytype: disable=not-instantiable
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\logging\__init__.py", line 288, in decorator
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\dataset_builder.py", line 1362, in __init__
    super().__init__(**kwargs)
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\logging\__init__.py", line 288, in decorator
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\dataset_builder.py", line 285, in __init__
    self.info.initialize_from_bucket()
    ^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\logging\__init__.py", line 168, in __call__
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\dataset_builder.py", line 474, in info
    info = self._info()
           ^^^^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\robotics\dataset_importer_builder.py", line 82, in _info
    features = self.get_ds_builder().info.features
               ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\robotics\dataset_importer_builder.py", line 150, in get_ds_builder    
    ds_builder = tfds.builder_from_directory(ds_location)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\read_only_builder.py", line 149, in builder_from_directory       
    return ReadOnlyBuilder(builder_dir=builder_dir)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\logging\__init__.py", line 288, in decorator
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\read_only_builder.py", line 65, in __init__
    info_proto = dataset_info.read_proto_from_builder_dir(builder_dir)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\dataset_info.py", line 1059, in read_proto_from_builder_dir      
    return read_from_json(info_path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\dataset_info.py", line 1035, in read_from_json
    json_str = epath.Path(path).read_text()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Zelong\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\etils\epath\abstract_path.py", line 157, in read_text
    return f.read()
           ^^^^^^^^
  File "C:\Users\Zelong\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\lib\io\file_io.py", line 116, in read
    self._preread_check()
  File "C:\Users\Zelong\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\lib\io\file_io.py", line 77, in _preread_check
    self._read_buf = _pywrap_file_io.BufferedInputStream(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tensorflow.python.framework.errors_impl.UnimplementedError: File system scheme 'gs' not implemented (file: 'gs://gresearch/robotics/berkeley_autolab_ur5/0.1.0/dataset_info.json')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\logging\__init__.py", line 168, in __call__
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\load.py", line 643, in load
    dbuilder = _fetch_builder(
               ^^^^^^^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\load.py", line 498, in _fetch_builder
    return builder(name, data_dir=data_dir, try_gcs=try_gcs, **builder_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2288.0_x64__qbz5n2kfra8p0\Lib\contextlib.py", line 81, in inner   
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\logging\__init__.py", line 168, in __call__
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\load.py", line 219, in builder
    with py_utils.try_reraise(
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2288.0_x64__qbz5n2kfra8p0\Lib\contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\utils\py_utils.py", line 422, in try_reraise
    reraise(e, *args, **kwargs)
  File "C:\Users\Zelong\Desktop\Berkeley data\datasets\tensorflow_datasets\core\utils\py_utils.py", line 389, in reraise
    raise exception from e
RuntimeError: UnimplementedError: Failed to construct dataset "berkeley_autolab_ur5", builder_kwargs "{'data_dir': None}": File system scheme 'gs' not implemented (file: 'gs://gresearch/robotics/berkeley_autolab_ur5/0.1.0/dataset_info.json')

Expected behavior It should find and download the dataset

fylux commented 6 months ago

Hi, some questions:

  1. Is the command that you posted what triggers the error? Or you have somewhere a command trying to build the dataset "berkeley_autolab_ur5" ?
git clone https://github.com/tensorflow/datasets.git
cd datasets
pip install -e .
python -c "import tensorflow as tf; print('TensorFlow version:', tf.__version__)"
  1. Can you paste the output of pip install -e . ? It could be that you are missing some dependency.
zelongjiang123 commented 5 months ago

Hi, some questions:

  1. Is the command that you posted what triggers the error? Or you have somewhere a command trying to build the dataset "berkeley_autolab_ur5" ?
git clone https://github.com/tensorflow/datasets.git
cd datasets
pip install -e .
python -c "import tensorflow as tf; print('TensorFlow version:', tf.__version__)"
  1. Can you paste the output of pip install -e . ? It could be that you are missing some dependency.

for q1: yes. the command is 

python -c "import tensorflow_datasets as tfds; tfds = tfds.load('berkeley_autolab_ur5')"

q2: output is

PS C:\Users\Zelong\Desktop\Berkeley data\datasets> pip install -e .
Obtaining file:///C:/Users/Zelong/Desktop/Berkeley%20data/datasets
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Requirement already satisfied: absl-py in c:\users\zelong\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-datasets==4.9.4+nightly) (1.4.0)
Requirement already satisfied: click in c:\users\zelong\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-datasets==4.9.4+nightly) (8.1.7)
Requirement already satisfied: dm-tree in c:\users\zelong\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-datasets==4.9.4+nightly) (0.1.8)
Requirement already satisfied: etils>=1.6.0 in c:\users\zelong\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from etils[enp,epath,epy,etree]>=1.6.0->tensorflow-datasets==4.9.4+nightly) (1.7.0)
Requirement already satisfied: immutabledict in c:\users\zelong\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-datasets==4.9.4+nightly) (4.2.0)
Requirement already satisfied: numpy in c:\users\zelong\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-datasets==4.9.4+nightly) (1.26.4)
Requirement already satisfied: promise in c:\users\zelong\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-datasets==4.9.4+nightly) (2.3)
Requirement already satisfied: protobuf>=3.20 in c:\users\zelong\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-datasets==4.9.4+nightly) (3.20.3)
Requirement already satisfied: psutil in c:\users\zelong\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-datasets==4.9.4+nightly) (5.9.8)
Requirement already satisfied: pyarrow in c:\users\zelong\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-datasets==4.9.4+nightly) (15.0.0)
Requirement already satisfied: requests>=2.19.0 in c:\users\zelong\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-datasets==4.9.4+nightly) (2.31.0)
Requirement already satisfied: tensorflow-metadata in c:\users\zelong\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-datasets==4.9.4+nightly) (1.14.0)
Requirement already satisfied: termcolor in c:\users\zelong\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-datasets==4.9.4+nightly) (2.4.0)
Requirement already satisfied: toml in c:\users\zelong\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-datasets==4.9.4+nightly) (0.10.2)
Requirement already satisfied: tqdm in c:\users\zelong\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-datasets==4.9.4+nightly) (4.66.2)
Requirement already satisfied: wrapt in c:\users\zelong\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-datasets==4.9.4+nightly) (1.14.1)
Requirement already satisfied: fsspec in c:\users\zelong\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from etils[enp,epath,epy,etree]>=1.6.0->tensorflow-datasets==4.9.4+nightly) (2024.2.0)
Requirement already satisfied: importlib_resources in c:\users\zelong\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from etils[enp,epath,epy,etree]>=1.6.0->tensorflow-datasets==4.9.4+nightly) (6.1.2)
Requirement already satisfied: typing_extensions in c:\users\zelong\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from etils[enp,epath,epy,etree]>=1.6.0->tensorflow-datasets==4.9.4+nightly) (4.10.0)
Requirement already satisfied: zipp in c:\users\zelong\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from etils[enp,epath,epy,etree]>=1.6.0->tensorflow-datasets==4.9.4+nightly) (3.17.0)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\zelong\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from requests>=2.19.0->tensorflow-datasets==4.9.4+nightly) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in c:\users\zelong\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from requests>=2.19.0->tensorflow-datasets==4.9.4+nightly) (3.6)
Requirement already satisfied: urllib3<3,>=1.21.1 in c:\users\zelong\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from requests>=2.19.0->tensorflow-datasets==4.9.4+nightly) (2.2.1)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\zelong\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from requests>=2.19.0->tensorflow-datasets==4.9.4+nightly) (2024.2.2)
Requirement already satisfied: colorama in c:\users\zelong\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from click->tensorflow-datasets==4.9.4+nightly) (0.4.6)
Requirement already satisfied: six in c:\users\zelong\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from promise->tensorflow-datasets==4.9.4+nightly) (1.16.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in c:\users\zelong\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-metadata->tensorflow-datasets==4.9.4+nightly) (1.62.0)
Building wheels for collected packages: tensorflow-datasets
  Building editable for tensorflow-datasets (pyproject.toml) ... done
  Created wheel for tensorflow-datasets: filename=tensorflow_datasets-4.9.4+nightly-0.editable-py3-none-any.whl size=9609 sha256=779ea94611e63c8c81ea76742f6a7c89a005c2b977260b5bc34307c149a67f81
  Stored in directory: C:\Users\Zelong\AppData\Local\Temp\pip-ephem-wheel-cache-w1kcf9us\wheels\1e\d5\9e\7330eb962857f089b89be7c5eb1f5c911033052a0a1cbfff31
Successfully built tensorflow-datasets
Installing collected packages: tensorflow-datasets
  Attempting uninstall: tensorflow-datasets
    Found existing installation: tensorflow-datasets 4.9.4
    Uninstalling tensorflow-datasets-4.9.4:
      Successfully uninstalled tensorflow-datasets-4.9.4
  WARNING: The script tfds.exe is installed in 'C:\Users\Zelong\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed tensorflow-datasets-4.9.4+nightly