Extraction completed...: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11801/11801 [00:31<00:00, 372.03 file/s]
Dl Size...: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1097/1097 [00:31<00:00, 34.58 MiB/s]
Dl Completed...: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:31<00:00, 31.72s/ url]
Traceback (most recent call last):
File "/lfs/ampere4/0/brando9/miniconda/envs/mds_env_gpu/lib/python3.9/site-packages/tensorflow_datasets/core/utils/py_utils.py", line 325, in incomplete_dir
File "/lfs/ampere4/0/brando9/miniconda/envs/mds_env_gpu/lib/python3.9/site-packages/tensorflow_datasets/core/dataset_builder.py", line 600, in download_and_prepare
File "/lfs/ampere4/0/brando9/miniconda/envs/mds_env_gpu/lib/python3.9/site-packages/tensorflow_datasets/core/dataset_builder.py", line 1399, in _download_and_prepare
File "/lfs/ampere4/0/brando9/miniconda/envs/mds_env_gpu/lib/python3.9/site-packages/tensorflow_datasets/core/split_builder.py", line 326, in submit_split_generation
File "/lfs/ampere4/0/brando9/miniconda/envs/mds_env_gpu/lib/python3.9/site-packages/tensorflow_datasets/core/split_builder.py", line 389, in _build_from_generator
File "/lfs/ampere4/0/brando9/miniconda/envs/mds_env_gpu/lib/python3.9/site-packages/tqdm/std.py", line 1185, in __iter__
File "/afs/cs.stanford.edu/u/brando9/diversity-for-predictive-success-of-meta-learning/meta-dataset/meta_dataset/data/tfds/example_generators.py", line 178, in generate_cu_birds_examples
File "/afs/cs.stanford.edu/u/brando9/diversity-for-predictive-success-of-meta-learning/meta-dataset/meta_dataset/data/tfds/example_generators.py", line 77, in _load_and_process_image
File "/lfs/ampere4/0/brando9/miniconda/envs/mds_env_gpu/lib/python3.9/site-packages/etils/epath/abstract_path.py", line 139, in read_bytes
File "/lfs/ampere4/0/brando9/miniconda/envs/mds_env_gpu/lib/python3.9/site-packages/etils/epath/gpath.py", line 226, in open
File "/lfs/ampere4/0/brando9/miniconda/envs/mds_env_gpu/lib/python3.9/site-packages/etils/epath/backend.py", line 104, in open
OSError: [Errno 24] Too many open files: '/lfs/ampere4/0/brando9/tensorflow_datasets/downloads/extracted/TAR_GZ.ucexport_download_id_1hbzc_P1FuxMkcabkgn9ZKinBDGhd9Vl6iySQn2p8nbbRHgCHM3eaZxdgr-94_rSb8IE/CUB_200_2011/images/193.Bewick_Wren/Bewick_Wren_0050_184919.jpg'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/lfs/ampere4/0/brando9/miniconda/envs/mds_env_gpu/bin/tfds", line 8, in <module>
File "/lfs/ampere4/0/brando9/miniconda/envs/mds_env_gpu/lib/python3.9/site-packages/tensorflow_datasets/scripts/cli/main.py", line 104, in launch_cli
File "/lfs/ampere4/0/brando9/miniconda/envs/mds_env_gpu/lib/python3.9/site-packages/absl/app.py", line 303, in run
File "/lfs/ampere4/0/brando9/miniconda/envs/mds_env_gpu/lib/python3.9/site-packages/absl/app.py", line 251, in _run_main
File "/lfs/ampere4/0/brando9/miniconda/envs/mds_env_gpu/lib/python3.9/site-packages/tensorflow_datasets/scripts/cli/main.py", line 99, in main
File "/lfs/ampere4/0/brando9/miniconda/envs/mds_env_gpu/lib/python3.9/site-packages/tensorflow_datasets/scripts/cli/build.py", line 233, in _build_datasets
File "/lfs/ampere4/0/brando9/miniconda/envs/mds_env_gpu/lib/python3.9/site-packages/tensorflow_datasets/scripts/cli/build.py", line 435, in _download_and_prepare
File "/lfs/ampere4/0/brando9/miniconda/envs/mds_env_gpu/lib/python3.9/site-packages/tensorflow_datasets/core/dataset_builder.py", line 613, in download_and_prepare
File "/lfs/ampere4/0/brando9/miniconda/envs/mds_env_gpu/lib/python3.9/contextlib.py", line 137, in __exit__
File "/lfs/ampere4/0/brando9/miniconda/envs/mds_env_gpu/lib/python3.9/site-packages/tensorflow_datasets/core/utils/py_utils.py", line 329, in incomplete_dir
File "/lfs/ampere4/0/brando9/miniconda/envs/mds_env_gpu/lib/python3.9/site-packages/tensorflow/python/lib/io/file_io.py", line 674, in delete_recursively_v2
tensorflow.python.framework.errors_impl.ResourceExhaustedError: /lfs/ampere4/0/brando9/tensorflow_datasets/meta_dataset/cu_birds/1.0.0.incompleteSRZTPR; Too many open files
brando90
commented
1 year ago