Open esparig opened 2 years ago
Rather than generating the imagenet inside the docker image, could your pre-generate imagenet .tfrecord
(e.g. with tfds build imagenet2012 --manual_dir=...
CLI), then only package the ~/tensorflow_datasets/imagenet2012/...
rather than the original ILSVRC2012_devkit_t12.tar.gz
Are you saying that is not possible to use TFDS from a Tensorflow Docker container? Why? Is it a problem with TF or with Docker? I already did the conversion to .tfrecords using another script, but I wanted to try tfds... The thing is I need to use containers in that host.
I was just proposing a workaround. Could you try to comment the following line: https://github.com/tensorflow/datasets/blob/30024eefca3aa0783e2374af32766717267335d0/tensorflow_datasets/core/dataset_builder.py#L404
We're using shutil.disk_usage
to estimate the available space. This might conflict for some reason with docker. Commenting the line will ignore the error, so we can be sure this is the cause.
This is strage, I did what you suggested and I get the following error:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/official/vision/beta/train.py", line 23, in <module>
from official.common import registry_imports
File "/usr/local/lib/python3.6/dist-packages/official/common/registry_imports.py", line 18, in <module>
from official.nlp.configs import experiment_configs
File "/usr/local/lib/python3.6/dist-packages/official/nlp/configs/experiment_configs.py", line 17, in <module>
from official.nlp.configs import finetuning_experiments
File "/usr/local/lib/python3.6/dist-packages/official/nlp/configs/finetuning_experiments.py", line 20, in <module>
from official.nlp.data import question_answering_dataloader
File "/usr/local/lib/python3.6/dist-packages/official/nlp/data/question_answering_dataloader.py", line 21, in <module>
from official.core import input_reader
File "/usr/local/lib/python3.6/dist-packages/official/core/input_reader.py", line 21, in <module>
import tensorflow_datasets as tfds
File "/usr/local/lib/python3.6/dist-packages/tensorflow_datasets/__init__.py", line 43, in <module>
from tensorflow_datasets.core import tf_compat
File "/usr/local/lib/python3.6/dist-packages/tensorflow_datasets/core/__init__.py", line 26, in <module>
from tensorflow_datasets.core import community # pylint: disable=g-bad-import-order
File "/usr/local/lib/python3.6/dist-packages/tensorflow_datasets/core/community/__init__.py", line 18, in <module>
from tensorflow_datasets.core.community.huggingface_wrapper import mock_builtin_to_use_gfile
File "/usr/local/lib/python3.6/dist-packages/tensorflow_datasets/core/community/huggingface_wrapper.py", line 28, in <module>
from tensorflow_datasets.core import dataset_builder
File "/usr/local/lib/python3.6/dist-packages/tensorflow_datasets/core/dataset_builder.py", line 44, in <module>
from tensorflow_datasets.core.utils import file_utils
ImportError: cannot import name 'file_utils'
Checking the content of /usr/local/lib/python3.6/dist-packages/tensorflow_datasets/core/utils/
I realized that there is no file_utils.py
. I tried to solve the issue by installing last version of tensorflow-datasets by doing pip install tensorflow-datasets
But it says that the requirement is already sastified:
---> Running in 2efd9d71d192
Requirement already satisfied: tensorflow-datasets in /usr/local/lib/python3.6/dist-packages (4.4.0)
Requirement already satisfied: termcolor in /usr/local/lib/python3.6/dist-packages (from tensorflow-datasets) (1.1.0)
Requirement already satisfied: dataclasses in /usr/local/lib/python3.6/dist-packages (from tensorflow-datasets) (0.8)
Requirement already satisfied: protobuf>=3.12.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow-datasets) (3.17.3)
Requirement already satisfied: promise in /usr/local/lib/python3.6/dist-packages (from tensorflow-datasets) (2.3)
Requirement already satisfied: requests>=2.19.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-datasets) (2.26.0)
Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from tensorflow-datasets) (1.15.0)
Requirement already satisfied: tensorflow-metadata in /usr/local/lib/python3.6/dist-packages (from tensorflow-datasets) (1.2.0)
Requirement already satisfied: importlib-resources in /usr/local/lib/python3.6/dist-packages (from tensorflow-datasets) (5.4.0)
Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from tensorflow-datasets) (0.18.2)
Requirement already satisfied: dill in /usr/local/lib/python3.6/dist-packages (from tensorflow-datasets) (0.3.4)
Requirement already satisfied: attrs>=18.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-datasets) (21.2.0)
Requirement already satisfied: absl-py in /usr/local/lib/python3.6/dist-packages (from tensorflow-datasets) (0.12.0)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.6/dist-packages (from tensorflow-datasets) (3.7.4.3)
Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from tensorflow-datasets) (4.62.3)
Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from tensorflow-datasets) (1.19.5)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests>=2.19.0->tensorflow-datasets) (2021.5.30)
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests>=2.19.0->tensorflow-datasets) (2.6)
Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.6/dist-packages (from requests>=2.19.0->tensorflow-datasets) (2.0.4)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests>=2.19.0->tensorflow-datasets) (1.26.6)
Requirement already satisfied: zipp>=3.1.0 in /usr/local/lib/python3.6/dist-packages (from importlib-resources->tensorflow-datasets) (3.5.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-metadata->tensorflow-datasets) (1.53.0)
Any ideas?
I think file_utils was added in tfds-nightly
and is not yet present in tensoflow_datasets
. You could try to pip install tfds-nightly
I was just proposing a workaround. Could you try to comment the following line:
We're using
shutil.disk_usage
to estimate the available space. This might conflict for some reason with docker. Commenting the line will ignore the error, so we can be sure this is the cause.
Any new update on this issue? I have similliar problem:
" OSError: Not enough disk space. Needed: 155.84 GiB (download: Unknown size, generated: 155.84 GiB."
Filesystem Size Used Avail Use% Mounted on
overlay 582G 45G 538G 8% /
tmpfs 64M 0 64M 0% /dev
tmpfs 30G 0 30G 0% /sys/fs/cgroup
shm 64M 0 64M 0% /dev/shm
/dev/xvda1 582G 45G 538G 8% /home
/dev/xvdf 400G 291G 110G 73% /home/tensorflow_datasets
Coming from Issue on how to train a Resnet50 using Imagenet from Scratch
What I need help with / What I was wondering I'm trying to train from scratch Resnet50 from TF Model garden using Imagenet. I need to prepara the dataset and I'm trying to use tfds (loaded from yaml config file, as suggested on previous opened issue).
I got an error that says "Not enough disk space", but I do have more than 200GB available. Any further suggestions?
Note that I need to execute everything from a docker container, because this container willl be used to test several infrastructures.
What I've tried so far Here you can see: 1) Imagenet data is downloaded, 2) I mount that volume in a docker container, 3) Inside that container I run train,py from model garden as was indicated in previous issue, a configuration is generated as shown. 4) I got the error: OSError: Not enough disk space. Needed: 155.84 GiB, 5) However,
df
shows that there is more than 200GB available.It would be nice if... Anyone has any suggestion about what I missed.
Environment information Here is the Dockerfile: