Open nanfangzhe opened 5 months ago
Can you please provide more information about your setup. Otherwise, it is not possible to resolve this problem.
Are you using the docker setup ? Which Tensorflow Version are you using ? Which command did you use for running train.py
I encountered the same issue. I used Docker to run the setup. Below are the environment configurations and commands used: Package Version
absl-py 1.0.0 astunparse 1.6.3 cachetools 5.1.0 certifi 2019.11.28 chardet 3.0.4 cloudpickle 3.0.0 dbus-python 1.2.16 decorator 5.1.1 dm-tree 0.1.8 easydict 1.13 flatbuffers 1.12 gast 0.4.0 google-auth 2.6.6 google-auth-oauthlib 0.4.6 google-pasta 0.2.0 grpcio 1.46.1 h5py 3.6.0 idna 2.8 importlib-metadata 4.11.3 keras 2.9.0 Keras-Preprocessing 1.1.2 libclang 14.0.1 Markdown 3.3.7 numpy 1.22.3 oauthlib 3.2.0 opt-einsum 3.3.0 packaging 21.3 pip 20.2.4 protobuf 3.20.1 pyasn1 0.4.8 pyasn1-modules 0.2.8 PyGObject 3.36.0 pyparsing 3.0.9 python-apt 2.0.0+ubuntu0.20.4.7 requests 2.22.0 requests-oauthlib 1.3.1 requests-unixsocket 0.2.0 rsa 4.8 scipy 1.10.1 setuptools 62.2.0 six 1.14.0 tensorboard 2.9.0 tensorboard-data-server 0.6.1 tensorboard-plugin-wit 1.8.1 tensorflow 2.9.0 tensorflow-compression 2.9.0 tensorflow-estimator 2.9.0 tensorflow-io-gcs-filesystem 0.25.0 tensorflow-probability 0.17.0 termcolor 1.1.0 typing-extensions 4.2.0 urllib3 1.25.8 Werkzeug 2.1.2 wheel 0.34.2 wrapt 1.14.1 zipp 3.8.0
python3 ./train.py \ --train_data_dir="demo_samples/training" \ --val_data_dir="demo_samples/validation" \ --train_output_dir="output" \ --model=additive_lstm_demo
Can you double check that the dataset loaded correctly and placed at the correct location? And double check that all pathes are correct. The error usually arises when the dataloader is empty and no training is conducted.
Insufficient training data may be the cause of this error
Traceback (most recent call last): File "train.py", line 104, in
train(args)
File "train.py", line 78, in train
model.fit(
File "/home/coplin/anaconda3/envs/tf/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/home/coplin/anaconda3/envs/tf/lib/python3.8/site-packages/keras/engine/training.py", line 1420, in fit
raise ValueError('Unexpected result of
train_function
' ValueError: Unexpected result oftrain_function
(Empty logs). Please useModel.compile(..., run_eagerly=True)
, ortf.config.run_functions_eagerly(True)
for more information of where went wrong, or file a issue/bug totf.keras
.