search

dvaldivia / hyper-scale-tensorflow-with-minio

Hyper-Scale Machine Learning with MinIO and TensorFlow
https://min.io
11 stars 6 forks source link

Epoch 1/10 failing with message "ValueError: Expect x to be a non-empty array or dataset." #6

Open amirkool opened 2 years ago

amirkool commented 2 years ago

I am trying to bring up a setup for Hyper-Scale Machine Learning with MinIO and TensorFlow following this blog And I am using your git-hub code form: https://github.com/dvaldivia/hyper-scale-tensorflow-with-minio.git (branch : dependabot/pip/tensorflow-2.5.3)

But it is failing with error message as below:

2022-03-09 15:59:44.768208: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session tear down.
Epoch 1/10
Traceback (most recent call last):
  File "MinIO TensorFlow.py", line 290, in <module>
    validation_data=mapped_validation,
  File "/root/PycharmProjects/hyper-scale-tensorflow-with-minio/venv/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 1189, in fit
    raise ValueError('Expect x to be a non-empty array or dataset.')
ValueError: Expect x to be a non-empty array or dataset.

Setup details:

OS: Ubuntu 18.04.6 LTS

Steps followed:

1- Created a python virtual environment with all the dependencies as mentioned 2- Reduced the data set to 500 samples/each folder to accelerate the issue reproduction

I am attaching the txt file having the full console output with errors as: error_run.txt and versions of packages used as: Package_version.txt

Request you to please provide any resolution or suggestion on how to proceed on this issue. I am new to ML and object storage. Any inputs/suggestions/help/resolution will highly assist me with this. Request you to please look into this. Thank you!

dvaldivia commented 2 years ago

Have you checked that the train data is properly loaded?