search

intel-analytics / analytics-zoo

Distributed Tensorflow, Keras and PyTorch on Apache Spark/Flink & Ray
https://analytics-zoo.readthedocs.io/
Apache License 2.0
16 stars 3 forks source link

Error "partitionNum should be larger than 0, but get 0" when running imageclassification example #1031

Closed nwang2 closed 5 years ago

nwang2 commented 5 years ago

We see below error message when running image classification example: 2019-07-15 09:34:04 INFO Engine$:402 - Find existing spark context. Checking the spark conf... Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: Utils.getBatchSize: partitionNum should be larger than 0, but get 0

The script is as below:

spark-submit \
     --master local[44] \
     --num-executors 1 \
     --executor-cores 44 \
     --driver-memory 60g \
     --executor-memory 30g \
     --verbose \
     --driver-class-path /root/analytics-zoo-jdk8/dist/lib/analytics-zoo-bigdl_0.8.0-spark_2.1.0-0.6.0-SNAPSHOT-jar-with-dependencies.jar \
     --class com.intel.analytics.zoo.examples.imageclassification.Predict \
     /root/analytics-zoo-jdk8/dist/lib/analytics-zoo-bigdl_0.8.0-spark_2.1.0-0.6.0-SNAPSHOT-jar-with-dependencies.jar \
     --model /root/analytics-zoo_resnet-50_imagenet_0.1.0.model \
     -f file:///root/val/ --partition 44 --topN 5
qiuxin2012 commented 5 years ago

Looks like your data is empty...

nwang2 commented 5 years ago

ah... I see. The images are in sub folders in the path. If I use -f file:///root/val/n01855672, it works. The error message is somehow misleading.

qiuxin2012 commented 5 years ago

@nwang2 we will throw a better exception. Close this issue.