databricks / spark-deep-learning

Deep Learning Pipelines for Apache Spark

https://databricks.github.io/spark-deep-learning

Apache License 2.0

1.99k stars 494 forks source link

Support and build against Keras 2.2.2 and TF 1.10.0 #151

Closed lu-wang-dl closed 6 years ago

lu-wang-dl commented 6 years ago

bump spark version to 2.3.1
bump tensorframes version to 0.4.0
bump keras==2.2.2 and tensorflow==1.10.0 to fix travis issues
TF_C_API_GRAPH_CONSTRUCTION added as a temp fix
Drop support for Spark <2.3 and hence Scala 2.10
add python3 friendly print
add pooling='avg' in resnet50 testing model beccause keras api changed
test arrays almost equal with whatever precision 5 in NamedImageTransformerBaseTestCase, test_bare_keras_module, keras_load_and_preproc
make keras model smaller in test_simple_keras_udf

This is a continued work from https://github.com/databricks/spark-deep-learning/pull/149.

codecov-io commented 6 years ago

Codecov Report

Merging #151 into master will not change coverage. The diff coverage is 100%.

@@           Coverage Diff           @@
##           master     #151   +/-   ##
=======================================
  Coverage   85.37%   85.37%           
=======================================
  Files          34       34           
  Lines        1922     1922           
  Branches       41       41           
=======================================
  Hits         1641     1641           
  Misses        281      281

Impacted Files	Coverage Δ
python/sparkdl/transformers/keras_applications.py	`85.71% <100%> (ø)`	:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 6be7772...ef39c1e. Read the comment docs.

lu-wang-dl commented 6 years ago

retest

yogeshg commented 6 years ago

Thanks for the update, Lu! As a summary of my comments:

[x] can we put the testingPrecision in SparkDLTestCase ?
[x] which test case fails with the standard setting of resnet50? and follow up questions inline.
[x] do you think we need to make matching changes in model and _testKerasModel methods of resnet50?
[x] assertArrayAlmostEqual to be introduced in a follow up pr
[x] tensorframes release to follow or precede this release

yogeshg commented 6 years ago

LGTM pending tests. @jkbradley , LGTY?

jkbradley commented 6 years ago

LGTM too, thanks!

jkbradley commented 6 years ago

I'll merge this now. Thanks @ludatabricks and @yogeshg !