databricks / spark-deep-learning

Deep Learning Pipelines for Apache Spark
https://databricks.github.io/spark-deep-learning
Apache License 2.0
1.99k stars 494 forks source link

[ML-5800] Fix warnings in doc build. #177

Closed MrBago closed 5 years ago

MrBago commented 5 years ago

We were using _list_files_with_extension to find the assembly jar in spark-deep-learning, and simply taking the last jar in the directory where we expect the assembly file to be. During build and release this directory can also have sources*.jar which can come after assembly*.jar.

I switched to using glob so we could check not only the extension but also part of the file name. Since _list_files_with_extension seems to be strictly a subset of glob, went ahead and replaced all usages of it.

There was another issue in our doc build as well, the indentation of index.rst was causing some of the doc files to be missed. These files were being left out of the doc build with this warning:

/mnt/sparkdl/python/docs/index.rst:15: WARNING: toctree contains reference to nonexisting document u' sparkdl.graph'
/mnt/sparkdl/python/docs/index.rst:15: WARNING: toctree contains reference to nonexisting document u' sparkdl.image'
/mnt/sparkdl/python/docs/index.rst:15: WARNING: toctree contains reference to nonexisting document u' sparkdl.transformers'
/mnt/sparkdl/python/docs/index.rst:15: WARNING: toctree contains reference to nonexisting document u' sparkdl.udf'
/mnt/sparkdl/python/docs/index.rst:15: WARNING: toctree contains reference to nonexisting document u' sparkdl.utils'

checking consistency... /mnt/sparkdl/python/docs/sparkdl.graph.rst: WARNING: document isn't included in any toctree
/mnt/sparkdl/python/docs/sparkdl.image.rst: WARNING: document isn't included in any toctree
/mnt/sparkdl/python/docs/sparkdl.transformers.rst: WARNING: document isn't included in any toctree
/mnt/sparkdl/python/docs/sparkdl.udf.rst: WARNING: document isn't included in any toctree
/mnt/sparkdl/python/docs/sparkdl.utils.rst: WARNING: document isn't included in any toctree