We were using _list_files_with_extension to find the assembly jar in spark-deep-learning, and simply taking the last jar in the directory where we expect the assembly file to be. During build and release this directory can also have sources*.jar which can come after assembly*.jar.
I switched to using glob so we could check not only the extension but also part of the file name. Since _list_files_with_extension seems to be strictly a subset of glob, went ahead and replaced all usages of it.
There was another issue in our doc build as well, the indentation of index.rst was causing some of the doc files to be missed. These files were being left out of the doc build with this warning:
/mnt/sparkdl/python/docs/index.rst:15: WARNING: toctree contains reference to nonexisting document u' sparkdl.graph'
/mnt/sparkdl/python/docs/index.rst:15: WARNING: toctree contains reference to nonexisting document u' sparkdl.image'
/mnt/sparkdl/python/docs/index.rst:15: WARNING: toctree contains reference to nonexisting document u' sparkdl.transformers'
/mnt/sparkdl/python/docs/index.rst:15: WARNING: toctree contains reference to nonexisting document u' sparkdl.udf'
/mnt/sparkdl/python/docs/index.rst:15: WARNING: toctree contains reference to nonexisting document u' sparkdl.utils'
checking consistency... /mnt/sparkdl/python/docs/sparkdl.graph.rst: WARNING: document isn't included in any toctree
/mnt/sparkdl/python/docs/sparkdl.image.rst: WARNING: document isn't included in any toctree
/mnt/sparkdl/python/docs/sparkdl.transformers.rst: WARNING: document isn't included in any toctree
/mnt/sparkdl/python/docs/sparkdl.udf.rst: WARNING: document isn't included in any toctree
/mnt/sparkdl/python/docs/sparkdl.utils.rst: WARNING: document isn't included in any toctree
We were using
_list_files_with_extension
to find the assembly jar in spark-deep-learning, and simply taking the last jar in the directory where we expect the assembly file to be. During build and release this directory can also havesources*.jar
which can come afterassembly*.jar
.I switched to using glob so we could check not only the extension but also part of the file name. Since
_list_files_with_extension
seems to be strictly a subset of glob, went ahead and replaced all usages of it.There was another issue in our doc build as well, the indentation of
index.rst
was causing some of the doc files to be missed. These files were being left out of the doc build with this warning: