docker image seems not to build anymore with new rocksdb version

[JJ-Author]

Step 11/19 : RUN pip install Cython pipenv                                                                                                                                                                  
 ---> Using cache                                                                                                                                                                                           
 ---> a4ca1760051d                                                                                                                                                                                          
Step 12/19 : RUN pipenv install --system                                                                                                                                                                    
 ---> Running in 547cb49dfebb                                                                                                                                                                               
Installing dependencies from Pipfile.lock (6da76c)…                                                                                                                                                         
An error occurred while installing python-rocksdb==0.7.0 --hash=sha256:aa692fbd06d8ed4ad641feee2970062fe01120e87b645a29403c71c628639e4f! Will try again.                                                    
Installing initially failed dependencies…                                                                                                                                                                   
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 1874, in do_install                                                                                  
[pipenv.exceptions.InstallError]:       keep_outdated=keep_outdated                                                                                                                                         
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 1253, in do_init                                                                                     
[pipenv.exceptions.InstallError]:       pypi_mirror=pypi_mirror,                                                                                                                                            
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 859, in do_install_dependencies                                                                      
[pipenv.exceptions.InstallError]:       retry_list, procs, failed_deps_queue, requirements_dir, **install_kwargs                                                                                            
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 763, in batch_install                                                                                
[pipenv.exceptions.InstallError]:       _cleanup_procs(procs, not blocking, failed_deps_queue, retry=retry)                                                                                                 
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 681, in _cleanup_procs                                                                               
[pipenv.exceptions.InstallError]:       raise exceptions.InstallError(c.dep.name, extra=err_lines)                                                                                                          
[pipenv.exceptions.InstallError]: ['Collecting python-rocksdb==0.7.0 (from -r /tmp/pipenv-urunsih4-requirements/pipenv-g3np_3k2-requirement.txt (line 1))', '  Downloading https://files.pythonhosted.org/pa
ckages/94/81/9a8e498a26ec34da9e1ae87e831468aedb4a6ba42aa9ea779b97896cfea7/python-rocksdb-0.7.0.tar.gz (219kB)', 'Installing collected packages: python-rocksdb', '  Running setup.py install for python-rock
sdb: started', "    Running setup.py install for python-rocksdb: finished with status 'error'", '    Complete output from command /usr/local/bin/python -u -c "import setuptools, tokenize;__file__=\'/tmp/p
ip-install-gxt_88ki/python-rocksdb/setup.py\';f=getattr(tokenize, \'open\', open)(__file__);code=f.read().replace(\'\\r\\n\', \'\\n\');f.close();exec(compile(code, __file__, \'exec\'))" install --record /
tmp/pip-record-4ocrkrwh/install-record.txt --single-version-externally-managed --compile:', '    running install', '    running build', '    running build_py', '    creating build', '    creating build/li
b.linux-x86_64-3.6', '    creating build/lib.linux-x86_64-3.6/rocksdb', '    copying rocksdb/errors.py -> build/lib.linux-x86_64-3.6/rocksdb', '    copying rocksdb/interfaces.py -> build/lib.linux-x86_64-
3.6/rocksdb', '    copying rocksdb/merge_operators.py -> build/lib.linux-x86_64-3.6/rocksdb', '    copying rocksdb/__init__.py -> build/lib.linux-x86_64-3.6/rocksdb', '    creating build/lib.linux-x86_64-
3.6/rocksdb/tests', '    copying rocksdb/tests/test_db.py -> build/lib.linux-x86_64-3.6/rocksdb/tests', '    copying rocksdb/tests/test_options.py -> build/lib.linux-x86_64-3.6/rocksdb/tests', '    copyin
g rocksdb/tests/__init__.py -> build/lib.linux-x86_64-3.6/rocksdb/tests', '    copying rocksdb/tests/test_memtable.py -> build/lib.linux-x86_64-3.6/rocksdb/tests', '    running egg_info', '    writing pyt
hon_rocksdb.egg-info/PKG-INFO', '    writing dependency_links to python_rocksdb.egg-info/dependency_links.txt', '    writing requirements to python_rocksdb.egg-info/requires.txt', '    writing top-level n
ames to python_rocksdb.egg-info/top_level.txt', "    reading manifest file 'python_rocksdb.egg-info/SOURCES.txt'", "    reading manifest template 'MANIFEST.in'", "    writing manifest file 'python_rocksdb
.egg-info/SOURCES.txt'", '    copying rocksdb/_rocksdb.cpp -> build/lib.linux-x86_64-3.6/rocksdb', '    copying rocksdb/_rocksdb.pyx -> build/lib.linux-x86_64-3.6/rocksdb', '    copying rocksdb/backup.pxd
 -> build/lib.linux-x86_64-3.6/rocksdb', '    copying rocksdb/cache.pxd -> build/lib.linux-x86_64-3.6/rocksdb', '    copying rocksdb/comparator.pxd -> build/lib.linux-x86_64-3.6/rocksdb', '    copying roc
ksdb/db.pxd -> build/lib.linux-x86_64-3.6/rocksdb', '    copying rocksdb/env.pxd -> build/lib.linux-x86_64-3.6/rocksdb', '    copying rocksdb/filter_policy.pxd -> build/lib.linux-x86_64-3.6/rocksdb', '   
 copying rocksdb/iterator.pxd -> build/lib.linux-x86_64-3.6/rocksdb', '    copying rocksdb/logger.pxd -> build/lib.linux-x86_64-3.6/rocksdb', '    copying rocksdb/memtablerep.pxd -> build/lib.linux-x86_64
-3.6/rocksdb', '    copying rocksdb/merge_operator.pxd -> build/lib.linux-x86_64-3.6/rocksdb', '    copying rocksdb/options.pxd -> build/lib.linux-x86_64-3.6/rocksdb', '    copying rocksdb/slice_.pxd -> b
uild/lib.linux-x86_64-3.6/rocksdb', '    copying rocksdb/slice_transform.pxd -> build/lib.linux-x86_64-3.6/rocksdb', '    copying rocksdb/snapshot.pxd -> build/lib.linux-x86_64-3.6/rocksdb', '    copying 
rocksdb/status.pxd -> build/lib.linux-x86_64-3.6/rocksdb', '    copying rocksdb/std_memory.pxd -> build/lib.linux-x86_64-3.6/rocksdb', '    copying rocksdb/table_factory.pxd -> build/lib.linux-x86_64-3.6/
rocksdb', '    copying rocksdb/universal_compaction.pxd -> build/lib.linux-x86_64-3.6/rocksdb', '    creating build/lib.linux-x86_64-3.6/rocksdb/cpp', '    copying rocksdb/cpp/comparator_wrapper.hpp -> bu
ild/lib.linux-x86_64-3.6/rocksdb/cpp', '    copying rocksdb/cpp/filter_policy_wrapper.hpp -> build/lib.linux-x86_64-3.6/rocksdb/cpp', '    copying rocksdb/cpp/memtable_factories.hpp -> build/lib.linux-x86
_64-3.6/rocksdb/cpp', '    copying rocksdb/cpp/merge_operator_wrapper.hpp -> build/lib.linux-x86_64-3.6/rocksdb/cpp', '    copying rocksdb/cpp/slice_transform_wrapper.hpp -> build/lib.linux-x86_64-3.6/roc
ksdb/cpp', '    copying rocksdb/cpp/utils.hpp -> build/lib.linux-x86_64-3.6/rocksdb/cpp', '    copying rocksdb/cpp/write_batch_iter_helper.hpp -> build/lib.linux-x86_64-3.6/rocksdb/cpp', '    warning: bui
ld_py: byte-compiling is disabled, skipping.', '    ', '    running build_ext', '    cythoning rocksdb/_rocksdb.pyx to rocksdb/_rocksdb.cpp', "    /usr/local/lib/python3.6/site-packages/Cython/Compiler/Ma
in.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-install-gxt_88ki/python-rocksdb/rocksdb/_rocksdb.pyx", '    
  tree = Parsing.p_module(s, pxd, full_module_name)', "    building 'rocksdb._rocksdb' extension", '    creating build/temp.linux-x86_64-3.6', '    creating build/temp.linux-x86_64-3.6/rocksdb', '    gcc 
-pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include/python3.6m -c rocksdb/_rocksdb.cpp -o build/temp.linux-x86_64-3.6/rocksdb/_rocksdb.o -std=c++11 -O3 -Wal
l -Wextra -Wconversion -fno-strict-aliasing -fno-rtti', '    In file included from rocksdb/_rocksdb.cpp:627:0:', '    /usr/include/rocksdb/filter_policy.h:27:25: fatal error: db/dbformat.h: No such file o
r directory', '     #include "db/dbformat.h"', '                             ^', '    compilation terminated.', "    error: command 'gcc' failed with exit status 1", '    ', '    -------------------------
---------------']
[pipenv.exceptions.InstallError]: ['Command "/usr/local/bin/python -u -c "import setuptools, tokenize;__file__=\'/tmp/pip-install-gxt_88ki/python-rocksdb/setup.py\';f=getattr(tokenize, \'open\', open)(__f
ile__);code=f.read().replace(\'\\r\\n\', \'\\n\');f.close();exec(compile(code, __file__, \'exec\'))" install --record /tmp/pip-record-4ocrkrwh/install-record.txt --single-version-externally-managed --comp
ile" failed with error code 1 in /tmp/pip-install-gxt_88ki/python-rocksdb/']
ERROR: ERROR: Package installation failed...

[aolieman]

I have not been able to reproduce this build issue locally, nor on Docker Hub. I'm willing to bet that you won't be able to reproduce this issue anymore either.

You attempted to build when rocksdb's master branch was in an unlucky state. See this commit that fixed the issue: https://github.com/facebook/rocksdb/commit/5265c5709eb19f49c047345b0046d8c5f09600ad

We can leave this issue open for now, because it might happen again. It could possibly be prevented in the future by explicitly installing rocksdb's latest release. There are several ways of doing this and I need to pick one.

[JJ-Author]

OK. I confirm it works again. I think we can close the issue. So far no need to deal with bad commits on a master branch. However, I don't know how fast python-rocksdb is updated so maybe it makes sense to checkout the latest release tag for rocksdb only to make it more stable?

[aolieman]

I thought it would be nice to get rid of git altogether. GitHub's GraphQL API looks pretty neat:

https://developer.github.com/v4/explorer/

{
  repository(owner: "facebook", name: "rocksdb") {
    releases(last: 1) {
      nodes {
        name,
        tagName,
        url
      }
    }
  }
}

Unfortunately, it requires authentication and that's annoying in a Dockerfile. I'm now testing an alternative download method, based on an available 302 redirect.

[aolieman]

I created a generalized script for downloading the latest release from a GitHub repo. It might interest you for your own Docker endeavors, if you want to stay away from git, that is.

Provided as a gist (well, sort of): https://github.com/stamkracht/gists/blob/master/github/download_latest_release.py