DB disconnected error - Githubissues

pyotr777 commented 9 years ago

I keep getting "DB is disconnected" error on every request to the registry. It was working fine before, but seems to broke after computer restart.

For example, v1/search request causes registry to print the following:

24/Dec/2014:17:00:12 +0900 WARNING: DB is disconnected. Reconnect to it.
2014-12-24 17:00:12,885 ERROR: Exception on /v1/search [GET]
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/Library/Python/2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Library/Python/2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Library/Python/2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Library/Python/2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Library/Python/2.7/site-packages/docker_registry/lib/mirroring.py", line 145, in wrapper
    resp = f(*args, **kwargs)
  File "/Library/Python/2.7/site-packages/docker_registry/search.py", line 21, in get_search
    results = INDEX.results(search_term=search_term)
  File "/Library/Python/2.7/site-packages/docker_registry/lib/index/db.py", line 64, in _retry
    raise e
OperationalError: (OperationalError) no such table: repository u'SELECT repository.id AS repository_id, repository.name AS repository_name, repository.description AS repository_description \nFROM repository' ()
24/Dec/2014:17:00:12 +0900 ERROR: Exception on /v1/search [GET]
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/Library/Python/2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Library/Python/2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Library/Python/2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Library/Python/2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Library/Python/2.7/site-packages/docker_registry/lib/mirroring.py", line 145, in wrapper
    resp = f(*args, **kwargs)
  File "/Library/Python/2.7/site-packages/docker_registry/search.py", line 21, in get_search
    results = INDEX.results(search_term=search_term)
  File "/Library/Python/2.7/site-packages/docker_registry/lib/index/db.py", line 64, in _retry
    raise e
OperationalError: (OperationalError) no such table: repository u'SELECT repository.id AS repository_id, repository.name AS repository_name, repository.description AS repository_description \nFROM repository' ()
172.19.7.74 - - [24/Dec/2014:17:00:12 +0900] "GET /v1/search HTTP/1.1" 500 291 "-" "curl/7.37.1"

I work on Mac OS with boot2docker.

dmp42 commented 9 years ago

Looks very much like one of the db issues mentioned earlier.

Try starting your registry with GUNICORN_OPTS=[--preload] (see https://github.com/docker/docker-registry#sqlalchemy)

pyotr777 commented 9 years ago

--preload doesn't help. docker-registry.db file is created by has 0 size.

dmp42 commented 9 years ago

You are on OSX, using system python, right? Does it work when you use brew python?

Did you pip install the registry using sudo? Do you then start the registry with sudo? What is the path of the database file?

pyotr777 commented 9 years ago

Thank you for your help!

Yes, I work on OSX.

Installing registry with sudo pip install . and starting with sudo doesn't help. Database file was under my home directory. Changing it to default /tmp didn't help either.

I also have two docker-registry images in boot2docker. One of them DB error when registry starts in docker container. The image I created before works fine, so I tried copying config.yml file into a docker container with broken registry and changing configuration to default driver. Neither of this helped.

Before restarting computer I was installing Symantec Endpoint Protection, so I wonder if it can be related to the errors? After restart I had to repair disk permissions with Mac OS Disk Utility. Some errors has been corrected, but registry still doesn't work.

I didn't tried installing python with brew yet. But since the same error occurs inside containers, python version should not matter?

Does installing python with brew involves configuration or it works right after install command?

pyotr777 commented 9 years ago

Manually creating a table with the same db file on OS X works:

~/work/docker-registry > python
Python 2.7.6 (default, Sep  9 2014, 15:04:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlalchemy
>>> engine=sqlalchemy.create_engine('sqlite:////tmp/docker-registry.db')
>>> con=engine.connect()
>>> result = con.execute("CREATE TABLE p (id INTEGER PRIMARY KEY ASC, name varchar(250) NOT NULL)")
>>> inspector=sqlalchemy.inspect(engine)
>>> print inspector.get_table_names()
[u'p']

Checkouting registry from git repository to an earlier version that did work before doesn't help.

dmp42 commented 9 years ago

I would recommend that you:

rm the database file
start from scratch, launching your registry with DEBUG=true and GUNICORN_OPTS as indicated
copy the output of the first launch

Alternatively, you should probably use homebrew python instead of the OSX system python.

pyotr777 commented 9 years ago

Happy New Year!

What do you mean by "from scratch"? To remove local registry folder and copy again? Or go through installation process including installing python libraries etc? Do I need to uninstall all packages first?

I tried to install registry in a docker container with Ubuntu. I mounted two folders: with docker registry and with my driver into container. After running the following commands registry seem to work:

1  apt-get update
2  apt-get install python-dev liblzma-dev libevent1-dev
3  pip install /docker-registry/depends/docker-registry-core
4  pip install file:///docker-registry#egg=docker-registry[bugsnag,newrelic,cors]
5  export DOCKER_REGISTRY_CONFIG=/docker-registry/config/config.yml
6  export SETTINGS_FLAVOR=dev
7  cd /docker-registry-driver-git/
8  pip install .

7 and 8 are for installing my driver.

I couldn't install python on OS X with brew. I get a checksum error:

brew install python
==> Downloading http://www.python.org/ftp/python/2.7.7/Python-2.7.7.tgz
######################################################################## 100.0%
==> ./configure --prefix=/usr/local/Cellar/python/2.7.7_1 --enable-ipv6 --datarootdir=/usr/local/Cellar/python/2.7.7_1/share --datadir=/
==> make
==> make install PYTHONAPPSDIR=/usr/local/Cellar/python/2.7.7_1
==> make frameworkinstallextras PYTHONAPPSDIR=/usr/local/Cellar/python/2.7.7_1/share/python
==> Downloading https://pypi.python.org/packages/source/s/setuptools/setuptools-4.0.1.tar.gz
######################################################################## 100.0%
Error: SHA1 mismatch
Expected: a43549f4a01f314bf54567628f8de7d1c03d5930
Actual: 6d417376509eee44c1da34692fb5d805fd2915c6
Archive: /Library/Caches/Homebrew/python--setuptools-4.0.1.tar.gz
To retry an incomplete download, remove the file above.

I even removed all files in /Library/Caches/Homebrew, but it didn't help.

dmp42 commented 9 years ago

Python 2.7.7 is an old version. You should see python 2.7.9 with brew.

dmp42 commented 9 years ago

Any update?

pyotr777 commented 9 years ago

Thank you for your attention!

Updated brew and installed python 2.7.9 on OS X.

Reinstalled registry and backend driver with sudo python setup.py develop.

By the way, as for the difference between python setup.py install and python setup.py develop. I suppose, the latter creates a link to the current directory (with source code), whereas the former copies an egg or the whole directory with the source code to a python library folder. Am I correct?

The problem with a database persists:

$ sudo ./registry-start

2015-01-08 11:44:45 [73530] [INFO] Starting gunicorn 19.1.0
2015-01-08 11:44:45 [73530] [INFO] Listening at: http://0.0.0.0:5000 (73530)
2015-01-08 11:44:45 [73530] [INFO] Using worker: gevent
2015-01-08 11:44:45 [73533] [INFO] Booting worker with pid: 73533
2015-01-08 11:44:45 [73530] [INFO] 1 workers
08/Jan/2015:11:44:45 +0900 WARNING: Cache storage disabled!
08/Jan/2015:11:44:45 +0900 WARNING: LRU cache disabled!
08/Jan/2015:11:44:45 +0900 DEBUG: Will return docker-registry.drivers.gitdriver.Storage
08/Jan/2015:11:44:45 +0900 INFO: Git backend driver 0.7.102 initialisation
08/Jan/2015:11:44:48 +0900 WARNING: DB is disconnected. Reconnect to it.
2015-01-08 11:44:48,061 ERROR: Exception on /v1/search [GET]
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/Library/Python/2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Library/Python/2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Library/Python/2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Library/Python/2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Library/Python/2.7/site-packages/docker_registry-0.9.0-py2.7.egg/docker_registry/lib/mirroring.py", line 145, in wrapper
    resp = f(*args, **kwargs)
  File "/Library/Python/2.7/site-packages/docker_registry-0.9.0-py2.7.egg/docker_registry/search.py", line 21, in get_search
    results = INDEX.results(search_term=search_term)
  File "/Library/Python/2.7/site-packages/docker_registry-0.9.0-py2.7.egg/docker_registry/lib/index/db.py", line 64, in _retry
    raise e
OperationalError: (OperationalError) no such table: repository u'SELECT repository.id AS repository_id, repository.name AS repository_name, repository.description AS repository_description \nFROM repository' ()
08/Jan/2015:11:44:48 +0900 ERROR: Exception on /v1/search [GET]
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/Library/Python/2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Library/Python/2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Library/Python/2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Library/Python/2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Library/Python/2.7/site-packages/docker_registry-0.9.0-py2.7.egg/docker_registry/lib/mirroring.py", line 145, in wrapper
    resp = f(*args, **kwargs)
  File "/Library/Python/2.7/site-packages/docker_registry-0.9.0-py2.7.egg/docker_registry/search.py", line 21, in get_search
    results = INDEX.results(search_term=search_term)
  File "/Library/Python/2.7/site-packages/docker_registry-0.9.0-py2.7.egg/docker_registry/lib/index/db.py", line 64, in _retry
    raise e
OperationalError: (OperationalError) no such table: repository u'SELECT repository.id AS repository_id, repository.name AS repository_name, repository.description AS repository_description \nFROM repository' ()
127.0.0.1 - - [08/Jan/2015:11:44:48 +0900] "GET /v1/search HTTP/1.1" 500 291 "-" "curl/7.37.1"
2015-01-08 11:45:15 [73530] [INFO] 1 workers

in registry-start I have:

gunicorn --access-logfile - --log-file - --debug -k gevent -b 0.0.0.0:5000 -w 1 docker_registry.wsgi:application

I also noticed that there is no file.py anymore in drivers directory of the current docker-registry on github. Where did it go?

And in s3.py I see:

from docker_registry.core import compat

but there is no core directory anymore. How does this import works?

dmp42 commented 9 years ago

[install vs. develop]

Correct.

About the rest: if you are using python brew, then you don't need to sudo. You should be able to do everything with your regular user account.

Now, looking at your logs, you are still using the system python it seems. You probably need to adjust your path.

pyotr777 commented 9 years ago

I removed old modules in /Library/Python/2.7 and reinstalled them with pip install.

Still having errors:

[2015-01-09 11:05:25 +0900] [85319] [INFO] Starting gunicorn 19.1.1
[2015-01-09 11:05:25 +0900] [85319] [INFO] Listening at: http://0.0.0.0:5000 (85319)
[2015-01-09 11:05:25 +0900] [85319] [INFO] Using worker: gevent
[2015-01-09 11:05:25 +0900] [85322] [INFO] Booting worker with pid: 85322
09/Jan/2015:11:05:25 +0900 WARNING: Cache storage disabled!
09/Jan/2015:11:05:25 +0900 WARNING: LRU cache disabled!
09/Jan/2015:11:05:25 +0900 DEBUG: Will return docker-registry.drivers.gitdriver.Storage
09/Jan/2015:11:05:25 +0900 INFO: Git backend driver 0.7.102 initialisation
09/Jan/2015:11:05:29 +0900 WARNING: DB is disconnected. Reconnect to it.
2015-01-09 11:05:29,242 ERROR: Exception on /v1/search [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/peterbryzgalov/work/docker-registry/docker_registry/lib/mirroring.py", line 145, in wrapper
    resp = f(*args, **kwargs)
  File "/Users/peterbryzgalov/work/docker-registry/docker_registry/search.py", line 21, in get_search
    results = INDEX.results(search_term=search_term)
  File "/Users/peterbryzgalov/work/docker-registry/docker_registry/lib/index/db.py", line 64, in _retry
    raise e
OperationalError: (OperationalError) no such table: repository u'SELECT repository.id AS repository_id, repository.name AS repository_name, repository.description AS repository_description \nFROM repository' ()

Python version:

~/work/docker-registry > python
Python 2.7.9 (default, Jan  3 2015, 14:21:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.54)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

dmp42 commented 9 years ago

One last check now - are you running the latest master?

pyotr777 commented 9 years ago

I'm running a forked version: https://github.com/pyotr777/docker-registry Should I abandon my fork and use master instead?

pyotr777 commented 9 years ago

Cloned docker-registry from https://github.com/docker/docker-registry, but cannot install it.

Running python setup.py develop or pip install . fails:

~/work/docker-registry-master > python setup.py develop
running develop
running egg_info
writing requirements to docker_registry.egg-info/requires.txt
writing docker_registry.egg-info/PKG-INFO
writing namespace_packages to docker_registry.egg-info/namespace_packages.txt
writing top-level names to docker_registry.egg-info/top_level.txt
writing dependency_links to docker_registry.egg-info/dependency_links.txt
writing entry points to docker_registry.egg-info/entry_points.txt
reading manifest file 'docker_registry.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'docker_registry.egg-info/SOURCES.txt'
running build_ext
Creating /usr/local/lib/python2.7/site-packages/docker-registry.egg-link (link to .)
docker-registry 1.0.0-dev is already the active version in easy-install.pth
Installing docker-registry script to /usr/local/bin

Installed /Users/peterbryzgalov/work/docker-registry-master
Processing dependencies for docker-registry==1.0.0-dev
Searching for M2Crypto==0.22.3
Reading https://pypi.python.org/simple/M2Crypto/
Best match: M2Crypto 0.22.3
Downloading https://pypi.python.org/packages/source/M/M2Crypto/M2Crypto-0.22.3.tar.gz#md5=573f21aaac7d5c9549798e72ffcefedd
Processing M2Crypto-0.22.3.tar.gz
Writing /var/folders/ym/nhg3k9pd693gbj_x1vpm0ml80000gn/T/easy_install-h5F6e2/M2Crypto-0.22.3/setup.cfg
Running M2Crypto-0.22.3/setup.py -q bdist_egg --dist-dir /var/folders/ym/nhg3k9pd693gbj_x1vpm0ml80000gn/T/easy_install-h5F6e2/M2Crypto-0.22.3/egg-dist-tmp-8tK9j9
SWIG/_m2crypto.i:30: Error: Unable to find 'openssl/opensslv.h'
SWIG/_m2crypto.i:33: Error: Unable to find 'openssl/safestack.h'
SWIG/_evp.i:12: Error: Unable to find 'openssl/opensslconf.h'
SWIG/_ec.i:7: Error: Unable to find 'openssl/opensslconf.h'
error: Setup script exited with error: command 'swig' failed with exit status 1

pyotr777 commented 9 years ago

I found similar issue here: https://github.com/Homebrew/homebrew/issues/33552

Running xcode-select --install helped. I could install registry with:

~/work/docker-registry-master > python setup.py develop

I had some warnings:

SWIG/_bio.i:64: Warning 454: Setting a pointer/reference variable may leak memory.
SWIG/_rand.i:21: Warning 454: Setting a pointer/reference variable may leak memory.
SWIG/_evp.i:160: Warning 454: Setting a pointer/reference variable may leak memory.
SWIG/_dh.i:36: Warning 454: Setting a pointer/reference variable may leak memory.
SWIG/_rsa.i:43: Warning 454: Setting a pointer/reference variable may leak memory.
SWIG/_dsa.i:31: Warning 454: Setting a pointer/reference variable may leak memory.
SWIG/_ssl.i:212: Warning 454: Setting a pointer/reference variable may leak memory.
SWIG/_x509.i:323: Warning 454: Setting a pointer/reference variable may leak memory.
SWIG/_pkcs7.i:44: Warning 454: Setting a pointer/reference variable may leak memory.
SWIG/_pkcs7.i:44: Warning 454: Setting a pointer/reference variable may leak memory.
SWIG/_util.i:11: Warning 454: Setting a pointer/reference variable may leak memory.
SWIG/_ec.i:111: Warning 454: Setting a pointer/reference variable may leak memory.
SWIG/_engine.i:168: Warning 454: Setting a pointer/reference variable may leak memory.

but registry got installed and is working now!

~/work/docker-registry-master > ./registry-start
[2015-01-09 15:31:44 +0900] [93448] [INFO] Starting gunicorn 19.1.1
[2015-01-09 15:31:44 +0900] [93448] [INFO] Listening at: http://0.0.0.0:5000 (93448)
[2015-01-09 15:31:44 +0900] [93448] [INFO] Using worker: gevent
[2015-01-09 15:31:44 +0900] [93451] [INFO] Booting worker with pid: 93451
09/Jan/2015:15:31:44 +0900 WARNING: Cache storage disabled!
09/Jan/2015:15:31:44 +0900 WARNING: LRU cache disabled!
09/Jan/2015:15:31:44 +0900 DEBUG: Will return docker-registry.drivers.gitdriver.Storage
09/Jan/2015:15:31:44 +0900 INFO: Git backend driver 0.7.102 initialisation
gitcom: <git.cmd.Git object at 0x109128730>
List repositories
127.0.0.1 - - [09/Jan/2015:15:31:57 +0900] "GET /v1/search HTTP/1.1" 200 46 "-" "curl/7.37.1"

I shall test my backend driver with this registry version. I have questions about nose tests, and shall post them later.

I appreciate your help!

dmp42 commented 9 years ago

Happy we got this working! Keep me posted on your progress.

docker-archive / docker-registry

DB disconnected error #870