fastmonkeys / stellar

Fast database snapshot and restore tool for development
MIT License
3.86k stars 119 forks source link

Not working properly with docker #62

Closed adis-io closed 7 years ago

adis-io commented 7 years ago

I used this docker image: https://hub.docker.com/r/k0st/alpine-postgres/

After I decided to update to 9.5 and it works fine.

Before this I got something "Database not exists" if I try to restore snapshot.

This issue just for information for new users. But I will add failing test case a little bit later (may be in a week or so) so somebody also can reproduce issue.

Update: I found out the problem: https://github.com/fastmonkeys/stellar/issues/62#issuecomment-315166308

Teemu commented 7 years ago

I am quite sure it works with PostgreSQL 9.4 as this project is 2-3 years old and I have used every PostgreSQL database version ever since.

adis-io commented 7 years ago

Also got this error today:

Background process missing, doing slow restore.
Traceback (most recent call last):
  File "/usr/local/bin/stellar", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/stellar/command.py", line 279, in main
    stellar()
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/stellar/command.py", line 126, in restore
    app.inline_slave_copy(snapshot)
  File "/usr/local/lib/python2.7/site-packages/stellar/app.py", line 198, in inline_slave_copy
    table.get_table_name('slave')
  File "/usr/local/lib/python2.7/site-packages/stellar/operations.py", line 65, in copy_database
    from_database
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 939, in execute
    return self._execute_text(object, multiparams, params)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1097, in _execute_text
    statement, parameters
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
    context)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1402, in _handle_dbapi_exception
    exc_info
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
    context)
  File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) database "stellar_14d11213cac10837" already exists
 [SQL: '\n                CREATE DATABASE "stellar_14d11213cac10837" WITH TEMPLATE "stellar_438b5eef4ecaf531";\n
adis-io commented 7 years ago

Turns out stellar is not working properly with docker. All due to the fact that docker container exits when its main process finishes. But here we see that stellar forks main process. That's why it was working not properly. I have two choices: 1) fork this repo and remove that forking process part or maybe make this configurable. 2) use sh wrapper that checks that all stellar instances finished

I chose second option and my wrapper script looks like this:

#!/bin/sh

stellar "$@"

n=0
while true
do
  ps aux | grep -v grep | grep -q {stellar} || break
  n=$((n+1))
  if [ $n -gt 20 ]; then
    exit 1;
  fi
  sleep 1
done

My docker configs: https://gist.github.com/adisos/c6a3daf47abf76f465a7a21b93b2562f

quantus commented 7 years ago

Thank you @adisos for your debugging effort and explanation of the issue!

For such use cases it would make sense to have some kind of --no-forks switch that would disable creation fo background processes. I've added it to my personal todo lists, but it will take some time before I have time to do it.