Closed rstuckey closed 9 years ago
I've also tried several times to get this working on my Mac and yet to succeed because of problems with Docker. I managed to use boot2docker and docker-compose but am stuck on another issue...I'll post it when I get home. It's good to know others have been having trouble too.
On Mon, Jun 22, 2015 at 9:30 AM, Roger Stuckey notifications@github.com wrote:
I had many issues getting the Docker containers going on my Mac, so I thought I'd share some workarounds here. I'm fairly new to Docker, so there may be mistakes! There are a couple of project-related issues at the end too :-)
First, install Docker, boot2docker https://github.com/boot2docker/osx-installer/releases and Docker-compose http://docs.docker.com/compose/.
There are problems https://github.com/boot2docker/boot2docker/issues/581 with the permissions on vboxsf mounts, preventing postgres from writing to /var/lib/postgresql/data. You need to either change the access permissions, or mount to your home folder using nfs instead. The latter is described here: http://syskall.com/using-boot2docker-using-nfs-instead-of-vboxsf/
You should also add "$(boot2docker shellinit 2> /dev/null)" to the bottom of your ~/.bash_profile.
Next, you may need to fix the ip/certification bug https://github.com/boot2docker/boot2docker/issues/824#issuecomment-113904164. To edit the profile, run:
$ boot2docker ssh -t sudo vi /var/lib/boot2docker/profile)
Now when you reach the initialization stage, you need to issue:
$ run assets build
to create the manifest.json first, before issuing run to see a list of what's available.
You then may need to replace all instances of "website_postgres_1" in cli/commands/cmd_db.py with the correct name of the postgresql host (eg. "buildasaasappwithflask_postgres_1").
The last issue is unresolved. When I try to initialize the database, I get the following error:
$ run db create DROP ROLE CREATE ROLE CREATE DATABASE catwatch; GRANT Traceback (most recent call last): File "/Users/rog/.virtualenvs/catwatch/bin/run", line 9, in
load_entry_point('Commands-to-help-manage-your-project==1.0', 'console_scripts', 'run')() File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/click/core.py", line 664, in call return self.main(_args, _kwargs) File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/click/core.py", line 644, in main rv = self.invoke(ctx) File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/click/core.py", line 991, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/click/core.py", line 991, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/click/core.py", line 837, in invoke return ctx.invoke(self.callback, _ctx.params) File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/click/core.py", line 464, in invoke return callback(_args, kwargs) File "/Users/rog/Workspace/build-a-saas-app-with-flask/cli/commands/cmd_db.py", line 157, in create return db.create_all() File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/flask_sqlalchemy/init.py", line 895, in create_all self._execute_for_all_tables(app, bind, 'create_all') File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/flask_sqlalchemy/init.py", line 887, in _execute_for_all_tables op(bind=self.get_engine(app, bind), extra) File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 3622, in create_all tables=tables) File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1853, in _run_visitor with self._optional_conn_ctx_manager(connection) as conn: File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 17, in enter return self.gen.next() File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1846, in _optional_conn_ctx_manager with self.contextual_connect() as conn: File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2037, in contextual_connect self._wrap_pool_connect(self.pool.connect, None), File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2076, in _wrap_pool_connect e, dialect, self) File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1405, in _handle_dbapi_exception_noconnection exc_info File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause reraise(type(exception), exception, tb=exc_tb) File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2072, in _wrap_pool_connect return fn() File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/pool.py", line 376, in connect return _ConnectionFairy._checkout(self) File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/pool.py", line 708, in _checkout fairy = _ConnectionRecord.checkout(pool) File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/pool.py", line 480, in checkout rec = pool._do_get() File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/pool.py", line 1055, in _do_get self._dec_overflow() File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in exit compat.reraise(exc_type, exc_value, exc_tb) File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/pool.py", line 1052, in _do_get return self._create_connection() File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/pool.py", line 323, in _create_connection return _ConnectionRecord(self) File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/pool.py", line 449, in init self.connection = self.connect() File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/pool.py", line 602, in connect connection = self.pool._invoke_creator(self) File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 97, in connect return dialect.connect(_cargs, _cparams) File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 385, in connect return self.dbapi.connect(_cargs, _cparams) File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/psycopg2/__init.py", line 164, in connect conn = _connect(dsn, connection_factory=connection_factory, async=async) sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL: role "catwatch" does not exist So now I'm stuck!
Thanks for the great project, Nick. I look forward to the finished product!
— Reply to this email directly or view it on GitHub https://github.com/nickjj/build-a-saas-app-with-flask/issues/18.
William J. Sankey Johns Hopkins University MA Public Policy '12
@stuckeyr Thanks for the detailed report.
I'm in the process of re-writing the db
command now to make it a lot more user friendly and generic. Once it's finished you won't have to edit the cmd_db.py
file and all of the commands will work as well as be idempotent.
That error you're experiencing at the end is mostly due to skimpy documentation on my part because I planned to create much better docs in the future.
To set up a brand new database for now (just to get rolling), you can run the 2 commands below:
docker exec -it buildasaasappwithflask_postgres_1 createdb -U postgres buildasaasappwithflask
docker exec -it buildasaasappwithflask_postgres_1 psql -U postgres -c "CREATE USER buildasaasappwithflask WITH PASSWORD 'securepassword'; GRANT ALL PRIVILEGES ON DATABASE buildasaasappwithflask to buildasaasappwithflask;"
Then, ensure buildasaasappwithflask
is set as the POSTGRES_USER
in your docker-compose.yml
file and that the password matches in the command above. In the repo it's set to catwatch
, that's why you're getting the role error.
https://github.com/nickjj/build-a-saas-app-with-flask/blob/master/docker-compose.yml#L4
Here's 2 main take aways:
buildasaasappwithflask_postgres_1
is how docker-compose names your container, but the first bit will be based on the folder name it's running from. I will do this in a better way once I patch it today, but for now that needs to match up too.For the second take away, you can always run docker ps
to get a list of all running containers to get their name.
As for the mac related issues. I don't personally use a Mac but I know a lot of devs do. When it comes time to record the screencasts I will try to get my hands on a Mac and create a short video on setting up boot2docker in such a way that once it's done it will align with the rest of the series.
I've pushed a new build to master that has the improved run db
commands. Please try it out and let me know if you have any issues. You won't have to run any of the extra commands from the reply above.
Of course your docker-compose.yml
file will still need to have the same values as your settings.py
file when it comes to the database related connection values.
Thanks Nick. I changed the username in settings.py
:
db_uri = 'postgresql://buildasaasappwithflask:bestpassword@localhost:5432/{0}'
and docker-compose.yml
, as suggested:
postgres:
image: postgres:9.4.3
environment:
POSTGRES_USER: buildasaasappwithflask
POSTGRES_PASSWORD: bestpassword
Unfortunately, I'm still getting errors :-(
$ run db reset
ERROR: role "buildasaasappwithflask" already exists
Traceback (most recent call last):
File "/Users/rog/.virtualenvs/catwatch/bin/run", line 9, in <module>
load_entry_point('Commands-to-help-manage-your-project==1.0', 'console_scripts', 'run')()
File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/click/core.py", line 664, in __call__
return self.main(*args, **kwargs)
File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/click/core.py", line 644, in main
rv = self.invoke(ctx)
File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/click/core.py", line 991, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/click/core.py", line 991, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/click/core.py", line 837, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/click/core.py", line 464, in invoke
return callback(*args, **kwargs)
File "/Users/rog/Workspace/build-a-saas-app-with-flask/cli/commands/cmd_db.py", line 226, in reset
ctx.invoke(create, databases=databases)
File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/click/core.py", line 464, in invoke
return callback(*args, **kwargs)
File "/Users/rog/Workspace/build-a-saas-app-with-flask/cli/commands/cmd_db.py", line 199, in create
return db.create_all()
File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/flask_sqlalchemy/__init__.py", line 895, in create_all
self._execute_for_all_tables(app, bind, 'create_all')
File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/flask_sqlalchemy/__init__.py", line 887, in _execute_for_all_tables
op(bind=self.get_engine(app, bind), **extra)
File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 3622, in create_all
tables=tables)
File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1853, in _run_visitor
with self._optional_conn_ctx_manager(connection) as conn:
File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1846, in _optional_conn_ctx_manager
with self.contextual_connect() as conn:
File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2037, in contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2076, in _wrap_pool_connect
e, dialect, self)
File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1405, in _handle_dbapi_exception_noconnection
exc_info
File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb)
File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2072, in _wrap_pool_connect
return fn()
File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/pool.py", line 376, in connect
return _ConnectionFairy._checkout(self)
File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/pool.py", line 708, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/pool.py", line 480, in checkout
rec = pool._do_get()
File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/pool.py", line 1055, in _do_get
self._dec_overflow()
File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/pool.py", line 1052, in _do_get
return self._create_connection()
File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/pool.py", line 323, in _create_connection
return _ConnectionRecord(self)
File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/pool.py", line 449, in __init__
self.connection = self.__connect()
File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/pool.py", line 602, in __connect
connection = self.__pool._invoke_creator(self)
File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 97, in connect
return dialect.connect(*cargs, **cparams)
File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 385, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/Users/rog/.virtualenvs/catwatch/lib/python2.7/site-packages/psycopg2/__init__.py", line 164, in connect
conn = _connect(dsn, connection_factory=connection_factory, async=async)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL: role "buildasaasappwithflask" does not exist
The docker-compose log complains:
postgres_1 | ERROR: role "buildasaasappwithflask" already exists
postgres_1 | STATEMENT: CREATE USER buildasaasappwithflask WITH PASSWORD 'bestpassword'
Incidentally, I switched to dinghy from boot2docker to address my previous issues and it seems to work well. For anyone else interested in trying it, don't forget to remove the $(boot2docker shellinit 2> /dev/null)
command from your ~/.bash_profile!
Did you pull in the changes from master?
Yep. By the way, the first time I run db reset
(after removing and recreating the containers with docker-compose rm && docker-compose up
), I get an error with catwatch again:
dropdb: database removal failed: ERROR: database "catwatch" does not exist
ERROR: role "buildasaasappwithflask" already exists
...
Here's the output of docker-compose ps
, if it helps:
Name Command State Ports
---------------------------------------------------------------------------------------------------
buildasaasappwithflask_faye_1 /bin/sh -c npm start Up 0.0.0.0:4242->4242/tcp
buildasaasappwithflask_postgres_1 /docker-entrypoint.sh postgres Up 0.0.0.0:5432->5432/tcp
buildasaasappwithflask_redis_1 /entrypoint.sh redis-server Up 0.0.0.0:6379->6379/tcp
Oh, it uses catwatch
because the APP_NAME
setting controls the db name by default.
https://github.com/nickjj/build-a-saas-app-with-flask/blob/master/config/settings.py#L7
There's 3 places to keep track of all db info:
APP_NAME
(defaults as the database name)SQLALCHEMY_DATABASE_URI
(database connection settings)Yes, I noticed that. I still get the role error if I change:
APP_NAME = 'buildasaasappwithflask'
Can you paste the output of the following 2 commands:
run db list
run db psql "\du"
Everything seems to be there...
$ run db list
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
------------------------+----------+----------+------------+------------+-----------------------
buildasaasappwithflask | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
catwatch | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
(5 rows)
$ run db psql "\du"
List of roles
Role name | Attributes | Member of
------------------------+------------------------------------------------+-----------
buildasaasappwithflask | Superuser | {}
postgres | Superuser, Create role, Create DB, Replication | {}
Is the role error preventing you from doing anything?
A common work flow to refresh everything would be:
run db reset buildasaasappwithflask buildasaasappwithflask_test
run add all
# ^ You may want to do `run add users` for now since "all" will
# make hits directly to the payment gateway to create coupons
# and you probably haven't set up your API keys yet.
I use the above commands to purge everything, or to perform a schema change without wanting to create a migration.
Alternatively if you just run run db reset
it will use the APP_NAME
as the default db name, but this won't touch your test database.
Edit: To be clear, the run add all
step is optional. The reset command will seed a single admin user, the add
commands just populate your db with a bunch of contextually accurate fake data.
Yes, it's preventing me from going any further. The commands you suggested produce the same error :-( Thanks again for putting the time into this. I'll look into it again tomorrow.
No problem. I will do a full test on a brand new virtual machine sometime later today or tomorrow and fix issues that arise.
If anyone else has issues related to this between now and then, please post them here.
I spun up a fresh VM last night. I was able to get everything working without issues.
You will see errors/warnings when running db reset
sometimes but that's just postgresql being noisy. It does work in the end with the intended results.
You may see additional errors about other apps using the database too, in this case you should stop your flask server before running the db reset
command.
Started from scratch, this time, using 'catwatch' for the db name... Same result :-( I suspect it's a permissions issue, as mentioned in my first post. I might try just using my local postgresql server instead. I was looking forward to trying Docker for the various services. Seems like there are still a lot of issues (on OSX) to be resolved though.
I'm not sure if it's a permission issue. You were able to create the database so it's capable of writing to disk. There's a lot of people using docker successfully with osx, I'll try to get a friend who has a mac to do a test run.
Finally figured it out! You were right - it wasn't an issue with the permissions. I had inadvertently been connecting to my local postgresql server (I thought I'd shut it down), which I'm guessing the other services - redis & faye - couldn't see.
That was only half the problem though: My VM hadn't exposed all the necessary ports (specifically, postgres: 5432) to the host. There are a couple of ways to do this, but I since I was using dinghy, I ended up using an http proxy by setting the VIRTUAL_HOST
field in docker-compose.yml
:
postgres:
image: postgres:9.4.3
environment:
POSTGRES_USER: catwatch
POSTGRES_PASSWORD: bestpassword
VIRTUAL_HOST: postgres.docker
ports:
- 5432:5432
volumes:
- ~/.docker-volumes/catwatch/postgresql/data:/Users/rog/var/lib/postgresql/data
And using the same virtual host in config/settings.py
:
db_uri = 'postgresql://catwatch:bestpassword@postgres.docker:5432/{0}'
I assume I'll need to do the same for redis ad faye, but haven't gotten around to those yet.
Oh, and one other amendment I forgot to mention earlier was to pip install requests[security]
to get around the OpenSSL incompatibility discussed here.
When I'm confident everything's set up correctly, I'll run the tests...
@stuckeyr Do you think you would try boot2docker again in the future once the certificate bug is fixed? It's schedule to be fixed in 1.7.1 which should be out before the videos are released. Once fixed you shouldn't have to install anything extra or mess around the boot2docker image.
I did run into the same certificate issues as you, it seems to be a bug affecting a lot of Mac users.
@nickjj Yep, sure. I kinda like dinghy/vagrant, but It would be nice to have a plain vanilla solution that didn't require any mucking around!
Going to close this for now as it's no longer an issue.
I had many issues getting the Docker containers going on my Mac, so I thought I'd share some workarounds here. I'm fairly new to Docker, so there may be mistakes! There are a couple of project-related issues at the end too :-)
First, install Docker, boot2docker and Docker-compose.
There are problems with the permissions on vboxsf mounts, preventing postgres from writing to /var/lib/postgresql/data. You need to either change the access permissions, or mount to your home folder using nfs instead. The latter is described here: http://syskall.com/using-boot2docker-using-nfs-instead-of-vboxsf/
You should also add "$(boot2docker shellinit 2> /dev/null)" to the bottom of your ~/.bash_profile.
Next, you may need to fix the ip/certification bug. To edit the profile, run:
Now when you reach the initialization stage, you need to issue:
to create the manifest.json first, before issuing
run
to see a list of what's available.You then may need to replace all instances of "website_postgres_1" in cli/commands/cmd_db.py with the correct name of the postgresql host (eg. "buildasaasappwithflask_postgres_1").
The last issue is unresolved. When I try to initialize the database, I get the following error:
So now I'm stuck!
Thanks for the great project, Nick. I look forward to the finished product!