Revisiting OS X Postgres permissions issue

[bradddd]

Has anyone found a quick fix for the vboxsf mounts on OS X that stuckeyr pointed out in #18 ?

The script he linked to uses the now deprecated boot2docker to mount the share as nfs within the host vm.

Is that still the only route to go?

In the interim, just to get it running, I was just commenting out the volumes settings. Correct me if I'm wrong, but that should just disable data persistence, right?

After doing that, I'm getting the following output when trying to run db reset:

dropdb: database removal failed: ERROR:  database "exampleproject" does not exist
CREATE ROLE
Traceback (most recent call last):

...

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name "postgres" to address: nodename nor servname provided, or not known

The postgres container output is:

postgres_1 | ERROR:  database "exampleproject" does not exist
postgres_1 | STATEMENT:  DROP DATABASE exampleproject;

[nickjj]

The new way to get things rolling on OSX is to use Docker Toolbox (Mac guide).

The sqlalchemy exception looks like something unrelated. Can you paste your docker-compose.yml file?

[bradddd]

Yeah. I've got the Docker Toolbox and am able to get the containers up and running without any problems if I comment out the volume keys in the docker-compose.yml file. Otherwise the postgres container complains about permission denied on the /var/lib/postgresql/data directory.

The only difference in my docker-compose.yml file is that I've got lines 8,9,15,16 commented out.

[bradddd]

I didn't address the permission issue with mounting the folders for data persistence yet, but I was able to get it running. I'm still having some issues w/ the db though.

The error part about postgres was because the port wasn't being properly forwarded to my host on OSX. I think I was confused about the output from docker-compose ps. I was under the impression that the ports were being forwarded to localhost, but I don't think that's the case. I think they're just forwarded to the docker host.

So, I needed to use the IP Instead of "postgres" and "redis" in the settings.py file. I used the address given from $ docker-machine ip default (where default is the name of my docker host. In this case it was 192.168.99.100)

But then I still run into issues when trying to initialize the db:

$ run db reset
ERROR:  role "catwatch" already exists

And if I go a step further and try to run the tests, it unravels.

>       conn = _connect(dsn, connection_factory=connection_factory, async=async)
E       OperationalError: (psycopg2.OperationalError) FATAL:  database "catwatch_test" does not exist

../venv/lib/python2.7/site-packages/psycopg2/__init__.py:164: OperationalError
==============7 passed, 82 error in 35.31 seconds ==============

Running run db init yields no output, so I'm assuming that's working.

Looking at the catwatch db, I see 6 tables, so those are at least being created. I also am able to register a user through the Flask app and see it appear in the users table.

[nickjj]

If your app isn't running in a container (it's not by default), then you would access postgres and redis through localhost under normal conditions. However, since you're using OSX you're right, you do need to use the output of docker-machine ip default.

The error when running run db reset is ok, that's just postgres being noisy.

The error when running run test is also normal because you can't just run run db reset if you want to run tests, you'd have to run run db reset catwatch catwatch_test because without doing that, you never set up the test database. I should probably update the README about that.

I thought about making run db reset also assume that you want another database with the _test suffix added to it but that's not always the case. For example in production you wouldn't want that db made, so I just went the other route where you have to be explicit when you want to create a test db.

Running run db init is something you don't have to do in development btw. run db reset catwatch catwatch_test is all you need to do, then afterwards if you want to seed your database with fake data you can do run add users. I recommend running run add to see what's available to add, just keep in mind some of them require having Stripe set up.

run db init is mainly used when provisioning a staging/production server because run db reset assumes you have Docker installed wherever you're running your web app. In a typical production scenario you would have your app running in its own container but Docker itself would not be installed in that container, so you couldn't run run db reset. run db init is something you'd run to set up your initial schema after creating the database/role directly.

[bradddd]

Thanks. That clears up a lot, but the annoying vboxsf mounts issue still remains. It's not a specific issue for your app though, but I'm still curious how others on OSX are handling it.

[nickjj]

Unfortunately I don't have a Mac to test this on. Are you not able to mount it into your home directory?

For instance I keep all of my docker volumes in ~/.docker-volumes.

[bradddd]

Since the containers can't run natively on OSX, they're running in the docker-machine vm, powered by virtualbox hypervisor. So the volume key in the docker-compose.yml file is specifying the vm folder to mount. As is, virtualbox handles that w/ a vboxsf which doesn't work. #18 linked to a solution that mounts the host OSX folder in the vm using nfs. It sounded as though that fixed that issue for #18, but it doesn't for me. (Here's a copy of the NFS script by the same author, that he ported from boot2docker to docker-machine.) This then let's you mount the OSX directory in the docker VM that you can then mount in the containers.

It mounts successfully, but then during docker-compose up the postgres setup generates this error:

postgres_1 | chown: changing ownership of `/var/lib/postgresql/data': Operation not permitted

Which appears to be a limitation of nfs shares.

For now, I think I'll just roll without the data persistence. If I have time, I might look into a data only container as a solution.

[nickjj]

I'd be curious if you ever find a solution. I'll keep this ticket open simply because this sounds like it'll be a problem for anyone using OSX.

@stuckeyr have you switched over to machine and were able to get volumes working?

[nickjj]

@bradddd Would you mind giving this tool a shot? It doesn't depend on Docker Toolbox or VirtualBox.

https://github.com/nlf/dlite

The docs are skimpy but apparently all you have to do is install it and then use Docker as if you were running it straight on Linux.

[bradddd]

Yeah. I'll try it later tonight or tomorrow. On Sep 25, 2015 12:50 PM, "Nick Janetakis" notifications@github.com wrote:

@bradddd https://github.com/bradddd Would you mind giving this tool a shot? It doesn't depend on Docker Toolbox or VirtualBox.

https://github.com/nlf/dlite

The docs are skimpy but apparently all you have to do is install it and then use Docker as if you were running it straight on Linux.

— Reply to this email directly or view it on GitHub https://github.com/nickjj/build-a-saas-app-with-flask/issues/28#issuecomment-143277539 .

[nickjj]

Thanks. It uses https://github.com/nlf/dhyve under the hood so you might need to run the "Usage" commands in that repo's README before being able to use dlite.

I'm in the process of trying to talk to people who can give a better answer or perhaps even share a small guide on how to get it all working.

[bradddd]

Sorry to not respond sooner. I messed with it weekend before last, but wasn't able to get anywhere.

The long story short is that it doesn't fix the issue. I ultimately just installed dhyve and set up a vm to run my containers (couldn't get dlite to work w/ docker-compose). Using dhyve instead of virtual box for the host does produce the NFS share and mount the /Users directory of the host machine into the vm, but the problem still exists with changing permissions of the directory once it's mounted into a container. There just isn't a great solution for that when dealing with NFS shares. You can alter the directory's permissions and uid/gid in the host before you mount the volume, but I stopped short of fooling around with all of that. I think I'm just going to install ubuntu and dual boot it on my macbook.

One benefit that dhyve has over the virtualbox setup is how it deals w/ localhost. It seems to tie things together better, but I was still having some issues getting all the containers and services talking.

As an aside, going back into your repo to change the local host references to IP addresses, I got confused between the config.py and settings.py files in the config and instance directories respectively. Does the settings file overwrite config when it's present? I couldn't remember off the top of my head.

[nickjj]

Thanks for the details, it's unfortunate you weren't able to get something going.

The difference between the 2 configs are explained in the videos.

The instance/settings.py file overwrites the config/settings.py file. The instance config is meant to be outside of version control and would contain private info like mail logins and API keys.

[bradddd]

No worries. It still works w/ minimal tweaking just by commenting out the volumes. I think I might play around with refactoring the volumes to a separate data container.

Do you have any tips on debugging containers? My first stab gets all of the containers running, but then when I run anything db related, they all stop. I assume that means there's some issue w/ postgres, but I'm not seeing any useful output,

[nickjj]

Permission related issues should be fixed by using named volumes that were introduced with Docker 1.9.