How to instantiate docker-compose from /app_dir/bin directory in Travis

MikeTheCanuck commented 7 years ago

The team-budget repo currently has the following file layout: .travis.yml /budget_proj/Dockerfile /budget_proj/docker-compose.yml /budget_proj/bin/build-proj.sh /budget_proj/budget_proj/settings.py [and other app files]

When a commit is made to the associated GitHub repo, Travis picks up its .travis.yml from the root of its current working directory (e.g. /home/travis/build/MikeTheCanuck/team-budget) and executes the included script directive.

The original directive ran ./bin/build-proj.sh which resulted in the following error:

$ ./bin/build-proj.sh
/home/travis/build.sh: line 62: ./bin/build-proj.sh: No such file or directory
The command "./bin/build-proj.sh" exited with 127.

We have changed the directive to run ./budget_proj/bin/build_proj.sh, which allows Travis to successfully find and run the build_proj.sh script.

However, even when build-proj.sh starts successfully, the included docker-compose build command is unable to find the docker-compose.yml that's in a parent directory:

$ ./budget_proj/bin/build-proj.sh
/home/travis/build/MikeTheCanuck/team-budget
ERROR: 
        Can't find a suitable configuration file in this directory or any
        parent. Are you in the right directory?
        Supported filenames: docker-compose.yml, docker-compose.yaml

This is despite the fact that docker-compose appears to have parent-directory-crawling built in.

EDIT: as discovered below, the reason for this error is that the current working directory for the ./budget_proj/bin/build-proj.sh script is the root of the project, e.g. /home/travis/build/hackoregon/team-budget, and docker-compose.yml is actually in a child directory, not a parent directory.

MikeTheCanuck commented 7 years ago

What if the current working directory of the build-proj.sh script isn't /budget_proj/bin but rather just / (i.e, /home/travis/build/MikeTheCanuck/team-budget)? Then it would need to find docker-compose.yml in the / root directory.

MikeTheCanuck commented 7 years ago

Yes, apparently the CWD is /home/travis/build/MikeTheCanuck/team-budget, and having docker-compose.yml at the root of the project fixes the "Can't find a suitable configuration file" error:

$ ./budget_proj/bin/build-proj.sh
/home/travis/build/MikeTheCanuck/team-budget
Building web
ERROR: Cannot locate specified Dockerfile: Dockerfile

I'll try moving the Dockerfile to the root of the project, even though I could swear that caused problems the last time I tried to do this.

MikeTheCanuck commented 7 years ago

Here's the next domino in this sequence of debugging the Travis-automated build and run of the Django app.

I've moved Dockerfile and docker-compose.yml to the / root of the repo alongside .travis.yml, and moved the docker-entrypoint.sh script to /build_proj/bin/ along with all the rest of the project's shell scripts.

It results in the following errors thrown once the docker-entrypoint.sh script runs:

web_1  | python3: can't open file 'manage.py': [Errno 2] No such file or directory
web_1  | python3: can't open file 'manage.py': [Errno 2] No such file or directory
...
web_1  | ImportError: No module named 'budget_proj.wsgi'

I'm trying to determine what Docker's CWD is when running these scripts.

I tried prepending budget_proj. to the manage.py call, but that didn't work (I was thinking like a Python developer, not like a bash user):

web_1  | python3: can't open file 'budget_proj.manage.py': [Errno 2] No such file or directory

Also, I noticed that the source ./bin/env.sh script was failing, because that too has a path that no longer matches the calling script's environment. I've hard-coded the /budget_proj/ prefix there too.

Wouldn't it be nice for all these scripts to work from a PATH environment that told the script to look in /budget_proj/bin/ for all scripts? Maybe I'll figure that out next - filed as team-budget issue 55.

MikeTheCanuck commented 7 years ago

After hard-coding the paths in docker-entrypoint.sh, the next build in Travis/Docker results in the following runtime errors (from this build):

...
web_1  |   File "/code/budget_proj/budget_proj/settings.py", line 15, in <module>
web_1  |     from . import project_config
web_1  | ImportError: cannot import name 'project_config'
...
web_1  |   File "/usr/local/lib/python3.5/importlib/__init__.py", line 126, in import_module
web_1  |     return _bootstrap._gcd_import(name[level:], package, level)
web_1  | ImportError: No module named 'budget_proj.settings'
...

Q: are these both because I haven't setup the S3 bucket download for the project_config.py file? Or is the second error independent of this dependency?

MikeTheCanuck commented 7 years ago

Hours later, I've added "budget_proj" to an insane number of folders, and things still aren't running correctly. You can see all the adjustments I've made in team-budget Pull request 54, and I'm still not done.

At this point I've gotten to a point where I'm getting a variety of errors like:

ImportError: No module named 'budget_proj.budget_proj' ImportError: No module named 'budget_app' ImportError: No module named 'budget_proj.wsgi' ImportError: No module named 'budget_proj.settings'

And now I'm chasing my tail in circles, just looking for any place where these strings are referenced and trying to insert Python- or Bash-isms that I was hoping would close the gaps.

Aside

I've got one more thing to try, and if that doesn't work I am going to need help to either (a) hack the necessary subdirectory path-ing into all the files (e.g. Import statements), or (b) figure out all the working directory locations so that we can call commands correctly without affecting any of the Imports. Worst case, I'd like to re-open the discussion on putting the entire app in a subdirectory, if we aren't able to get the Docker build to stabilize.

MikeTheCanuck commented 7 years ago

Success, of a limited (but repeatable) sort:

I wiped all branches and started over from master
generated a new docker_and_travis branch
used the -f flag with docker-compose to force it to look for the docker-compose.yml template in a subdirectory from Travis' current working directory

Small victory, but I can live with it.

This is resolved as of Team-Budget PR #57.

MikeTheCanuck commented 7 years ago

After giving it some time to figure out what to do, I have rewritten all .sh scripts and references to run as if they're executed from the root of the repo, and explicitly call everything via './budget_proj/`, e.g.

docker-compose -f budget_proj/docker-compose.yml build budget-service

./budget_proj/bin/getconfig.sh

hackoregon / devops-17

How to instantiate docker-compose from /app_dir/bin directory in Travis #34

Aside