Resolve docker-compose issues in #69

squ1rr3lly commented 4 years ago

This pull request resolves #69 and brings the Dockerfile and docker-compose architecture into more modern versions and best-practices.

The main functional changes include:

Using environment variables to control properties
- E.g. SHAARK_PORT=8080 SHAARK_URL=shaark.awesome docker-compose up -d will expose the service on port 8080 and set the APP_URL rather than using the defaults in the compse file.
Using entrypoint script entrypoint-shaark.sh to ensure database is only seeded once, and migrated only on demand when env SHAARK_MIGRATE_DB=true
Resolved the container race condition in #69 using wait-for-it.sh script in the the shaark entrypoint to ensure mariadb is up
setEnv.sh script allows linking docker-compose.override.yml file to either 'dev' or 'production' mode. 'dev' mode exposes mariadb on 3306, enables debugging, and enables migrations.

This does remove the nginx network and expose the shaark container directly. I think it's more straight-forward for development or running behind a load balancer etc., and someone who wanted to use a reverse-proxy could easily modify the compose file. Could also make revese-proxy another setEnv.sh mode.

For the user who just wants it to go, cloning the repo and running docker-compose up will bring shaark up on port 80 with all the defaults.

Known issues:

[x] youtube-dl doesn't work
[x] Archiving doesn't work
[x] e-mail doesn't work

Enhancements:

[ ] Update documentation
[ ] Add testing mode to setEnv.sh
[x] Clean up the seed/migrate logic in the entrypoint scripts

MarceauKa commented 4 years ago

Thank you @squ1rr3lly This PR is now in draft until know issues are resolved

squ1rr3lly commented 4 years ago

All features including archiving and backups runs with docker-compose and the shaark Dockerfile would be more usable in other environments. I think this is production-ready, and would even be able to scale in a K8s/ECS/etc. deployment. I also made some code updates outside of just Docker, primarily switching the library used for PDF archiving to something that seemed better supported (spatie/browsershot).

Main changes:

Code:

- Replaced nesk/puphpeteer with spatie/browsershot; seems better supported and Shaark uses a lot of other spatie packages. I think this will be much easier to maintain with same functionality.
- The PDF Archive test gets http://example.com instead of app->(home)

Dockerfile:

- Updated image to php:alpine
- Chromium and spatie/browsershot for archiving
- Install youtube-dl for archiving
- Created regular user "shaarkuser" that runs the server with necessary capabilities set on entrypoint script; no longer have to run as root
- Set up crontab for backups

docker-compose:

- made the app/storage directory a volume; stores archives, backups, and all settings through rebuild/redeploy
- SHAARK_ADMIN_PASSWORD env variable allows setting initial admin user pass; defaults to "secret"
- Sets random mysql root password

Entrypoint:

- Cleaned up logic for determining when migrations/seeding should be performed
- Starts the artisan work queue

tborychowski commented 4 years ago

Hi, I've checked-in this branch and spent a couple of hours but can't make it work :-( Getting the same error no matter what I do. Without any fiddling this is what I do:

SHAARK_PORT=8080 docker-compose up -d

logs show:

shaark_1   | wait-for-it.sh: waiting 30 seconds for mariadb:3306
shaark_1   | wait-for-it.sh: mariadb:3306 is available after 1 seconds
shaark_1   | Clearing any cached config.
shaark_1   | Configuration cache cleared!
shaark_1   | Database migration skipped.
shaark_1   | Starting Shaark!
shaark_1   | Laravel development server started: http://0.0.0.0:80
shaark_1   | [Tue Aug 25 07:30:51 2020] PHP 7.4.9 Development Server (http://0.0.0.0:80) started
shaark_1   | [Tue Aug 25 07:30:53 2020] 172.25.0.1:36878 Accepted
shaark_1   | [Tue Aug 25 07:30:53 2020] 172.25.0.1:36878 Closing
shaark_1   | [Tue Aug 25 07:30:53 2020] 172.25.0.1:36886 Accepted
shaark_1   | [Tue Aug 25 07:30:53 2020] 172.25.0.1:36886 [200]: GET /favicon.ico

But the browser's devtools network tab shows:

MarceauKa commented 4 years ago

Thank you @squ1rr3lly

squ1rr3lly commented 4 years ago

Hey @tborychowski,

Have you rebuilt the images with a command such as docker-compose build or docker-compose up -d --build? If you still have an older shark:latest image in there it will get used, but won't work with the new compose file.

tborychowski commented 4 years ago

Yes I did. And did it again just now. Same issue. The only other thing in the logs is this warnings:

mariadb_1  | 2020-08-25 17:18:03 0 [Warning] 'proxies_priv' entry '@% root@f6a0e13d670a' ignored in --skip-name-resolve mode.
mariadb_1  | 2020-08-25 17:18:03 0 [Note] Reading of all Master_info entries succeeded
mariadb_1  | 2020-08-25 17:18:03 0 [Note] Added new Master_info '' to hash table
mariadb_1  | 2020-08-25 17:18:03 0 [Note] mysqld: ready for connections.
mariadb_1  | Version: '10.5.5-MariaDB-1:10.5.5+maria~focal'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
mariadb_1  | 2020-08-25 17:18:03 0 [Note] InnoDB: Buffer pool(s) load completed at 200825 17:18:03
mariadb_1  | 2020-08-25 17:18:04 3 [Warning] Aborted connection 3 to db: 'unconnected' user: 'unauthenticated' host: '172.26.0.4' (This connection closed normally without authentication)

Maybe I just wait until the official image is out? :-)

tborychowski commented 4 years ago

Ok I finally managed to fix my issue. The problem was that the container's storage folder was owned by root (which I verified from inside of the container). Once I changed the volume to be mounted into the host-local storage folder (and set the permissions to 777 just in case) - it started working!

squ1rr3lly commented 4 years ago

@tborychowski

Those WARNINGS are expected, the aborted db connection is the entrypoint making sure the DB is up before starting shaark.

I just can't reproduce what you're seeing. I usually only saw a 500 when APP_URL wasn't right, but http://localhost is the default. Have you tried destroying the mariadb and redis volumes so they are rebuilt clean; a mismatch in that database volume is the first thing coming to mind? Anything in /app/storage/logs on the shaark container? You could also try running app/setEnv.sh dev to enable debugging and some extra logs then run again.

[edit] Awesome! Glad it's working for you. Was that something specific to your system or a change you had to make to the Docker/compose file? The Dockerfile is supposed to make everything in /app owned by shaarkuser.

tborychowski commented 4 years ago

@squ1rr3lly, I have no idea whether this was specific to that system. I will try to do that on another machine (ubuntu this time) and see if it works straight away or do I need to make the same changes.

The only thing I had to change in docker-compose is the volumes entry for shaark service, to this:

    volumes:
      - ./storage:/app/storage

Which is good for me as I actually prefer local folders (they make it easier for my backup scripts). Anyway, thanks for your help and looking forward to using this app! (and to the official image in docker hub!) 😄

edit: I've just tested it on Ubuntu 20.04 and it worked without any changes! Not sure why it wasn't on a mac :-|

MarceauKa / shaark

Resolve docker-compose issues in #69 #83