Docker fixes - new compose (podman support also)

hozza commented 1 year ago

Potentially helping docker related issues #741 and #523

Repo Linking

There are several links under different usernames/repos. I'm not sure how mantainers want to manaage this but it would help with docker issues if there could be one official repo that's linked everywhere and also published on on Docker Hub under this name also.

"https://github.com/stringer-rss/stringer/" links to "https://github.com/swanson/stringer" under the repos about section.
The docker docs reference "stringer-rss/stringer" (presumably expanding to "docker.io/stringer-rss/stringer")
The docker-compose file references "mockdeep/stringer" (presumably expanding to "docker.io/mockdeep/stringer")

Compose Tweaks

Fixes and improvement suggestion plus support rootless containers, (podman, podman-compose), if you're happy with these suggestions I'll submit PR. These tweaks make the docker docs accurate also.

services:
  stringer-postgres:
    image: docker.io/postgres:9.5-alpine # full registry paths for non-docker compatibility/verbosity 
    container_name: stringer-postgres # helps with podman-compose
    restart: unless-stopped
    volumes:
      - stringer:/var/lib/postgresql/data # use volume rather than mount point
    environment:
      - PUID=1000 # perms issues for the old ~/ vol mount
      - PGID=1000
      - POSTGRES_USER=db_user
      - POSTGRES_DB=stringer
      - POSTGRES_PASSWORD=super_secret_password # TODO change me

  stringer:
    image: docker.io/mockdeep/stringer # maybe stringer-rss/stringer? # full registry paths for non-docker compatibility/verbosity 
    container_name: stringer # helps with podman-compose
    depends_on:
      - stringer-postgres
    restart: unless-stopped
    ports:
      - 8080:8080 # not on privileged port for rootless
    environment:
      - PORT=8080
      - DATABASE_URL=postgres://db_user:super_secret_password@stringer-postgres:5432/stringer  # TODO change me
#      - SECRET_KEY_BASE=<your configuration>  # TODO change me
#      - ENCRYPTION_PRIMARY_KEY<your configuration>  # TODO change me
#      - ENCRYPTION_DETERMINISTIC_KEY=<your configuration>  # TODO change me
#      - ENCRYPTION_KEY_DERIVATION_SALT=<your configuration>  # TODO change me

Container suggestions

can we migrate to alpine/chainguard base image? Image is currently >1GB.
can the DATABASE_URL be generated with other env vars to make the config dryer? maybe using docker-compose env file.
can we get an explanation about the new encryption env vars rather than just commit msg? maybe on main README or in this compose

mockdeep commented 1 year ago

@hozza thanks for opening this! A PR would be very welcome, as my Docker experience is extremely limited. I end up running in circles every time I try to dig in. We've also been chatting about it in #1050. Maybe @guidopetri will have opinions on this as well.

guidopetri commented 1 year ago

I'll agree on all points above :) I'm not super confident on the PUID/PGID nor on the podman side of things (I don't run it myself) but migrating to alpine base image sounds good, and using an env file also sounds great (and I do this myself, actually, heh). One potential addition here is to make the port binding localhost only (127.0.0.1:8080:8080) and use a reverse proxy in front of stringer to make it more secure.

I also agree that the different links (swanson, mockdeep, stringer-rss) are a bit confusing. I'm partial to keeping everything under stringer-rss personally, but I'm also not the maintainer :)

mockdeep commented 1 year ago

I think the stringer-rss namespace makes sense.

hozza commented 1 year ago

Ok great stuff. So it looks like there's a few work streams here:

Repo Namespace. Settling on "stringer-rss", can the repo about link and docs be updated? @mockdeep also is the image published to this namespace on docker hub also? (can we remove or redirect the others?)
Compose updates. I'll submit a PR with updated compose docs and new compose file. However I'll leave the namespace/repo references out of the compose PR - @guidopetri can you share your compose/env file setup? I could include this in the PR.
Docker image building CI updates. This would tackle arm and potential alpine base migration for more manageable sizes.

guidopetri commented 1 year ago

It's nothing fancy. I have a bunch of other services in the same compose file, so here's the section for stringer:

stringer:
  build: https://github.com/guidopetri/stringer.git#rails_env
  container_name: stringer
  restart: always
  ports:
    - "127.0.0.1:5000:5000"
  depends_on:
    - postgres
  env_file:
    - ${HOME}/.env_files/.stringer_env

And the env file is of the format:

SECRET_TOKEN=...
PORT=5000
DATABASE_URL=...
ENCRYPTION_PRIMARY_KEY=...
ENCRYPTION_DETERMINISTIC_KEY=...
SECRET_KEY_BASE=...
ENCRYPTION_KEY_DERIVATION_SALT=...

For the Docker image building in CI, I was going to work on adding it to CI + making it build all three of amd64/arm/arm64. If you're more familiar with CircleCI though, feel free to take that on :) I have #1053 open to make the Dockerfile accept other architectures, ~though I have yet to test those changes~ they are tested now!

mockdeep commented 1 year ago

@guidopetri thanks for the updates! I fixed one bundler issue, but now it's giving an error while installing gems:

    can't create Thread: Operation not permitted
    /usr/local/lib/ruby/3.2.0/timeout.rb:101:in `initialize'
    /usr/local/lib/ruby/3.2.0/timeout.rb:101:in `new'
    /usr/local/lib/ruby/3.2.0/timeout.rb:101:in `create_timeout_thread'
    /usr/local/lib/ruby/3.2.0/timeout.rb:134:in `block in ensure_timeout_thread_created'
    /usr/local/lib/ruby/3.2.0/timeout.rb:132:in `synchronize'
    /usr/local/lib/ruby/3.2.0/timeout.rb:132:in `ensure_timeout_thread_created'
    /usr/local/lib/ruby/3.2.0/timeout.rb:181:in `timeout'
    /usr/local/lib/ruby/3.2.0/net/http.rb:1269:in `connect'
    /usr/local/lib/ruby/3.2.0/net/http.rb:1248:in `do_start'
    /usr/local/lib/ruby/3.2.0/net/http.rb:1243:in `start'
    /usr/local/lib/ruby/3.2.0/rubygems/request/http_pool.rb:43:in `setup_connection'
    /usr/local/lib/ruby/3.2.0/rubygems/request/https_pool.rb:7:in `setup_connection'
    /usr/local/lib/ruby/3.2.0/rubygems/request/http_pool.rb:39:in `make_connection'
    /usr/local/lib/ruby/3.2.0/rubygems/request/http_pool.rb:20:in `checkout'
    /usr/local/lib/ruby/3.2.0/rubygems/request.rb:129:in `connection_for'
    /usr/local/lib/ruby/3.2.0/rubygems/request.rb:188:in `perform_request'
    /usr/local/lib/ruby/3.2.0/rubygems/request.rb:154:in `fetch'
    /usr/local/lib/ruby/3.2.0/rubygems/remote_fetcher.rb:309:in `request'
    /usr/local/lib/ruby/3.2.0/rubygems/remote_fetcher.rb:209:in `fetch_http'
    /usr/local/lib/ruby/3.2.0/rubygems/remote_fetcher.rb:248:in `fetch_path'
    /usr/local/lib/ruby/3.2.0/rubygems/source.rb:88:in `dependency_resolver_set'
    /usr/local/lib/ruby/3.2.0/rubygems/resolver/best_set.rb:23:in `block in pick_sets'
    /usr/local/lib/ruby/3.2.0/rubygems/source_list.rb:94:in `each'
    /usr/local/lib/ruby/3.2.0/rubygems/source_list.rb:94:in `each_source'
    /usr/local/lib/ruby/3.2.0/rubygems/resolver/best_set.rb:22:in `pick_sets'
    /usr/local/lib/ruby/3.2.0/rubygems/resolver/best_set.rb:28:in `find_all'
    /usr/local/lib/ruby/3.2.0/rubygems/resolver/installer_set.rb:170:in `find_all'
    /usr/local/lib/ruby/3.2.0/rubygems/resolver/installer_set.rb:61:in `add_always_install'
    /usr/local/lib/ruby/3.2.0/rubygems/dependency_installer.rb:322:in `resolve_dependencies'
    /usr/local/lib/ruby/3.2.0/rubygems/commands/install_command.rb:205:in `install_gem'
    /usr/local/lib/ruby/3.2.0/rubygems/commands/install_command.rb:230:in `block in install_gems'
    /usr/local/lib/ruby/3.2.0/rubygems/commands/install_command.rb:223:in `each'
    /usr/local/lib/ruby/3.2.0/rubygems/commands/install_command.rb:223:in `install_gems'
    /usr/local/lib/ruby/3.2.0/rubygems/commands/install_command.rb:169:in `execute'
    /usr/local/lib/ruby/3.2.0/rubygems/command.rb:327:in `invoke_with_build_args'
    /usr/local/lib/ruby/3.2.0/rubygems/command_manager.rb:252:in `invoke_command'
    /usr/local/lib/ruby/3.2.0/rubygems/command_manager.rb:192:in `process_args'
    /usr/local/lib/ruby/3.2.0/rubygems/command_manager.rb:150:in `run'
    /usr/local/lib/ruby/3.2.0/rubygems/gem_runner.rb:51:in `run'
    /usr/local/bin/gem:10:in `<main>'
The command '/bin/sh -c gem install bundler:$BUNDLER_VERSION && bundle install' returned a non-zero code: 1

Exited with code exit status 1

CircleCI received exit code 1

I don't have a clear sense of what to do about that. Any ideas?

mockdeep commented 1 year ago

I opened an issue on the CircleCI Docker orb.

guidopetri commented 1 year ago

Sorry for the delay here, I try not to do a lot of open source during the week :) Unfortunately I have also got no idea where that error is coming from. I can investigate a bit more closely this weekend or next perhaps?

If the faulty build is blocking main, I think we can revert it and add it back in later, too.

mockdeep commented 1 year ago

@guidopetri no hurry, It's not blocking merges or anything, though it would be nice to have a passing main branch. I may open a support request with CircleCI, since the build runs fine locally.

guidopetri commented 1 year ago

This might be a bundler problem, not a CircleCI problem: https://github.com/rubygems/rubygems/issues/4353 , https://github.com/rubygems/bundler/issues/4367 maybe we need to set the number of threads to 1? I'm not sure how to do that, though.

guidopetri commented 1 year ago

@mockdeep did you end up opening a support request with CircleCI? I can try and carve out some time to find a solution if not.

mockdeep commented 1 year ago

@guidopetri so far I've opened an issue on the orb, as well as opening a discussion on their forum, but no response on either. I was starting to think maybe we could move to using Github Actions for the docker image instead. It would be nice to have a build step for each PR, and only publish on main. That way we can test the images before merging them.

guidopetri commented 1 year ago

Ok, I can give that a shot. Would you want to remove the circleCI testing or keep both circleCI and gh actions?

mockdeep commented 1 year ago

@guidopetri great thanks! I think for now we can focus on getting the docker image on GA and leave the tests on CircleCI.

stringer-rss / stringer

Docker fixes - new compose (podman support also) #1051

Repo Linking

Compose Tweaks

Container suggestions