search

exercism / development-environment

16 stars 19 forks source link

Improve whole docker compose stack and add custom stack builder #8

Closed joshgoebel closed 3 years ago

joshgoebel commented 3 years ago

First:

Checklist for the whole larger Docker niceness project:

Docker Hub:

Tooling enhancements (shell/tests) for:

Runners:

Some of this runner stuff may need to move into a new issue and remain open...

Josh:

joshgoebel commented 3 years ago

Builds on the work in https://github.com/exercism/v3-docker-compose/pull/7

joshgoebel commented 3 years ago

The naming here has always bugged me. Assuming we're stuck with two (one for production) and another that builds on top of it to give it a web hook I propose:

// before => after
javascript_test_runner_base => javascript_test_runner          (Dockerfile)
javascript_test_runner      => javascript_test_runner_webhook  (dev.Dockerfile)

@iHiD Thoughts? So you have the runner itself, then the runner and webhook.

iHiD commented 3 years ago

@iHiD Thoughts? So you have the runner itself, then the runner and webhook.

I think that we should do this slightly differently in general. I think

joshgoebel commented 3 years ago

Ok, I completely agree (and then we'd just have a single "service", not a dummy one that starts and fails) but where does this appending/updating happen? It can't be JUST inside a GHA, the actual file needs to be correct in the repository incase it's needed to be used for a local build as well.

Ok we OK to start by just appending the files manually (to simplify everything else) with a comment marker and then figuring out how to do the auto-update part later?

joshgoebel commented 3 years ago

@iHiD Actually that doesn't get us past the naming issue. There are really TWO runners a person might interact with during development (I've used both myself already):

It's wholly reasonable to hack on the code and then run use ./run-in-docker to test it... but for the WHOLE stack you'd of course want the web wrapped version.

This means someone doing work on test runners likely has GOOD reasons to use BOTH Dockerfile and dev.Dockerfile... I can't think of why we'd want to bloat production with a bunch of Ruby runtimes that aren't needed... so that means we need two image names.

I suppose we could do this though:

Thoughts?

iHiD commented 3 years ago

So I think the Javascript Test Runner should have a dev.Dockerfile which containers the Dockerfile with the extra ruby server stuff, and that the image it produces should be suffixed with -server, so for example:

exercism/javascript-test-runner-server

This is what would appear in the docker-compose.

The normal Dockerfile doesn't need to go into the docker-compose and I don't think it needs to go on Dockerhub, but if it does it should be:

exercism/javascript-test-runner
iHiD commented 3 years ago

Looking great (and close)!!

Two thoughts below:

  1. Can we add a config setting for daemon, which is set to true by default, but if false just doesn't use the -d flag? I would choose to run this not in daemon mode so that the logs stream in the console.
  1. With reference to the screenshot below, can we check the exit status of the docker-compose up to check whether to display the "It may take just a moment for everything to spool up." message?
Screenshot 2020-08-09 at 00 41 56
joshgoebel commented 3 years ago

exercism/javascript-test-runner

Don't love it but also not super attached to my ideas. So yeah it's quite likely that someone use dev would not even have runners at all, only runner-servers.

joshgoebel commented 3 years ago

Can we add a config setting for daemon, which is set to true by default, but if false just doesn't use the -d flag? I would choose to run this not in daemon mode so that the logs stream in the console.

Not opposed, but is that a feature request rather than something that should hold up merging? :-) you can always docker-compose tail the logs.

iHiD commented 3 years ago

is that a feature request rather than something that should hold up merging? :-)

Good question. Probably a feature request, yes.

iHiD commented 3 years ago

@joshgoebel I get an issue building the setup image locally.

Running

docker build -t exercism/setup --build-arg exercism_config_version=0.38.0 -f setup/dev.Dockerfile ./setup

Gives me:

...

Successfully installed bundler-2.1.4
1 gem installed
Fetching gem metadata from https://rubygems.org/.........
Fetching gem metadata from https://rubygems.org/.
Resolving dependencies...
Using rake 12.3.3
Fetching ast 2.4.1
Installing ast 2.4.1
Fetching aws-eventstream 1.1.0
Installing aws-eventstream 1.1.0
Fetching aws-partitions 1.354.0
Installing aws-partitions 1.354.0
Fetching aws-sigv4 1.2.1
Installing aws-sigv4 1.2.1
Fetching jmespath 1.4.0
Installing jmespath 1.4.0
Fetching aws-sdk-core 3.104.3
Installing aws-sdk-core 3.104.3
Fetching aws-sdk-dynamodb 1.51.0
Installing aws-sdk-dynamodb 1.51.0
Fetching aws-sdk-kms 1.36.0
Installing aws-sdk-kms 1.36.0
Fetching aws-sdk-s3 1.78.0
Installing aws-sdk-s3 1.78.0
Using bundler 2.1.4
Fetching docile 1.3.2
Installing docile 1.3.2
Fetching mandate 0.3.0
Installing mandate 0.3.0
Fetching zeitwerk 2.4.0
Installing zeitwerk 2.4.0
Using exercism-config 0.38.0 from source at `.`
Fetching gem-release 2.1.1
Installing gem-release 2.1.1
Fetching json 2.3.1
Installing json 2.3.1 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /usr/local/bundle/gems/json-2.3.1/ext/json/ext/generator
/usr/local/bin/ruby -I /usr/local/lib/ruby/2.6.0 -r
./siteconf20200811-1-12aj39r.rb extconf.rb
creating Makefile

current directory: /usr/local/bundle/gems/json-2.3.1/ext/json/ext/generator
make "DESTDIR=" clean
sh: make: not found

current directory: /usr/local/bundle/gems/json-2.3.1/ext/json/ext/generator
make "DESTDIR="
sh: make: not found

make failed, exit code 127

Gem files will remain installed in /usr/local/bundle/gems/json-2.3.1 for
inspection.
Results logged to
/usr/local/bundle/extensions/x86_64-linux/2.6.0/json-2.3.1/gem_make.out

An error occurred while installing json (2.3.1), and Bundler cannot continue.
Make sure that `gem install json -v '2.3.1' --source 'https://rubygems.org/'`
succeeds before bundling.

In Gemfile:
  simplecov was resolved to 0.17.1, which depends on
    json
The command '/bin/sh -c gem install bundler:${bundler_version} &&     bundle install' returned a non-zero code: 5
iHiD commented 3 years ago

Amazing work, @joshgoebel :D