search

InnerSourceCommons / archive.innersourcecommons.org

The old version of the ISC site
Creative Commons Attribution Share Alike 4.0 International
41 stars 29 forks source link

Development environment setup with docker #91

Open dellagustin opened 4 years ago

dellagustin commented 4 years ago

I have recently forked and cloned the repository. As I like keeping a clean host, I will try to create a Dockerfile for easily setting up the environment without needing to install all the dependencies on the host.

My initial goal is to have a docker build followed by a docker run that will build the page, as described on https://github.com/InnerSourceCommons/innersourcecommons.org/blob/3f99fb032d47c93513c08c629a9a5e23e6da3632/CONTRIBUTING.md,

Stretch goals:

I am still investigating, so I will not set a definition of done yet, but eventually the stretch goals will be documented on separate tasks.

Acceptance Criteria

dellagustin commented 4 years ago

References

dellagustin commented 4 years ago

Error during tests with 

FROM ruby:2.7.0-alpine3.11

WORKDIR /InnerSource
ADD Gemfile .
ADD Gemfile.lock .

RUN bundle install
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /usr/local/bundle/gems/eventmachine-1.2.7/ext
/usr/local/bin/ruby -I /usr/local/lib/ruby/2.7.0 -r
./siteconf20200217-1-9pc7kt.rb extconf.rb
checking for -lcrypto... *** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
        --with-opt-dir
        --without-opt-dir
        --with-opt-include
        --without-opt-include=${opt-dir}/include
        --with-opt-lib
        --without-opt-lib=${opt-dir}/lib
        --with-make-prog
        --without-make-prog
        --srcdir=.
        --curdir
        --ruby=/usr/local/bin/$(RUBY_BASE_NAME)
        --with-ssl-dir
        --without-ssl-dir
        --with-ssl-include
        --without-ssl-include=${ssl-dir}/include
        --with-ssl-lib
        --without-ssl-lib=${ssl-dir}/lib
        --with-openssl-config
        --without-openssl-config
        --with-pkg-config
        --without-pkg-config
        --with-crypto-dir
        --without-crypto-dir
        --with-crypto-include
        --without-crypto-include=${crypto-dir}/include
        --with-crypto-lib
        --without-crypto-lib=${crypto-dir}/lib
        --with-cryptolib
        --without-cryptolib
/usr/local/lib/ruby/2.7.0/mkmf.rb:471:in `try_do': The compiler failed to
generate an executable file. (RuntimeError)
You have to install development tools first.
        from /usr/local/lib/ruby/2.7.0/mkmf.rb:564:in `try_link0'
        from /usr/local/lib/ruby/2.7.0/mkmf.rb:582:in `try_link'
        from /usr/local/lib/ruby/2.7.0/mkmf.rb:801:in `try_func'
        from /usr/local/lib/ruby/2.7.0/mkmf.rb:1029:in `block in have_library'
        from /usr/local/lib/ruby/2.7.0/mkmf.rb:971:in `block in checking_for'
        from /usr/local/lib/ruby/2.7.0/mkmf.rb:361:in `block (2 levels) in postpone'     
        from /usr/local/lib/ruby/2.7.0/mkmf.rb:331:in `open'
        from /usr/local/lib/ruby/2.7.0/mkmf.rb:361:in `block in postpone'
        from /usr/local/lib/ruby/2.7.0/mkmf.rb:331:in `open'
        from /usr/local/lib/ruby/2.7.0/mkmf.rb:357:in `postpone'
        from /usr/local/lib/ruby/2.7.0/mkmf.rb:970:in `checking_for'
        from /usr/local/lib/ruby/2.7.0/mkmf.rb:1024:in `have_library'
        from extconf.rb:8:in `block in check_libs'
        from extconf.rb:8:in `all?'
        from extconf.rb:8:in `check_libs'
        from extconf.rb:95:in `<main>'

To see why this extension failed to compile, please check the mkmf.log which can
be found here:

/usr/local/bundle/extensions/x86_64-linux-musl/2.7.0/eventmachine-1.2.7/mkmf.log

extconf failed, exit code 1
Gem files will remain installed in /usr/local/bundle/gems/eventmachine-1.2.7 for
inspection.
Results logged to
/usr/local/bundle/extensions/x86_64-linux-musl/2.7.0/eventmachine-1.2.7/gem_make.out     

An error occurred while installing eventmachine (1.2.7), and Bundler cannot
continue.
Make sure that `gem install eventmachine -v '1.2.7' --source
'https://rubygems.org/'` succeeds before bundling.

In Gemfile:
  jekyll-redirect-from was resolved to 0.16.0, which depends on
    jekyll was resolved to 4.0.0, which depends on
      em-websocket was resolved to 0.5.1, which depends on
        eventmachine

Solved based on https://github.com/docker-library/ruby/issues/163#issuecomment-335522378

rrrutledge commented 4 years ago

This is fantastic stuff, @dellagustin. We were having need to uplevel some of our tech game! Looking forward to seeing it develop!

dellagustin commented 4 years ago

When trying to serve the page from the docker container with command docker run --rm -it -v <host path of the page>:/source <image>, and the dockerfile having the command CMD jekyll serve --config /source/_config.yml,/source/_config_dev.yml -s /source -d /dist, I get the error below:

  Conversion error: Jekyll::Converters::Scss encountered an error while converting 'assets/css/atom.scss':
                    undefined method `to_sym' for true:TrueClass Did you mean? to_s
                    ------------------------------------------------
      Jekyll 4.0.0   Please append `--trace` to the `serve` command 
                     for any additional information or backtrace. 
                    ------------------------------------------------
Traceback (most recent call last):
        30: from /usr/local/bundle/bin/jekyll:23:in `<main>'
        29: from /usr/local/bundle/bin/jekyll:23:in `load'
        28: from /usr/local/bundle/gems/jekyll-4.0.0/exe/jekyll:15:in `<top (required)>'
        27: from /usr/local/bundle/gems/mercenary-0.3.6/lib/mercenary.rb:19:in `program'
        26: from /usr/local/bundle/gems/mercenary-0.3.6/lib/mercenary/program.rb:42:in `go'
        25: from /usr/local/bundle/gems/mercenary-0.3.6/lib/mercenary/command.rb:220:in `execute'
        24: from /usr/local/bundle/gems/mercenary-0.3.6/lib/mercenary/command.rb:220:in `each'
        23: from /usr/local/bundle/gems/mercenary-0.3.6/lib/mercenary/command.rb:220:in `block in execute'
        22: from /usr/local/bundle/gems/jekyll-4.0.0/lib/jekyll/commands/serve.rb:86:in `block (2 levels) in init_with_program'
        21: from /usr/local/bundle/gems/jekyll-4.0.0/lib/jekyll/command.rb:89:in `process_with_graceful_fail'
        20: from /usr/local/bundle/gems/jekyll-4.0.0/lib/jekyll/command.rb:89:in `each'
        19: from /usr/local/bundle/gems/jekyll-4.0.0/lib/jekyll/command.rb:89:in `block in process_with_graceful_fail'
        18: from /usr/local/bundle/gems/jekyll-4.0.0/lib/jekyll/commands/build.rb:36:in `process'
        17: from /usr/local/bundle/gems/jekyll-4.0.0/lib/jekyll/commands/build.rb:65:in `build'
        16: from /usr/local/bundle/gems/jekyll-4.0.0/lib/jekyll/command.rb:28:in `process_site'
        15: from /usr/local/bundle/gems/jekyll-4.0.0/lib/jekyll/site.rb:76:in `process'
        14: from /usr/local/bundle/gems/jekyll-4.0.0/lib/jekyll/site.rb:202:in `render'
        13: from /usr/local/bundle/gems/jekyll-4.0.0/lib/jekyll/site.rb:515:in `render_pages'
        12: from /usr/local/bundle/gems/jekyll-4.0.0/lib/jekyll/site.rb:515:in `each'
        11: from /usr/local/bundle/gems/jekyll-4.0.0/lib/jekyll/site.rb:516:in `block in render_pages'
        10: from /usr/local/bundle/gems/jekyll-4.0.0/lib/jekyll/site.rb:523:in `render_regenerated'
         9: from /usr/local/bundle/gems/jekyll-4.0.0/lib/jekyll/renderer.rb:63:in `run'
         8: from /usr/local/bundle/gems/jekyll-4.0.0/lib/jekyll/renderer.rb:84:in `render_document'
         7: from /usr/local/bundle/gems/jekyll-4.0.0/lib/jekyll/renderer.rb:100:in `convert'
         6: from /usr/local/bundle/gems/jekyll-4.0.0/lib/jekyll/renderer.rb:100:in `reduce'
         5: from /usr/local/bundle/gems/jekyll-4.0.0/lib/jekyll/renderer.rb:100:in `each'
         4: from /usr/local/bundle/gems/jekyll-4.0.0/lib/jekyll/renderer.rb:102:in `block in convert'
         3: from /usr/local/bundle/gems/jekyll-sass-converter-2.0.1/lib/jekyll/converters/scss.rb:180:in `convert'
         2: from /usr/local/bundle/gems/jekyll-sass-converter-2.0.1/lib/jekyll/converters/scss.rb:173:in `sass_configs'
         1: from /usr/local/bundle/gems/jekyll-sass-converter-2.0.1/lib/jekyll/converters/scss.rb:231:in `sourcemap_required?'
/usr/local/bundle/gems/jekyll-sass-converter-2.0.1/lib/jekyll/converters/scss.rb:224:in `sourcemap_option': undefined method `to_sym' for true:TrueClass (NoMethodError)
Did you mean?  to_s

I already found some references when googling, but I need some more investigation.

rrrutledge commented 4 years ago

Thanks for this update @dellagustin.

dellagustin commented 4 years ago

I reported an issue on https://github.com/jekyll/jekyll-sass-converter: https://github.com/jekyll/jekyll-sass-converter/issues/102

I already found some references when googling, but I need some more investigation. The results I have seen before did not lead anywhere.

dellagustin commented 4 years ago

Ok, the issue with jekyll-sass-converter as actually on our side. I am surprised no one had this before, but if one includes the config file _config_dev.yml, then the error should occur.

The problem is the the property sass.sourcemap is currently set to true, and it should either be never, always or development: https://github.com/jekyll/jekyll-sass-converter/blob/b044e9e3ad9ed9ce647fab76c621aeaa89775999/README.md#configuration-options.

This is actually a bug in the existing development setup, I will create separate issue for this one.

dellagustin commented 4 years ago

Now, I got past most issues and it should be working, but I think I am facing some networking issues as I am currently using docker toolbox (poor man's version for windows 10 home ...).

I am almost installing a full fledged virtual machine. I'll troubleshoot that, and soon we should have a docker based development environment version 1.

maxcapraro commented 4 years ago

Thank you for working on this, @dellagustin. I always had my shenanigans with local builds and your work will make it significantly easier for everyone to contribute to the ISC website!

maxcapraro commented 4 years ago

One note: If you like, consider creating a "WIP" pull request. Then others can jump in with their docker expertise and maybe help making sense of the error messages :)

dellagustin-sap commented 4 years ago

@maxcapraro

One note: If you like, consider creating a "WIP" pull request. Then others can jump in with their docker expertise and maybe help making sense of the error messages :)

That would be my normal working way 😉 , but this time it was very experimental to me, so I decided to keep the WIP transparency through comments on the issue. I will test it today in a Docker for Mac installation, I am expecting it to work. If that happens, I will send a PR with the current status.

dellagustin-sap commented 4 years ago

And here is why you should take the build once, run anywhere moto of docker has to be taken with a grain of salt. I use this branch to test it here in macOS: https://github.com/dellagustin/innersourcecommons.org/tree/bug-report__converters-scss-undefined-method-to_sym I am quite sure that at least the build worked for this branch (it failed when trying to run the container, because of the issue I describe in https://github.com/InnerSourceCommons/innersourcecommons.org/issues/91#issuecomment-587935972 )

Now try to build the image in, what looks to me, the same way, I get the following error:

Step 6/7 : RUN bundle install
 ---> Running in b45682a1cbf1
/usr/local/lib/ruby/2.7.0/rubygems.rb:275:in `find_spec_for_exe': Could not find 'bundler' (1.16.1) required by your /InnerSource/Gemfile.lock. (Gem::GemNotFoundException)
To update to the latest version installed on your system, run `bundle update --bundler`.
To install the missing version, run `gem install bundler:1.16.1`
        from /usr/local/lib/ruby/2.7.0/rubygems.rb:294:in `activate_bin_path'
        from /usr/local/bin/bundle:23:in `<main>'
The command '/bin/sh -c bundle install' returned a non-zero code: 1

back to google and stack overflow... 🤔

dellagustin-sap commented 4 years ago

Looking back at how I am currently working:

  1. try, error, solve,
  2. try, error, solve,
  3. ...

I will have some trouble explaining the reasoning behind the lines on the final Dockerfile without polluting it with comments. I will try to make a very complete commit message / PR message.

While doing so, I though a different working mode could be having a feature branch + Milestone + WIP PR, at each try -> fail -> solve cycle I would have a commit explaining how I solved that particular error, and maybe represent the error as a Issue that is part of the Milestone, even if that issue does not refer to master.

Maybe over-engineering, but I think it could be worth a try for the next experimental contribution I am doing.

There is an agility tradeoff, as it would require some additional github rights to setup the milestone and assign the issues to it though.

lenucksi commented 4 years ago

Thanks for approaching this @dellagustin-sap! I've found myself in the "appreciating all the Jekyll things" phase repeatedly only to be overwhelmed by a case a radical pragmatism every time - "screw non-system-wide installation of Jekyll, let's get this done!" ;). As for the desired features you mentioned initially - I'm fully on board with wanting to have them and Docker looks like a way to go here.

However I'd like to play a bit of the "advocatus diaboli" here in the hopes of saving quite a bit of effort and maintenance in the long run for a 1 or 2 afternoon conversion effort. I assume the following as a current and desired position:

I successfully run my website on Hugo as do quite a lot of other people. Would you be open to consider the idea of attempting a Hugo conversion for the website and migration to Hugo GitHub Actions instead of the Docker route? I could also help a bit with this as I already successfully run Hugo.

dellagustin-sap commented 4 years ago

Ok, so the plot thickens...

After following the advice from the error text on my last issue (To update to the latest version installed on your system, run `bundle update --bundler`.), I added the line RUN bundle update --bundler to my Dockerfile, resulting in:

FROM ruby:2.7.0-alpine3.11

WORKDIR /InnerSource
ADD Gemfile .
ADD Gemfile.lock .

RUN apk add --no-cache g++ gcc make musl-dev
RUN bundle update --bundler
RUN bundle install

CMD jekyll serve --config /source/_config.yml,/source/_config_dev.yml -s /source -d /dist

It added a lot of build time, as it seems to be fetching and installing a lot of stuff, but it worked. This may be a symptom of a flaky dependency settings (i.e. recently released versions of dependencies break the build / serve somehow) - which creates an unstable development environment experience - something we may want to look into the futere.

Now, it builds, it serves, but I can't reach it. Differently from the problem I was having with Docker Toolbox, I am now getting the error ERR_EMPTY_RESPONSE:

image

While researching for the possible cause, I actually found something reasonable that would explain the issue I was having with the connection refused (I already suspected it was something on this direction, as I had this type of issue before). When running locally, jekyll by default only accepts connections from localhost and 127.0.0.1. As we are running on docker, it is as if we were accessing it from another machine, thus the connection is blocked. I found references that mentioned a common solution for both ERR_EMPTY_RESPONSE and ERR_CONNECTION_REFUSED, so I will try that: adding the following arguments to the jekyll commands --host=0.0.0.0, this is supposed to enable connections from outside localhost and 127.0.0.1.

References:

dellagustin-sap commented 4 years ago

Hi @lenucksi,

About https://github.com/InnerSourceCommons/innersourcecommons.org/issues/91#issuecomment-588280567

First a small note about your comment, I believe you have an issue with some links on the 3rd sub-bullet of the Jekyll topic. You may want to fix that.

Now, the discussion about which solution to use for static website generation tooling to use is, from my perspective, a valid one, but I think it deserves its own issue, how do you feel about creating a new issue and copying the content there?

About the discussion itself, I am always fond o knowing new tooling, but at the moment I can't contribute too much to it, because my knowledge on Jekyll is very sparse, and on Hugo, non-existing.

BR, Guilherme.

dellagustin-sap commented 4 years ago

Aaaand it works!

It is a bit ludicrous that a developer has, still in this day and age, to spend so much time troubleshooting just to get some comfort on his/her development setup...

All right, now I know what to do, I want to test it also on my Docker Toolbox setup and adjust the documentation.

Also, I will try to add a docker-compose.yml file to reduce the setup to a docker-compose up command. One of the things I never did, but I think is possible, is using the current dir as a volume using compose, but I am wiling to try.

I still need to double check If my stretch goal number 1 (well, the first bullet) is actually working - it should, as jekyll serve, should handle that.
The second one I don't expect to get out of the box, but let's see, it will most likely outlive this issue.

The PR will still take some time, as there is a lot of documentation to go with it, but if someone wants to try it, here is the current version of the Dockerfile:

FROM ruby:2.7.0-alpine3.11

WORKDIR /InnerSource
ADD Gemfile .
ADD Gemfile.lock .

RUN apk add --no-cache g++ gcc make musl-dev
RUN bundle update --bundler
RUN bundle install

CMD jekyll serve --host 0.0.0.0 --config /source/_config.yml,/source/_config_dev.yml -s /source -d /dist

To build and run:

  1. docker build .
  2. docker run --rm -it -v $(pwd):/source -p 4000:4000 <image name>

PS: it was so beautiful I had to take a screenshot: image

dellagustin-sap commented 4 years ago

Update on https://github.com/InnerSourceCommons/innersourcecommons.org/issues/91#issuecomment-587935972, I created a follow up issue for that: #93

I want to fix that one before sending a PR for the docker enablement, as the result of this PR would fail without a fix for #93

lenucksi commented 4 years ago

Hi @lenucksi,

About #91 (comment)

First a small note about your comment, I believe you have an issue with some links on the 3rd sub-bullet of the Jekyll topic. You may want to fix that.

Thanks for noticing, fixed it!

Now, the discussion about which solution to use for static website generation tooling to use is, from my perspective, a valid one, but I think it deserves its own issue, how do you feel about creating a new issue and copying the content there?

I entirely agree and opened #94 for it. Just wanted to field the idea here.

About the discussion itself, I am always fond o knowing new tooling, but at the moment I can't contribute too much to it, because my knowledge on Jekyll is very sparse, and on Hugo, non-existing.

On the positive side that also means that its still easy to pivot. ;) Hugo effectively feels like Jekyll, mostly some differences in file paths and config markup. Happy to help with the knowledge I have acquired so far. Maybe I'll have a bit of time to whip up some demo content.

BR, Guilherme.

BR, Johannes

dellagustin commented 4 years ago

Based on the research work I created an acceptance criteria, could someone please review it? I want to also add something about documentation.

rrrutledge commented 4 years ago

Thanks for this work, @dellagustin.

While serving the website, the solution should be able to rebuild it when the source is changed, without restarting the container

Jekyll already provides this functionality for content files. Hopefully the new solution doesn't remove this ability.

Other parts of acceptance criteria look good.

rrrutledge commented 4 years ago

Oh, and congrats on the beautiful screenshot! Looking forward to the beautiful PR!

dellagustin commented 4 years ago

Back to my Docker Toolbox, I rebuilt the image without RUN bundle update --bundler just to check, I could not see the error I mention in https://github.com/InnerSourceCommons/innersourcecommons.org/issues/91#issuecomment-588248548, but I wanted to double check.

I have no idea what is causing the discrepancy, but adding RUN bundle update --bundler does not harm the build of the image in Docker Toolbox, except perhaps for build time, but I did not measure.

Now, adding --host 0.0.0.0 improved the situation, but not by much, there is still a nasty side effect that I was not expecting - the assets are being fetched from 0.0.0.0 when I access the page on the host using the Docker Machine ip: image

That is a bit weird, it should not happen and it requires more investigation. I decided to cut my losses and focus on non-Docker Toolbox hosts for the scope of this issue. I will increment the definition of done with the task to create a follow up issue for this.

dellagustin commented 4 years ago

@rrrutledge about https://github.com/InnerSourceCommons/innersourcecommons.org/issues/91#issuecomment-588454023, I added that part because although I expect it out of the box from jekyll serve, I have so far not tested that yet, and also, I wanted to make it explicit, as it may be other users do not know this is a Jekyll feature (as I did not) - but I don't expect the solution to remove this feature.

dellagustin commented 4 years ago

Somehow, the stretch goal:

While serving the website, the solution should be able to rebuild it when the source is changed, without restarting the container

was not accomplished by #99 , although this was expected out of the box (maybe my instincts where telling me something when I documented it as stretch goal 😁 )

Update: After testing on Docker for Mac, I can see that the website is automatically updated, so this is probably a issue caused by Docker Toolbox

dellagustin commented 4 years ago

Collecting some references for toolbox enablement, and possibly some issues that could come with Docker for Windows:

dellagustin commented 4 years ago

I recorded the leftover issues for Docker Toolbox in https://github.com/InnerSourceCommons/innersourcecommons.org/issues/100

dellagustin-sap commented 4 years ago

Based on https://tonyho.net/jekyll-docker-windows-and-0-0-0-0/ (See section jekyll serve and URLs)

When jekyll serve is run with --host=0.0.0.0, Jekyll sets the host portion of the site URL to 0.0.0.0 by default, regardless of the url setting in the config. This also affects any absolute URL that Jekyll generates.

This is not ideal for the combination of Jekyll, Docker, and Windows.

Fortunately, Jekyll provides a way to override this:

If JEKYLL_ENV is any value except development (its default value), Jekyll will not overwrite the value of url in your config.

The way I read it: My PR (#99 ) will work on mac, as it understands http://0.0.0.0/ as http://localhost/, but not on windows (be it Docker for windows or Docker Toolbox)

I need to test the JEKYLL_ENV tip, this could actually solve the problems I describe on #100 and also make for setup that works in Docker for Mac, Docker for Windows and Docker Toolbox (with a few tweaks).