search

martinthomson / i-d-template

A template for IETF internet draft git repositories
Other
204 stars 180 forks source link

Updates to Gemfile handling breaks qlog #433

Closed rmarx closed 4 weeks ago

rmarx commented 1 month ago

Hello there,

Seems like the changes in https://github.com/martinthomson/i-d-template/commit/e303013d6c68d43cf9a2432b010bf6052123b4e2 (and related commits) to solve https://github.com/martinthomson/i-d-template/issues/428 broke the qlog github actions autobuild (https://github.com/quicwg/qlog, https://github.com/quicwg/qlog/actions/runs/9446234091/job/26018995910#step:6:12) :)

We have a simple custom Gemfile needed to install the cddl Gem (https://github.com/quicwg/qlog/blob/main/Gemfile). We made some changes to our Makefile to incorporate that (https://github.com/quicwg/qlog/blob/main/Makefile#L26). It's automatically picked up by this project's deps.mk and run as part of the github action as expected (e.g., https://github.com/quicwg/qlog/actions/runs/9446234091/job/26018995910#step:6:12)

However, since the changes it seems something is preventing the Ruby bundler from picking up the native dependencies/path/something... Some representative failed builds with logs:

I've tried a bit to work around it by checking out a previous version of the i-d-template lib, but seems no dice (https://github.com/quicwg/qlog/actions/runs/9448120766/job/26021447700#step:6:8).

I don't really use Ruby so I have no idea what's really going wrong (or what I'm doing...), so I was hoping you might have an idea on how to fix this (either here or in qlog) :) thanks!

Example failing log:

git clone -q --depth 10  \
    -b main https://github.com/martinthomson/i-d-template lib
Created cache directory at /github/workspace/.refcache
warning: lib/default-branch.py: correcting the default branch locally:
warning: lib/default-branch.py:     git symbolic-ref refs/remotes/origin/HEAD refs/remotes/origin/main
VERSIONED="versioned" lib/build-targets.sh draft-ietf-quic-qlog-h3-events draft-ietf-quic-qlog-main-schema draft-ietf-quic-qlog-quic-events >>.targets.mk
bundle install  --gemfile=/github/workspace/Gemfile
Don't run Bundler as root. Installing your bundle as root will break this
application for all non-root users on this machine.
Fetching gem metadata from https://rubygems.org/........
Resolving dependencies...
Fetching abnftt 0.2.4
Fetching abnc 0.1.1
Fetching base45_lite 1.0.1
Fetching base32 0.3.4
Installing base45_lite 1.0.1
Installing abnc 0.1.1
Installing abnftt 0.2.4
Using bundler 2.4.15
Fetching cbor-canonical 0.1.2
Fetching cbor-deterministic 0.1.3
Fetching cbor-packed 0.1.5
Installing cbor-canonical 0.1.2
Installing cbor-deterministic 0.1.3
Fetching json_pure 2.7.2
Installing cbor-packed 0.1.5
Fetching neatjson 0.10.5
Fetching polyglot 0.3.5
Installing json_pure 2.7.2
Installing base32 0.3.4
Installing polyglot 0.3.5
Fetching colorize 1.1.0
Fetching regexp_property_values 1.5.2
Fetching scanf 1.0.0
Installing colorize 1.1.0
Installing regexp_property_values 1.5.2 with native extensions
Installing scanf 1.0.0
Fetching treetop 1.[6](https://github.com/quicwg/qlog/actions/runs/9448742325/job/26023411176#step:6:7).12
Installing treetop 1.6.12
Installing neatjson 0.10.5
Fetching cbor-diag 0.8.[7](https://github.com/quicwg/qlog/actions/runs/9448742325/job/26023411176#step:6:8)
Installing cbor-diag 0.8.7
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

current directory:
/github/workspace/.gems/ruby/3.2.0/gems/regexp_property_values-1.5.2/ext/regexp_property_values
/usr/bin/ruby extconf.rb
mkmf.rb can't find header files for ruby at /usr/lib/ruby/include/ruby.h

You might have to install separate package for the ruby development
environment, ruby-dev or ruby-devel for example.

extconf failed, exit code 1

Gem files will remain installed in
/github/workspace/.gems/ruby/3.2.0/gems/regexp_property_values-1.5.2 for
inspection.
Results logged to
/github/workspace/.gems/ruby/3.2.0/extensions/x[8](https://github.com/quicwg/qlog/actions/runs/9448742325/job/26023411176#step:6:9)6_64-linux-musl/3.2.0/regexp_property_values-1.5.2/gem_make.out

  /usr/lib/ruby/3.2.0/rubygems/ext/builder.rb:11[9](https://github.com/quicwg/qlog/actions/runs/9448742325/job/26023411176#step:6:10):in `run'
  /usr/lib/ruby/3.2.0/rubygems/ext/ext_conf_builder.rb:28:in `build'
  /usr/lib/ruby/3.2.0/rubygems/ext/builder.rb:187:in `build_extension'
  /usr/lib/ruby/3.2.0/rubygems/ext/builder.rb:221:in `block in build_extensions'
  /usr/lib/ruby/3.2.0/rubygems/ext/builder.rb:218:in `each'
  /usr/lib/ruby/3.2.0/rubygems/ext/builder.rb:218:in `build_extensions'
  /usr/lib/ruby/3.2.0/rubygems/installer.rb:846:in `build_extensions'
/usr/lib/ruby/gems/3.2.0/gems/bundler-2.4.15/lib/bundler/rubygems_gem_installer.rb:72:in
`build_extensions'
/usr/lib/ruby/gems/3.2.0/gems/bundler-2.4.15/lib/bundler/rubygems_gem_installer.rb:28:in
`install'
/usr/lib/ruby/gems/3.2.0/gems/bundler-2.4.15/lib/bundler/source/rubygems.rb:201:in
`install'
/usr/lib/ruby/gems/3.2.0/gems/bundler-2.4.15/lib/bundler/installer/gem_installer.rb:54:in
`install'
/usr/lib/ruby/gems/3.2.0/gems/bundler-2.4.15/lib/bundler/installer/gem_installer.rb:16:in
`install_from_spec'
/usr/lib/ruby/gems/3.2.0/gems/bundler-2.4.15/lib/bundler/installer/parallel_installer.rb:156:in
`do_install'
/usr/lib/ruby/gems/3.2.0/gems/bundler-2.4.15/lib/bundler/installer/parallel_installer.rb:147:in
`block in worker_pool'
/usr/lib/ruby/gems/3.2.0/gems/bundler-2.4.15/lib/bundler/worker.rb:62:in
`apply_func'
/usr/lib/ruby/gems/3.2.0/gems/bundler-2.4.15/lib/bundler/worker.rb:57:in
`block in process_queue'
/usr/lib/ruby/gems/3.2.0/gems/bundler-2.4.15/lib/bundler/worker.rb:54:in
`loop'
/usr/lib/ruby/gems/3.2.0/gems/bundler-2.4.15/lib/bundler/worker.rb:54:in
`process_queue'
/usr/lib/ruby/gems/3.2.0/gems/bundler-2.4.15/lib/bundler/worker.rb:90:in
`block (2 levels) in create_threads'

An error occurred while installing regexp_property_values (1.5.2), and Bundler
cannot continue.

In Gemfile:
  cddl was resolved to 0.[12](https://github.com/quicwg/qlog/actions/runs/9448742325/job/26023411176#step:6:13).2, which depends on
    regexp-examples was resolved to 1.6.0, which depends on
      regexp_property_values
make[1]: *** [lib/deps.mk:226: Gemfile.lock] Error 5
make[1]: Target 'latest' not remade because of errors.
make: *** [lib/main.mk:[16](https://github.com/quicwg/qlog/actions/runs/9448742325/job/26023411176#step:6:17): all] Error 2
martinthomson commented 1 month ago

Let's see what we can do with a clean bill of health. It is possible that this is coincidental, not causal. Your attempts to revert my changes would support that position (especially, since those changes are a noop in this environment).

Can you give this a go?

(My commit closed this, which was not my intent.)

martinthomson commented 1 month ago

By the way, I noticed a few things with your makefile. You shouldn't need to add "Gemfile.lock" to the "clean" target like that (I have a "clean-all" target that should clean up dependencies.

More substantively, your cddl target should name output files. As in cddl: $(addsuffix .cddl,$(drafts)), plus with a %.cddl: %.md rule. That way, your CDDL tool runs and validates each only when the draft changes. It looks like you create .cddl files as a side-effect, which isn't really a great way to operate.

rmarx commented 1 month ago

Thanks for the quick turnaround on this Martin; I know you're busy.

Firstly, I made your suggested changes to the Makefile. I agree the setup is a bit ad-hoc, but none of us are Makefile enthusiasts :) but your examples helped getting something going.

However, as expected, your latest changes don't magically fix the problem, sadly :( See a run here.

I tried to dig a bit deeper, to rule out other potential causes. I already checked the versions of the CDDL gem and its dependencies yesterday. They did have some changes around the same time stuff started breaking, but manually reverting the CDDL gem (and dependencies) to older versions didn't help (I also tried this yesterday before creating this issue).

Next, I thought it might be the ruby version. We have another separate action that runs just the CDDL logic which had the Ruby version hard set at 2.6. I updated that to 3.3.0 (which the template currently uses), but the other action just works with that, so it's not the ruby version...

Finally, I tried manually installing the packages it complains about (ruby-dev ruby-devel) even though I think it's more a case that bundler can't find the deps, not that they're not installed. This seems somewhat confirmed by the fact that ruby-dev already seems present at the latest version and the errors continue.

So yeah... at this point I'm struggling to see a path forward. Conceptually, we don't HAVE to run the CDDL in the i-d-template action, since we have the separate cddl_validate action for that. However, we DO want to run it locally as part of the makefile (to catch errors before commit). I can't seem to see an easy way to keep the Gemfile around just locally and tell the i-d-template not to use it in CI (I can disable running the actual cddl check in CI/Makefile, but that doesn't help since the error is at install time).

Any further suggestions for workarounds (or potential fix of course) is appreciated :) Thanks again!

martinthomson commented 1 month ago

OK, I have a new theory. It has nothing to do with the changes I made (though more work there is probably a good thing).

The problem is that the docker image that is in use does not support the installation of native Ruby gems. That is, gems that need to be compiled with a C compiler at installation time. The image is small and does not include a C compiler. Note that this is not the same docker image as the one that GitHub provides.

What you are seeing is coincidence, not causation. The gem changed to include a new dependency that needs to be compiled, as opposed to being pure ruby. As a result, the installation fails. It might be possible to find an alternative, disable some features, or something. The gem used to work, after all. However, like you, I only have a passing familiarity with Ruby.

cabo commented 4 weeks ago

Since May 20, cddl dependency "regexp-examples" seems to require "regexp_property_values", which requires a native build. Would this work in the math docker image?

cabo commented 4 weeks ago

I have pushed a cddl 0.12.4 that pins regexp-examples at the previous revision 1.5.1. Please try again.

rmarx commented 4 weeks ago

Hello both,

Thanks for the support on this! I first tried to confirm @martinthomson's new theory about it just being the native extension by itself by punting to quite old versions of cddl and regexp-examples and that indeed seems to have worked.

I then saw the new cddl version from @cabo and tried that as well, and that indeed also fixes the problem it seems.

I thought I had checked for this being the problem before even opening this issue by pinning the versions, but I apparently pinned just cddl (which still pulled in the latest regexp-examples) or cddl + regexp_property_values (the latter being the native extension, so if Martin is correct, that would indeed still fail...)

Happy that this is fixed and sorry for the wild goose chase @martinthomson... it was indeed an unhappy coincidence of multiple factors.

P.S. now that I have you here @cabo: is the source for the cddl gem open somewhere? rubygems.org links to https://github.com/cabo/cddl, but that gives a 404...

cabo commented 4 weeks ago

P.S. now that I have you here @cabo: is the source for the cddl gem open somewhere? rubygems.org links to https://github.com/cabo/cddl, but that gives a 404...

The new tool, cddlc, is on github. The legacy tool cddl has too much technical debt that I would want people to read it (it is open source, but I'm giving you an extra nut to chew). I'm moving (and refactoring) code from cddl to cddlc, but neither generator nor validator are complete yet.