search

metanorma / metanorma-cli

CLI (Command Line Interface) for Metanorma
BSD 2-Clause "Simplified" License
5 stars 5 forks source link

(URGENT) Stabilize the release process #174

Closed ronaldtse closed 3 years ago

ronaldtse commented 4 years ago

A failure in the toolchain has tremendous impact on downstream users, especially for those who integrate metanorma into their custom build processes. This is seen in #172 and #173 reported by multiple parties.

  1. We must use PRs for ALL updates to ensure all tests pass. If the tests do not pass due to external failures, we do not merge until those are fixed.

  2. We need an extra staging gate prior to official releases. The issue in #173 was due to tests passing but the code was not actually tested outside RSpec (i.e development mode). We should use the RubyGems PRE stage to release gems in pre-release, create pre-release executables in metanorma-docker / packed-mn, have them tested, before going full release. Moreover, the false test failures (e.g. IETF site down, RubyGems install failure) have lowered our guard for real failures. We MUST NOT RELEASE IN ANY FAILURE. We should either fix those tests, stub those network requests for testing purposes so that they never mask any real failure behind. For example, the solution for the IETF site issue with fetching workgroups, is probably to cache their list in a new GitHub repository, or in a static file in the metanorma-ietf gem.

  3. We do not currently have a dashboard indicating all the green lights for dependencies used for release. For example, the latexmath gem caused #173, but the build in latexmath was failing. We cannot use a failing dependency for a new Metanorma release (it will be packaged inside docker and packed-mn!!)

  4. ALWAYS do releases in the beginning of week, and beginning of day. This gives us time to fix it during the day and whatever time necessary. NEVER do releases at the end of week, especially at the end of day of that.

Thoughts please.

cc: @petithug @Kritner

ronaldtse commented 4 years ago

Related to:

CAMOBAP commented 4 years ago

To be on the same page let me describe what happens for release 1.3.7.2

Preface

What happens with v1.3.7.2 release

What happens with v1.3.7.3 release

Proposals

As we will have issues with our release flow, my proposal:

We still will have a problem with tags for which tests were failed, but we will have no pushes for rubygems.org, if tests are failed

Let me know how do you think?

w00lf commented 4 years ago

We can use a set of integration tests for metanorma-cli and run them only when there is a new release being created(with github actions if control flow). For network manipulations we can use something like https://github.com/vcr/vcr, I have been using it a lot for international testing with 3-d party api-s, or we can use dependency injection with test adapter. Both options have their caveats, with vcr we need to ensure that format of 3-d party api-s wont change in the future and if it did we need to change our stabs.

ronaldtse commented 4 years ago

I agree that network requests for specs should be VCR'ed. We can't allow network failures to break specs. This probably means that in rspec we will need to have two separate builds:

  1. Normal specs are run using vcr cassettes
  2. A "switched" spec that verifies that the vcr cassette is up to date, perhaps running separately every day.
andrew2net commented 4 years ago

2. A "switched" spec that verifies that the vcr cassette is up to date, perhaps running separately every day.

@ronaldtse the VCR has a setting which allows us to define a rerecording interval. Once the interval is expired VCR vill refresh all the cassettes.

opoudjis commented 4 years ago

@ronaldtse the VCR has a setting which allows us to define a rerecording interval. Once the interval is expired VCR vill refresh all the cassettes.

 VCR.use_cassette('example', :re_record_interval => 7.days, :match_requests_on => [:method, :host, :path]) do

A "switched" spec that verifies that the vcr cassette is up to date, perhaps running separately every day.

You mean expire every week, right? Every day is going to end up being the same problem this is supposed to fix.

opoudjis commented 4 years ago

Have put 7 day expiration in all metanorma gems using VCR; thank you, @andrew2net!

opoudjis commented 4 years ago

I really don't want to work with prerelease gems locally if they will make life difficult for me to do local testing (and they do), and would suggest I hardcode minimum gem versions in metanorma-cli instead, and we only have metanorma-cli be prerelease

CAMOBAP commented 3 years ago

Quick update on this

I think we have some progress on stabilization direction:

The bottleneck now is packed-mn (at the current moment https://github.com/metanorma/packed-mn/issues/83).

@ronaldtse @opoudjis could you please share your feedback, is anything else I can do in the scope of this task?

ronaldtse commented 3 years ago

I believe we can close this now? The release process has been improved.

We just need to confirm that the final deliverables are properly triggered on updates and that they can be built/fixed independently:

Ping @CAMOBAP

CAMOBAP commented 3 years ago

@ronaldtse yep, I can confirm