search

thoughtbot / appraisal

A Ruby library for testing your library against different versions of dependencies.
https://thoughtbot.com
MIT License
1.25k stars 107 forks source link

Appraisal v3 #204

Open nickcharlton opened 1 year ago

nickcharlton commented 1 year ago

It's been a while since the last release and I know folks are keen for a new one. I'm currently tracking/working on the following, some of which are much easier to fix than others but this issue should communicate where we are.

It's going to be v3 because it's going to change dependency requirements on Ruby and Bundler. But I don't know how much just yet.

New Features

Remaining

nickcharlton commented 1 year ago

I'm currently stuck on understanding why Ruby 2.6, 2.7 and 3.2 fail. The 2.* series are the same reason, but 3.2 is close but not completely the same.

I'm also thinking of dropping 2.5 as it's so old, but with it working with very little effort I'm okay with keeping it for now.

joe-sharp commented 1 year ago

Just a suggestion Nick, I would drop support for Ruby 2.5 and 2.6. According to ruby-lang.org they are both past EOL: https://www.ruby-lang.org/en/downloads/branches/

nickcharlton commented 1 year ago

Yeah, I suspect I will end up doing that. It's definitely not worth too much effort, even for a library where I want to make things easier for folks.

I'm more concerned with 3.2, but if the solution for that fixes problems then I'd be happy for a little longer. I suspect 3.1 or whatever in six months would then drop it.

unicornzero commented 1 year ago

Hi! My app is already on Ruby 3.2 and I need to use dd-trace-rb for monitoring, but am running into an issue for which a fix appears to be blocked on needing Appraisal support for Ruby 3.2.

What kind of timeline might we be looking at for Ruby 3.2 support?

Thanks!

nickcharlton commented 1 year ago

It's hard to put a timeline to it, unfortunately. I'm dipping into when I can find a bit of time/inspiration on what might be causing the tests to fail on that CI branch I started but I can only do that in between a busy client project and management responsibilities.

So er, any thoughts are welcome on that CI branch!

nickcharlton commented 1 year ago

I've started looking at the deprecation warnings, with the hope that this either solves our CI problem or indicates what it might be, starting with:

Maimer commented 1 year ago

I have been trying to get the tests passing on CI and the more I look at it the more I think it's an issue of what bundler version is being tested against and less what Ruby version is being tested against. Unfortunately bundler doesn't follow semantic versioning and as of recently there have been lots of releases. It almost feels like the test matrix should be both Ruby version and bundler version, but it's hard to know which ones to test. Should the appraisal gem be updated to support a minimum bundler version?

nickcharlton commented 1 year ago

Yeah, I think that's exactly the problem we're facing.

I'd like to identify the lowest-possible Bundler release we can target and then start testing from that release onwards. If anyone has a good suggestion, that'd be brilliant.

I think then we should be testing major (we'll need to figure that out ourselves, I guess?) releases of Bundler in another matrix on CI.

kyrofa commented 1 year ago

Still using ruby 2.5 over here, FYI :wave: . If it's not too much trouble, I would like to see it continue to be supported.

Maimer commented 1 year ago

Yeah, I think that's exactly the problem we're facing.

I'd like to identify the lowest-possible Bundler release we can target and then start testing from that release onwards. If anyone has a good suggestion, that'd be brilliant.

I think then we should be testing major (we'll need to figure that out ourselves, I guess?) releases of Bundler in another matrix on CI.

Personally, I wouldn't start support at bundler version 2.2 and go up from there. It's hard to gauge how hard it is going to be to be able to support so many different bundler versions. Maybe we just start a matrix with each available version and see what passes and what doesn't.

Maimer commented 1 year ago

Still using ruby 2.5 over here, FYI 👋 . If it's not too much trouble, I would like to see it continue to be supported.

What version of bundler are you using with Ruby 2.5?

nickcharlton commented 1 year ago

@kyrofa, I think a tool like Appraisal should support as wide a set of Ruby versions as possible so unless it becomes clearly prohibitive to keep supporting, 2.5 will stay.

Of course, Ruby 2.5 is one of the ones which is working fine with Bundler 2.3.10 so so far, that'll stay!

kyrofa commented 1 year ago

What version of bundler are you using with Ruby 2.5?

We like cutting edge technology, so v1.17.3 of course. cough

joe-sharp commented 1 year ago

@nickcharlton are there any tasks I can try to take off your list? I have some free time this weekend and I would love to help get this release out if there is anything I can do to help! 😄

kyrofa commented 1 year ago

Not to speak out of turn, but @joe-sharp if you have some time it sounds like he could use some support on the github actions side of things: https://github.com/thoughtbot/appraisal/pull/206 .

Maimer commented 1 year ago

Any reason to use Github Actions over something like CircleCI?

kyrofa commented 1 year ago

Any reason to use CircleCI over GitHub Actions? Feels like the in-house, integrated solution should be the default unless there's a good reason to use something else.

nickcharlton commented 1 year ago

@kyrofa is right — it's GitHub Actions/CI that's the main problem right now.

That said, anything you could do to help with the deprecation warnings would be excellent, too. I think that'd be great to launch a new version with no warnings at all for recent Bundler versions.

Finally, knowing which Bundler versions we should support (for Ruby 2.5 and upwards) would be very helpful too.

Maimer commented 1 year ago

Any reason to use CircleCI over GitHub Actions? Feels like the in-house, integrated solution should be the default unless there's a good reason to use something else.

I find CircleCI to be a bit more comprehensive in terms of what it offers (probably because it's a company solely focused on CI), but I will admit my experience with Github Action has been limited. Also, CircleCI is free for open source projects in case that was concern. However, all that matters is we get a working CI pipeline that tests across all versions of ruby and bundler that need to be tested.

nickcharlton commented 1 year ago

Sorry, @Maimer, I missed your question.

I've been having a generally better time with GitHub Actions (although, I dislike the centralisation on a single service). Something I'd seen a lot on Administrate was even though we (thoughtbot) were spending a lot with them, open source builds would still sit in a long queue and worse, fail randomly.

Anyway, for Appraisal any CI is better than no CI.

Maimer commented 1 year ago

Sorry, @Maimer, I missed your question.

I've been having a generally better time with GitHub Actions (although, I dislike the centralisation on a single service). Something I'd seen a lot on Administrate was even though we (thoughtbot) were spending a lot with them, open source builds would still sit in a long queue and worse, fail randomly.

Anyway, for Appraisal any CI is better than no CI.

We use CircleCI at Appfolio (where I work) and I've had very little issue with it and no queue issues or random failures. Working with it every day is probably why I feel more comfortable with it. I have a branch I have been working on that tests a matrix of Ruby versions and Bundler versions. I have been trying to get it passing against a single Ruby version first as I really don't feel there will be any issues for different Ruby versions, but rather Bundler versions. I have it down to the same test failure for all Bundler versions and haven't had success in figuring out how to get it to pass. It's definitely a configuration setup and not an issue with this gem. You, or anyone else, is welcome to take a look at my changes in this branch and see if you have any ideas: https://github.com/Maimer/appraisal/tree/try_circleci (Changes: https://github.com/thoughtbot/appraisal/compare/main...Maimer:appraisal:try_circleci). As I said before, I don't believe Bundler generally follows semantic versioning, so it's hard to know which versions may have issues. So I just chose the latest version for each patch version available.

matrix:
  parameters:
    ruby_version:
      - '3.2.1'
      - '3.1.2'
      - '3.0.5'
      - '2.7.6'
      - '2.6.9'
      - '2.5.9'
    bundler_version:
      - '2.4.7'
      - '2.3.26'
      - '2.2.34'
      - '2.1.4'
      - '1.17.3'

However, there is definitely a unique test failure for Bundler version 2.4.7 that may require a change in the appraisal gem to fix. Here is the latest CI run where you can see the failures: https://app.circleci.com/pipelines/github/Maimer/appraisal/55/workflows/b48d6ca2-d165-407d-a184-6f1d06ac3d07. The main failure is around the tests not being able to find bundler when the --path option is used as it sets the path configuration for bundler when that flag is used.

javierav commented 1 year ago

Any avance on v3 release?

nickcharlton commented 12 months ago

Alas, no.

I've not managed to find any time to go back to investigating the outstanding problems — which is partly been because of some more intense client project time recently and also because I bought a house and that's like having a second job for six months, apparently.

Whilst I'm through with both of those now, which is great for me, I'm wary of setting any expectations otherwise right now. I'll make it obvious on either this issue or others when I'm able to make time though!

nickcharlton commented 11 months ago

I've just released v2.5.0, which at least ships everything done so far and should fix a lot of the recent bug reports.

kyrofa commented 11 months ago

Great idea @nickcharlton, thank you for that.

luke-hill commented 4 months ago

Any updates here, we're pinning ourselves on this and some of the functionality regarding the customize gemfiles also isn't working for later rubies

nickcharlton commented 4 months ago

Last week I was getting back into it (now that I feel I just about put out the fire that's been Administrate's asset problems), starting off with trying to get GHA working on an old branch I had to refresh my memory of problems and attempting to get my head around some Bundler deprecations and incompatibilities. It looks like I'm going to have more time this week too, which is great.

My main focus is figuring out what on earth is going on with this code base and changes in Bundler, and then coming up with a plan to handle it better. It's been rough recently.

@luke-hill, is this the issue you're facing, or something different? If it's different, could you open a/some new issues so we can track them?

luke-hill commented 4 months ago

Sorry, it's a new issue. I think it's partially related to some things that you may have already tackled. So what I'll do is come back full circle in a couple of weeks and try using the latest versions.

FYI the issues I encountered are in CI runs in cucumber-rails somewhere but I can't spot them. So I think what is best is I wait until you've cut a v3 and then try use it on our matrix.

I think the error is closer to the ERR::ENOENT error on ruby 3.2 if memory serves