Help Request: Testing a separate ruby project within another

PragTob commented 4 years ago

Hello there and thanks for aruba :green_heart:

My question/scenario might be a bit weird but I'll try either way. Code ref: https://github.com/colszowka/simplecov/pull/814 (PR where I stumbled upon this problem trying to implement something)

This is a request for "How do I do this? Can I do this?" not "please change the way you're doing things"

Versions

Ruby: 2.7 Aruba: 0.14.14

Context

SimpleCov. We need to make sure SimpleCov works in many different contexts as such we have a "fake project" that we run tests in and then want to make sure the returned coverage results for running tests there are right.

To do this, we have a slightly awkward Before hook replacing the project contents with the faked project of our choice (note: I didn't write this and in general, better ways of doing this are welcome :D )

Before do
  this_dir = File.dirname(__FILE__)

  # Clean up and create blank state for fake project
  cd(".") do
    FileUtils.rm_rf "project"
    FileUtils.cp_r File.join(this_dir, "../../spec/faked_project/"), "project"
  end

  step 'I cd to "project"'
end

(adapted in the current PR to support different projects)

Problem

To the best of my knowledge/debugging aruba executes command like bundle or bundle exec parallel_rspec spec in the context of the bundle of the original app (simplecov, not our faked project). Commands are run via "When I successfully run cmd"

output captured from running the cucumber tests (from the PR):

``` tobi@speedy:~/github/simplecov(more-test-projects)$ be cucumber features/parallel_tests.feature Using the default profile... Using rake 13.0.1 Using public_suffix 4.0.2 Using addressable 2.7.0 Using mini_mime 1.0.2 Using mini_portile2 2.4.0 Using nokogiri 1.10.7 Using rack 2.0.8 Using rack-test 1.1.0 Using regexp_parser 1.6.0 Using xpath 3.2.0 Using capybara 3.30.0 Using websocket-extensions 0.1.4 Using websocket-driver 0.7.1 Using apparition 0.4.0 Using childprocess 3.0.0 Using contracts 0.16.0 Using builder 3.2.4 Using backports 3.15.0 Using cucumber-tag_expressions 1.1.1 Using gherkin 5.1.0 Using cucumber-core 3.2.1 Using cucumber-expressions 6.0.1 Using cucumber-wire 0.0.1 Using diff-lcs 1.3 Using multi_json 1.14.1 Using multi_test 0.1.2 Using cucumber 3.1.2 Using ffi 1.11.3 Using rspec-support 3.9.2 Using rspec-expectations 3.9.0 Using thor 1.0.1 Using aruba 0.14.14 Using ast 2.4.0 Using bundler 2.1.2 Using coderay 1.1.2 Using docile 1.3.2 Using jaro_winkler 1.5.4 Using method_source 0.9.2 Using parallel 1.19.1 Using parser 2.7.0.1 Using power_assert 1.1.5 Using pry 0.12.2 Using rainbow 3.0.0 Using rspec-core 3.9.1 Using rspec-mocks 3.9.1 Using rspec 3.9.0 Using ruby-progressbar 1.10.1 Using unicode-display_width 1.6.0 Using rubocop 0.78.0 Using simplecov-html 0.11.0.beta1 Using simplecov 0.18.0.beta1 from source at `.` Using test-unit 3.3.4 Bundle complete! 11 Gemfile dependencies, 52 gems now installed. Gems in the group benchmark were not installed. Use `bundle info [gemname]` to see where a bundled gem is installed. . bundler: failed to load command: parallel_rspec (/home/tobi/.asdf/installs/ruby/2.7.0/bin/parallel_rspec) Gem::Exception: can't find executable parallel_rspec for gem parallel_tests. parallel_tests is not currently included in the bundle, perhaps you meant to add it to your Gemfile? /home/tobi/.asdf/installs/ruby/2.7.0/lib/ruby/gems/2.7.0/gems/bundler-2.1.2/lib/bundler/rubygems_integration.rb:374:in `block in replace_bin_path' /home/tobi/.asdf/installs/ruby/2.7.0/lib/ruby/gems/2.7.0/gems/bundler-2.1.2/lib/bundler/rubygems_integration.rb:402:in `block in replace_bin_path' /home/tobi/.asdf/installs/ruby/2.7.0/bin/parallel_rspec:23:in `' F-- (::) failed steps (::) expected "bundle exec parallel_rspec spec" to be successfully executed (RSpec::Expectations::ExpectationNotMetError) ./features/step_definitions/simplecov_steps.rb:29:in `/^I open the coverage report generated with `([^`]+)`$/' features/parallel_tests.feature:11:in `When I open the coverage report generated with `bundle exec parallel_rspec spec`' Failing Scenarios: cucumber features/parallel_tests.feature:9 # Scenario: 1 scenario (1 failed) 4 steps (1 failed, 2 skipped, 1 passed) ```

output when manually running the command in the tmp/aruba/project folder:

``` tobi@speedy:~/github/simplecov/tmp/aruba/project(more-test-projects)$ bundle Fetching gem metadata from https://rubygems.org/.......... Resolving dependencies... Using bundler 2.1.2 Using diff-lcs 1.3 Using docile 1.3.2 Using parallel 1.19.1 Using parallel_tests 2.30.0 Using rspec-support 3.9.2 Using rspec-core 3.9.1 Using rspec-expectations 3.9.0 Using rspec-mocks 3.9.1 Using rspec 3.9.0 Using simplecov-html 0.11.0.beta1 Using simplecov 0.18.0.beta1 from source at `../../..` Bundle complete! 3 Gemfile dependencies, 12 gems now installed. Use `bundle info [gemname]` to see where a bundled gem is installed. tobi@speedy:~/github/simplecov/tmp/aruba/project(more-test-projects)$ bundle exec parallel_rspec spec 4 processes for 4 specs, ~ 1 specs per process [DEPRECATED] use `bundle info rspec-core` instead of `bundle show rspec-core` [DEPRECATED] use `bundle info rspec-core` instead of `bundle show rspec-core` [DEPRECATED] use `bundle info rspec-core` instead of `bundle show rspec-core` [DEPRECATED] use `bundle info rspec-core` instead of `bundle show rspec-core` .. Finished in 0.00191 seconds (files took 0.09317 seconds to load) 2 examples, 0 failures .. Finished in 0.00213 seconds (files took 0.09731 seconds to load) 2 examples, 0 failures Coverage report generated for (1/8), (2/8), (3/8), (4/8) to /home/tobi/github/simplecov/tmp/aruba/project/coverage. 23 / 33 LOC (69.7%) covered. Coverage report generated for (1/8), (2/8), (3/8), (4/8) to /home/tobi/github/simplecov/tmp/aruba/project/coverage. 32 / 39 LOC (82.05%) covered. . . Finished in 0.00184 seconds (files took 0.09976 seconds to load) 1 example, 0 failures Finished in 0.00241 seconds (files took 0.09835 seconds to load) 1 example, 0 failures Coverage report generated for (1/8), (2/8), (3/8), (4/8) to /home/tobi/github/simplecov/tmp/aruba/project/coverage. 37 / 43 LOC (86.05%) covered. Coverage report generated for (1/8), (2/8), (3/8), (4/8) to /home/tobi/github/simplecov/tmp/aruba/project/coverage. 42 / 47 LOC (89.36%) covered. 6 examples, 0 failures Took 0 seconds ```

Notice not only how it works but that the installed gems are also different.

I want the commands to execute in the context of the faked project so that I can have the gems listed only in its Gemfile and that the bundle exec in there then works.

The reason why I want have my dependencies separate in this example projects is because short to long term I want to test against different versions of Rails or parallel tests or what not. Theoretically I could switch on some environment variables in the Gemfile as a workaround but I'd really prefer if they were more isolated.

I think that's it, happy to give more background etc. and again thanks for your help and your project! :tada:

mvz commented 4 years ago

It looks like what you want is the effect of Bundler.with_clean_env (described in the Shelling out section of bundle help exec.

Aruba already tries to do something like that with its #unset_bundler_env_vars method, but recent experience with rspec-rails show that it's not doing its job. If it were doing its job, you could add the tag @disable-bundler to the scenario and that would be that. In the end, some code was added to rspec-rails to handle this: https://github.com/rspec/rspec-rails/commit/f9873aaa06494def8a577ecb4380115f0e6b99bc.

I really want to fix this in Aruba itself. Chances are that will be in version 1.0. Coming soon, I hope.

PragTob commented 4 years ago

@mvz thanks for all the background and help! Sadly, @disable-bundler isn't working (much like you suspected) - I'll give the workaround from rspec a shot. Thanks!

PragTob commented 4 years ago

@mvz thanks, the rspec-rails workaround works for me/us :ok_hand: Adapted it a little bit: https://github.com/colszowka/simplecov/pull/814/files#diff-a5541ecebed4f51eb37fffc180511ee7

Thanks a bunch! :green_heart:

IMG_20190215_122642_Bokeh

PragTob commented 4 years ago

@deivid-rodriguez not sure if you meant to close this one? While we've fixed it in simplecov my understanding was that @mvz wanted to explore a way to make it possible without freedom patching in aruba 1.0 :)

deivid-rodriguez commented 4 years ago

Definitely not, not sure how that happened.

PragTob commented 4 years ago

I think it was me formulating a commit message to that the magic fix targeted this, I guess when you merged it to your fork master it took you rights here to close it.

It's a fun world.

deivid-rodriguez commented 4 years ago

Yeah, that looks like what happened :man_shrugging:

mvz commented 4 years ago

~I'm trying to reproduce this as a small test case and I'm confused how Travis is supposed to know it needs to install parallel_tests.~ Ah found it: "Given I install dependencies". Makes sense :slightly_smiling_face:.

deivid-rodriguez commented 4 years ago

Do you mean in simplecov? It's included in the test project's Gemfile: https://github.com/colszowka/simplecov/blob/7b4235cc4f9db9c37b75e7ec51364508e4cd3e28/test_projects/parallel_tests_project/Gemfile.

deivid-rodriguez commented 4 years ago

Oh, you figured it out, great :)

mvz commented 4 years ago

The suggested method using @disable-bundler does work with Aruba master. I made a pull request: https://github.com/colszowka/simplecov/pull/827.

mvz commented 4 years ago

Closing this as disabling Bundler works in master.

cucumber / aruba