DataDog / datadog-ci-rb

Ruby library for Datadog test visibility
https://docs.datadoghq.com/continuous_integration/tests/ruby
Other
8 stars 4 forks source link

[CIVIS-9332] unskippable tests for ITR #167

Closed anmarchenko closed 6 months ago

anmarchenko commented 6 months ago

What does this PR do? Adds "unskippable by ITR tests" support for RSpec, Cucumber, and Minitest. It allows application developers using intelligent test runner to mark some of their test to make sure that they are always run and never skipped by ITR.

Motivation ITR is not able to correctly track dependencies between tests and external non-code resources. Also, Ruby code coverage does not work across threads and/or processes.

Examples of tests that should be unskippable by ITR:

Additional Notes Usage examples are below

RSpec:

# mark the whole suite unskippable
RSpec.describe MyClass, datadog_itr_unskippable: true do
  # tests here
end

# mark one block unskippable
RSpec.describe MyClass do
  describe "#read_csv", datadog_itr_unskippable: true do
    it "reads CSV file" do
    end
  end
end

# mark one test unskippable
RSpec.describe MyClass do
  it "works", datadog_itr_unskippable: true do
  end
end

Cucumber:

# mark the whole feature unskippable
@datadog_itr_unskippable
Feature: My feature
  Scenario: Main page opens
    Given open main page
    Then see Hello world text

# mark one scenario unskippable
Feature: My feature
  @datadog_itr_unskippable
  Scenario: Main page opens
    Given open main page
    Then see Hello world text

Mintiest:

# mark test suite unskippable
class UnskippableTest < Minitest::Test
  datadog_itr_unskippable

  def test_1
  end

  def test_2
  end
end

# mark one test unskippable
class PartiallyUnskippableTest < Minitest::Test
  datadog_itr_unskippable "test_1"

  def test_1
  end

  def test_2
  end
end

Note that for minitest the developer's experience is not on the same level as for RSpec or Cucumber: this is due to the fact that minitest does not have test tagging capabilities out of the box. Currently it will be required to pass test names of unskippable tests to datadog_itr_unskippable method. We'll gather feedback on this and if the need arises, will provide better tagging capabilities (similar to what https://github.com/bernardoamc/minitag or https://github.com/jbodah/minitest-tagz can do).

How to test the change? Unit tests are provided.

codecov-commenter commented 6 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 99.06%. Comparing base (f0592d1) to head (67c63db).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #167 +/- ## ======================================= Coverage 99.06% 99.06% ======================================= Files 196 196 Lines 9377 9404 +27 Branches 407 409 +2 ======================================= + Hits 9289 9316 +27 Misses 88 88 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.