Allow execution of non-feature tests

[crandles]

Support the execution of tests without requiring use of the features package.

To the Environment interface, I propose the addition of two new functions:

type Environment interface {
    RunTest(ctx context.Context, t *testing.T, cfg *envconf.Config) bool
    RunBenchmark(ctx context.Context, b *testing.B, cfg *envconf.Config) bool
}

I am unsure what if anything makes sense as return values, t.Run returns a bool which may fit here; additionally RunBenchmark might deserve its own issue, given the current lack of support overall for benchmarks.

Note, there is some overlap in wording with this existing function, unsure if the above naming will be confusing:

    // Run Launches the test suite from within a TestMain
    Run(*testing.M) int

The goal is to allow a more stream-lined path to writing go tests natively or otherwise with the framework of choice; don't force usage of the features package.

Potential usage:

testenv.RunTest(ctx context.Context, t *testing.T, cfg *envconf.Config) (context.Context, error){
  ...
  if err != nil {
    t.Fatal(err)
  }
  ...
   t.Run("...", func(t testing.T){
     t.Parallel()
     ...
   })

})

[harshanarayana]

@crandles https://github.com/kubernetes-sigs/e2e-framework/issues/133 We kind of sort of dig into the idea of running Benchmark workflows. But didn't manage to get it ahead. I would love to see that being picked up again. it would be a wonderful addition

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[vladimirvivien]

This may still be something we need to check out cc @crandles @harshanarayana

/remove-lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[vladimirvivien]

/remove-lifecycle stale

I like the idea of offering a light way of launching feature tests.

[crandles]

Started to look into this again:

• The simplest option seems to be to allow accessing the environment config: https://github.com/kubernetes-sigs/e2e-framework/pull/451
• I explored adding a RunTest helper that called the same underlying helper functions as Test, so that the before/after helpers ran -- and this works -- but without further code modifications, the feature/assesment concepts still leak through test names (if feature name is empty, it is rendered as Feature-1, assessment as Assessment-1, etc)
• benchmark helpers would need a whole new suite of functions, as testing.T is used throughout the current set of functions, and we need testing.B
  • migrating to testing.TB might help, but would be breaking
  • this prompted me to simply expose the env config, this can be used from a benchmark test

[vladimirvivien]

@crandles this sounds awesome. Yes, this framework did not have benchmark in mind, but would be a great addition. This is exciting, can't wait to see how this turns out.