search

cucumber / cucumber-js

Cucumber for JavaScript
https://cucumber.io
MIT License
5.03k stars 1.09k forks source link

Programmatic API for running cucumber-js #1711

Closed davidjgoss closed 2 years ago

davidjgoss commented 3 years ago

Problem

Currently we don't have a good way to programmatically run cucumber-js. The need is from several angles:

As-is

What tends to happen at the moment is a new instance of Cli is created with strung-together argv input. It's obviously very unweildy and also isn't on the public API.

Sometimes (possibly due to perceived fragility of the above) frameworks will just rely on the cucumber-js CLI but struggle to find ways to integrate and have their own options.

The Runtime class is currently part of the public API but it's not useful in these contexts, depending on the pickles and support code to be provided by the caller.

Proposal

Two components in the project:

runCucumber

New async function that executes a test run in-process. Responsibilities:

This would be part of the public API and we'd encourage framework maintainers to use it when "wrapping" cucumber-js. We'd also use it for our own testing.

As much as possible, it would avoid direct interaction with process, instead accepting normalised options and stream interfaces for output, and leaving it to the caller to decide how to exit based on the result or an unhandled error.

Also Runtime should come off the public API as it's really an internal thing.

CLI

Effectively a "client" of runCucumber. Responsibilities:

This would continue to not be on the public API. Also it would only use functions/interfaces that are on the public API, such that we could easily break it out into its own package at some point, as is a common pattern now with projects like Jest.

This decoupling also paves the way for some interesting new CLI features without having them bleed into the internals, e.g.:

etc

We would also expose functions (consumable by the CLI and by others) for:

Timescale

We'd target this at the upcoming 8.0.0 release.

davidjgoss commented 3 years ago

Paging for initial feedback: @aslakhellesoy @charlierudolph @aurelien-reeves @mattwynne @nicojs @jan-molak

aslakhellesoy commented 3 years ago

I love this proposal! We already an API like this in fake-cucumber's runCucumber

jan-molak commented 3 years ago

@davidjgoss - sounds great!

For your reference, here's how Serenity/JS invokes Cucumber at the moment - CucumberCLIAdapter And here's the logic around converting configuration parameters to argv - CucumberOptions.

Being able to provide an options object instead of argv would be much nicer 👍🏻

aurelien-reeves commented 3 years ago

Love it!

aurelien-reeves commented 3 years ago

While specifying that new public API, we may also consider what recently happened with issue #1489 and think about providing public APIs to have more and better interaction with the filters and the resulting features under test

aslakhellesoy commented 3 years ago

And to add to that - let's make something that would make it easier for Cucumber-Electron too:

nicojs commented 3 years ago

Having a public API is better than nothing, so go ahead 👍!

Preferably I would also have an API to load profiles using the same rules as cucumber-js does, so I can mimic the exact behavior of a normal cucumber-js call.

loadProfiles(directory = process.cwd()): Record<string, Profile>

StrykerJS will also rely heavily on the custom_formatters API and the events published by the eventBroadcaster. Could we add those to the public API as well? See: https://github.com/stryker-mutator/stryker-js/blob/03b1f20ed933d3a50b52022cfe363c606c2b16c5/packages/cucumber-runner/src/stryker-formatter.ts#L45-L69

davidjgoss commented 3 years ago

Preferably I would also have an API to load profiles using the same rules as cucumber-js does, so I can mimic the exact behavior of a normal cucumber-js call.

This is a good point. Profiles (in their current form at least) are fundamentally coupled to the CLI so it feels right to keep them on that side of the boundary, but we could still expose a function to load them and generate a partial options object.

davidjgoss commented 3 years ago

While specifying that new public API, we may also consider what recently happened with issue #1489 and think about providing public APIs to have more and better interaction with the filters and the resulting features under test.

I think we could include an option for providing a custom pickle filter when calling the API (in addition to the names, tags etc that drive the built-in filtering).

aslakhellesoy commented 3 years ago

The current syntax for profiles if very command-line-y.

I would ❤️❤️❤️ to be able to specify profiles in a more generic format such as JSON, JavaScript, YAML or environment variables. In JSON it could look like this:

.cucumber.json

{
  "default": {
    "requireModule": ["ts-node/register"],
    "require": ["support/**/*./ts"],
    "worldParameters": {
      "appUrl": "http://localhost:3000/",
    },
    "format": ["progress-bar", "html:./cucumber-report.html"]
  },
  "ci": {
    "requireModule": ["ts-node/register"],
    "require": ["support/**/*./ts"],
    "worldParameters": {
      "appUrl": "http://localhost:3000/",
    },
    "format": ["html:./cucumber-report.html"],
    "publish": true
  }
}

Or, using JavaScript

.cucumber.js

const common = {
  "requireModule": ["ts-node/register"],
  "require": ["support/**/*./ts"],
  "worldParameters": {
    "appUrl": "http://localhost:3000/",
  }
}

module.exports = {
  default: {
    ...common,
    "format": ["progress-bar", "html:./cucumber-report.html"]
  },
  ci: {
    ...common,
    "format": ["html:./cucumber-report.html"],
    "publish": true
  }
}

Or even with environment variables (for example loaded with a tool like dotenv):

.cucumber.env

CUCUMBER_PROFILE_DEFAULT_REQUIREMODULE=ts-node/register
CUCUMBER_PROFILE_DEFAULT_REQUIRE=ts-node/register
CUCUMBER_PROFILE_DEFAULT_WORLDPARAMETERS_APPURL=http://localhost:3000/
CUCUMBER_PROFILE_DEFAULT_FORMAT=progress-bar,html:./cucumber-report.html
CUCUMBER_PROFILE_CI_REQUIREMODULE=ts-node/register
CUCUMBER_PROFILE_CI_REQUIRE=ts-node/register
CUCUMBER_PROFILE_CI_WORLDPARAMETERS_APPURL=http://localhost:3000/
CUCUMBER_PROFILE_CI_FORMAT=progress-bar,html:./cucumber-report.html
CUCUMBER_PROFILE_CI_PUBLISH=true

In fact, the config library does exactly this. We never ended up integrating it in Cucumber-JVM because other stuff got in the way, but maybe we could give it a go with a JavaScript implementation?

davidjgoss commented 3 years ago

@aslakhellesoy agree that would be awesome! I'll try and get a POC going for this proposal so we have something a bit more concrete to talk around, and would love to make profiles right as part of it (4.5 years and counting on #751 😄)

aurelien-reeves commented 3 years ago

Refs. #1004

nicojs commented 3 years ago

This is a good point. Profiles (in their current form at least) are fundamentally coupled to the CLI so it feels right to keep them on that side of the boundary, but we could still expose a function to load them and generate a partial options object.

Yes, that would be awesome and much appreciated from the point of view of plugin creators.

I would ❤️❤️❤️ to be able to specify profiles in a more generic format such as JSON, JavaScript, YAML or environment variables. In JSON it could look like this:

That sounds great! And is also exactly the reason I would appreciate an API to load them the same way as cucumberJS does. Loading a single cucumber.js file is trivial. Replicating a configuration file loading algorithm, including precedence, file format, etc AND maintaining it is something else entirely 😅.

nicojs commented 3 years ago

Q: Would I be able to run runCucumber twice in succession _without clearing the require cache_? This is important for the mutation testing use case.

We want to load the environment and run the tests multiple times in quick succession while changing a global variable in order to switch the active mutant.

Right now, we're using the cli private API and we need to clear the step definition files from the require.cache between each test run. This is not ideal for CommonJS and won't work at all for esm.

Pseudo code of our use case:

const profiles = await loadProfiles();
const options = {
  ...profiles.default,
  formatter: require.resolve('./our-awesomely-crafted-formatter'),
  some: 'other options we want to override',
}
const cucumber = new Cucumber(options);

// Allow cucumber to load the step definitions once. 
// This is `async`, so support for esm can be added without a breaking change
await cucumber.initialize();

// Initial test run ("dry run"), without mutants active
await cucumber.run();

collectMutantCoveragePerTestFromFormatter();

// Start mutation testing:

global.activeMutant = 1;
await cucumber.run({ features: ['features/a.feature:24']);
collectResultsFromFormatterToDetermineKilledOrSurvivedMutant()

global.activeMutant = 2;
await cucumber.run({ features: ['features/b.feature:24:25:26', 'features/c.feature:12']);
collectResultsFromFormatterToDetermineKilledOrSurvivedMutant()

// etc
davidjgoss commented 3 years ago

@nicojs definitely agree we need this, it's come up a few times before with e.g. people wanting to run cucumber in a lambda, and I'd also like to add an interactive mode which would need it too.

What I had sketched out so far was in more of a functional style but same fundamental concept I think:

const runnerOptions = {
    support: {
        require: ['features/support/**/*.js']
    }
}

// initial run returns support code library
const { support } = await runCucumber(runnerOptions)

// subsequent run reuses support code library
await runCucumber({
    ...runnerOptions,
    support
})
nicojs commented 3 years ago

That works for us 👍

jan-molak commented 3 years ago

Works for us too 👍🏻

rkrisztian commented 3 years ago

I really think that something like this would be extremely useful as an alternative for the great mess we have today with testing tool integrations (Jest, Cypress), for example I found these problems (in the order of importance):

I would rather see some minimal glue code between Jest/Karma/Cypress/etc. and cucumber-js so I don't have to suffer for all those missing features I need to use.

mattwynne commented 3 years ago

Great suggestion @davidjgoss 👍

This separation of concerns between the command-line user interface and the "business logic" of parsing and executing scenarios as tests reminds me of the hexagonal architecture pattern.

In cucumber-ruby we actually split the core domain logic (or "inner hexagon") into a separate gem package, as we were rebuilding it from scratch in a "clean room". I realise that's not the context here, but it might be worth drawing some inspiration from, or feeding back innovations from this design into the Ruby API. There's an example in the cucumber-ruby-core gem's README of how to use that API.

davidjgoss commented 3 years ago

Okay, here's a first pass at the API signature for the "run" bit. It's heavily based on the IConfiguration object we have internally (so shouldn't cause too much refactoring lower down) but just a little less "flat":

export interface IRunCucumberOptions {
  cwd: string
  features: {
    defaultDialect?: string
    paths: string[]
  }
  filters: {
    name?: string[]
    tagExpression?: string
  }
  support:
    | {
        transpileWith?: string[]
        paths: string[]
      }
    | ISupportCodeLibrary
  runtime: {
    dryRun?: boolean
    failFast?: boolean
    filterStacktraces?: boolean
    parallel?: {
      count: number
    }
    retry?: {
      count: number
      tagExpression?: string
    }
    strict: boolean
    worldParameters?: any
  }
  formats: {
    stdout: string
    files: Record<string, string>
    options: IParsedArgvFormatOptions
  }
}

export interface IRunResult {
  success: boolean
  support: ISupportCodeLibrary
}

export async function runCucumber(
  options: IRunCucumberOptions
): Promise<IRunResult> {
  // do stuff
}

And a very contrived example usage:

const result = await runCucumber({
  cwd: process.cwd(),
  features: {
    paths: ['features/**/*.feature'],
  },
  filters: {
    name: ['Acme'],
    tagExpression: '@interesting',
  },
  support: {
    transpileWith: ['ts-node'],
    paths: ['features/support/**/*.ts'],
  },
  runtime: {
    failFast: true,
    retry: {
      count: 1,
      tagExpression: '@flaky',
    },
    strict: true,
    worldParameters: {
      foo: 'bar',
    },
  },
  formats: {
    stdout: '@cucumber/pretty-formatter',
    files: {
      'report.html': 'html',
      'TEST-cucumber.xml': 'junit',
    },
    options: {
      printAttachments: false,
    },
  },
})

Feedback welcome! Note this doesn't cover the profile/config loading stuff which would be another function.

nicojs commented 3 years ago

I think this looks good. Question: How would I configure a custom formatter?

davidjgoss commented 3 years ago

@nicojs sorta like on the CLI

formats: {
    files: {
      'report.html': './my/fancy-reporter.js',
      'other-report.html': '@me/reporter-package',
    }
  },
aslakhellesoy commented 3 years ago

Great to see progress on this @davidjgoss!

I don't want to slow down the progress on this, but at the same time I want to make sure we adopt a format that can work for other Cucumber implementations too.

Eventually a JSON schema, but while we discuss it I think TypeScript types are easier for us humans to parse.

I suggest we create a new issue about the proposed format in the cucumber/common monorepo and invite the core team to discuss there.

davidjgoss commented 3 years ago

@aslakhellesoy will do.

What would you think about the programmatic API not being tied to the common options structure? Like we'd map from that to the runCucumber options. It adds maybe a little complexity but appeals because of things like having a support block that's either parameters to load, or a previously loaded support code library. Could do similar for features+pickles too. And there are various options we'd support on the CLI (e.g. --exit) that aren't appropriate on the programmatic API.

aslakhellesoy commented 3 years ago

What would you think about the programmatic API not being tied to the common options structure?

I think that's fine, as long as we provide a function that converts from the options file contents to the data structure runCucumber wants.

nicojs commented 3 years ago

Eventually a JSON schema, but while we discuss it I think TypeScript types are easier for us humans to parse.

Why do we need to choose? We're using JSON schema in StrykerJS to generate typescript using json-schema-to-typescript. We're not committing the TS output files to source control, instead we generate them on the fly using a prebuild step.

JSON schemas are still somewhat readable for humans IMO. We've already had PR's on the Stryker repo and people seem to know what to do 🤷‍♀️

sorta like on the CLI

formats: {
    files: {
      'report.html': './my/fancy-reporter.js',
      'other-report.html': '@me/reporter-package',
    }
  },

How would this work for a reporter that doesn't need an output file name? Like so:

formats: {
  files: {
    '': require.resolve('./my-awesome-stryker-formatter')
  }
}
aslakhellesoy commented 3 years ago

Why do we need to choose?

I think we should use a JSON Schema as a single source of truth for the structure of the configuration. -And then generate TypeScript/Java/Whatever code from that schema.

But JSON Schema is a bit hard to read for humans, so while we are discussing the schema in a GitHub issue in cucumber/common I was suggesting TypeScript to facilitate the discussion.

See what I mean?

aslakhellesoy commented 3 years ago

JSON schemas are still somewhat readable for humans IMO

Not to me :-) Too verbose.

davidjgoss commented 3 years ago

How would this work for a reporter that doesn't need an output file name?

formats: {
    stdout: './my-awesome-stryker-formatter',
    files: {
      'report.html': './my/fancy-reporter.js',
      'other-report.html': '@me/reporter-package',
    }
  },

(only one formatter can use the stdout stream)

shawnbot commented 3 years ago

This sounds great to me. Over in #1724 I was asking if there's a way to modify Cucumber step results so that I can integrate with a test runner that needs the step configuration upfront (which I was hoping to build up from step definitions), runs the whole test suite (scenario) async, then returns results for each step. Typically this would be done in a hook, but none of the Cucumber hooks runs at the right time or gives me (public) access to the step results so that I can update them before they're passed to the formatter.

I could probably do this work in a custom formatter, but that involves (AFAICT) having to share state between the custom world objects (that build up test suites) and the formatter, which feels like a gross hack. Wrapping the test runner programmatically would at least make it easier and a bit more graceful. Without a hooks-like API for runCucumber(), I would expect to be able to do something like this:

import { runCucumber, formatResults } from '@cucumber/cucumber'
import { cucumberDefaults, transformTestResults } from '@sfgov/ghost-cucumber'

const results = await runCucumber({
  ...cucumberDefaults,
  formats: { /* no formatting? */ }
})

const modifiedResults = await transformTestResults(results)

formatResults(modifiedResults, {
  // use the default formatter if none is provided?
  file: '/dev/stdout'
})
  .then(({ status }) => process.exit(status)) // does the formatter report status, or...?
OhadR commented 2 years ago

note that i have spent HOURS on this, trying to run cucumber by its Cli. After struggling, I have noticed that it can be invoked only once, and in the second run it does not find the step-definitions (because of SupportCodeLibraryBuilder - nodejs cache the step-definitions so in the 2nd run it does not "require" them again, in getSupportCodeLibrary())

davidjgoss commented 2 years ago

@OhadR we're going to be catering for this as part of the new runCucumber API which is planned for 8.0.0- see https://github.com/cucumber/cucumber-js/issues/1894.

In the meantime there are workarounds with removing things from the require cache yourself https://nodejs.org/api/modules.html#requirecache

binarymist commented 2 years ago

In the meantime there are workarounds with removing things from the require cache yourself

Or Just create a process for each cuc cli instance as we've been doing for about 4 years.

binarymist commented 2 years ago

Currently we don't have a good way to programmatically run cucumber-js

Hmm, we've been running it fine for four years.

Projects that use cucumber-js as part of a framework (e.g. Serenity, Stryker)

You could add PurpleTeam to this list.

We have multiple (app-scanner, tls-scanner, server-scanner soon) Testers, architecture diagram here. For all of our Testers.

Currently with the move from:

our CLI's test command still works, but our testplan is broken for all Testers.

The following example is of our app-tester:

testplan

  1. For the testplan command we...
  2. createCucumberArgs
  3. We pass the new CucCli instance to getActiveFeatureFileUris
  4. Which calls the legacy getConfiguration method of the cucumber CLI instance which no longer exists

@aurelien-reeves mentioned in Slack:

A lot may have changed, but actually you should be able to find the same things than before, they may have just moved.

What should we now use to replace this removed functionality?

test

In the app-tester we spawn multiple cucumber CLI instances based on the number of TestSessions passed from the CLI to the orchestrator to the Testers.

  1. We create the cucumber args like this
  2. Then we spawn our bin file like this
  3. Which instantiates and runs the cucumber CLI like this

Will this still work with the proposed changes, if not what will we need to do to refactor?

@davidjgoss does your IRunCucumberOptions example here include: path to step definitions?

What would we need to change to move to the new runCucumber?

Thanks.

davidjgoss commented 2 years ago

Thanks for the detailed info @binarymist, that will be very useful. For now I’d say sit tight until the next RC as there is still more to do on this piece eg configuration loading (see linked milestone). It’ll be fully documented when ready.

binarymist commented 2 years ago

Hi @davidjgoss

@aurelien-reeves mentioned in Slack:

A lot may have changed, but actually you should be able to find the same things than before, they may have just moved.

What should we now use to replace this removed functionality?

This is our imediate problem... as this functionality is just moved, where can we find it?

Thanks.

nicojs commented 2 years ago

@davidjgoss I might have a use case to be able to load the support code first, activate the mutant and then run the tests. I think we would be able to implement this feature through one of the on('envelope') events in our custom Formatter, but maybe it is easy for you to add this on its own?

const support = await loadSupport({
    transpileWith: ['ts-node'],
    paths: ['features/support/**/*.ts'],
});
globalThis.__stryker__.activeMutant = "1";
const result = await runCucumber({ support });

See https://github.com/stryker-mutator/stryker-js/issues/3442 for a bunch of details 😉

davidjgoss commented 2 years ago

@nicojs that should be possible although it would also require the "sources" part of the configuration, since the resolution of support code takes the feature paths into account. But as a consumer I think you would just need to load the whole configuration object anyway. So like:

import {loadConfiguration, loadSupport, runCucumber} from '@cucumber/cucumber`

const configuration = await loadConfiguration()
const support = await loadSupport(configuration)
const result = await runCucumber({...configuration, support})

Does that make sense?

nicojs commented 2 years ago

That would be sweet 😌. Let me know when we can do tests with this setup.

davidjgoss commented 2 years ago

Just calling out that as we'll have loadSupport() to load the support code separately to actually doing a test run, we might want something similar like loadSources() to do the Gherkin loading and processing as a standalone thing. A good use case is the example from @binarymist where a test plan is calculated and reported without doing a run.

nicojs commented 2 years ago

That would also help mutation testing performance a lot! 😍

jan-molak commented 2 years ago

@davidjgoss - and do you think it could be possible for the API to tell the consumer how many scenarios are going to be executed in each .feature file? Or at least that there will / won't be scenarios to be executed in a given file?

WebdriverIO has a nice feature where they can avoid launching the browser for feature files that have no scenarios to be executed - for example, when all scenarios in a feature file are excluded because of tags. However, this functionality requires WebdriverIO to parse feature files, which duplicates what Cucumber is already doing, and which implementation has to be kept in sync with any changes to Cucumber internals.

Perhaps if the process of launching Cucumber could be a bit more fine-grained:

import {loadConfiguration, loadSupport, loadSpecs, runCucumber} from '@cucumber/cucumber`

const configuration = await loadConfiguration()
const support = await loadSupport(configuration)
const specs = await loadSpecs(configuration)
const result = await runCucumber({...configuration, support, specs})

where specs contains information about scenarios to run in each feature file?

Maybe @christian-bromann has some ideas here too?

davidjgoss commented 2 years ago

@jan-molak yeah i didn't describe it that well but I think my loadSources and your loadSpecs are kinda the same idea - loadSources would be giving you the features and pickles, either pre-filtered or with the easy means to filter (based on tags, names, urls, lines).

Cucumber has the concept of a "test case" but these are the composition of filtered pickles with support code including hooks etc which I don't think is right for the use case you describe - you're just interested in which scenarios are going to get run regardless of the state of support code, right?

jan-molak commented 2 years ago

Hey @davidjgoss!

you're just interested in which scenarios are going to get run regardless of the state of support code, right?

Let me think aloud :) Consider this example. Let's say we have a couple of feature files like the ones below:


# features/first.feature

Feature: First feature                            # 1
                                                  # 2
  @current                                        # 3
  Scenario: First scenario                        # 4
                                                  # 5
  Scenario: Second scenario                       # 6

# features/second.feature

Feature: Second feature

  Scenario: Some other scenario

When I ask Cucumber to run:

await loadSpecs({
  specs: [`features/first.feature`]
  tags: []
})

I'd like it to tell me that it will run:

[
  `features/first.feature:4`,
  `features/first.feature:6`
]

When I ask it to run:

await loadSpecs({
  specs: [`features/first.feature:4`]
  tags: []
})

I'd like it to tell me that it will run:

[
  `features/first.feature:4`
]

When I use tags (or any other filter):

await loadSpecs({
  specs: [`features/*.feature`]
  tags: [`@current`]
})

I'd like it to tell me that it will run:

[
  `features/first.feature:4`
]

So, given a configuration that includes paths to feature files, filters, tags, etc, I'd like to receive a list of scenario locations that Cucumber is going to execute when I invoke run.

When the list is empty, I know I don't need to start the browser. When it's non-empty, I could use it to "load-balance" scenarios between browsers.

Does it make sense?

davidjgoss commented 2 years ago

@jan-molak it does, thanks! I can't promise it'll look exactly like that, but it'll be possible.

davidjgoss commented 2 years ago

Quick follow-up on the loadSources/loadPickles/loadSpecs (or whatever we call it) function which I'm going to work on today. There are two possible use cases I can see:

My thought for 8.0.0 is to focus on the first one but ensure the API is flexible enough to add the second one later in a non-breaking way. Thoughts?

aurelien-reeves commented 2 years ago

Good plan 👌

My though is more related to v8.0.0: did we released 8.0.0-rc.1 a little bit early? 🤔

davidjgoss commented 2 years ago

did we released 8.0.0-rc.1 a little bit early?

Yes, I think so in retrospect.

davidjgoss commented 2 years ago

8.0.0-rc.3 has been released including this new API, there are some docs here - a little thin at the moment but will continue to expand them.

If anyone has any early feedback that'd be fantastic (cc @jan-molak @nicojs @binarymist).

MarufSharifi commented 2 years ago

Hey thanks for doing this great job, I need to use from @cucumber/cucumber: 8.0.0, when it will be available to use?