cucumber html meta data

[a1dutch]

Hi,

How is it possible to pass the product name and product version meta data so its included in the new html report?

I'm using cucumber jvm 6.9.1 and the html plugin.

Cheers

Andy

[aslakhellesoy]

These fields are provided by the Cucumber implementation. They are not meant to be set by users. More details here: https://github.com/cucumber/cucumber/blob/master/messages/messages.md#meta

[a1dutch]

Hi @aslakhellesoy,

How do we get these populated in the end report?

This is all we see, we would like the product name and version displayed (plus some other custom ones if possible)

[aslakhellesoy]

How do we get these populated in the end report?

Sorry, I don’t understand the question. What do you mean by “these”? Are you talking about the name of the product you are testing/developing with Cucumber?

Do you want to add additional information to the report summary? Can you give me a concrete example (some real values) of what information you would like to add? Could you also explain why you would like to add it?

[a1dutch]

In the cucumber messsages docs you posted this has lots of additional information (CI, Git, Product, Version, ...)

Inside cucumber react which is used to generate the html report it should show this information if present: https://github.com/cucumber/cucumber/blob/master/react/javascript/src/components/app/ExecutionSummary.tsx, https://github.com/cucumber/cucumber/blob/master/react/javascript/src/components/app/CICommitLink.tsx

when running from with cucumber-junit via maven none of this information is present or displayed in the html report.

i would like the following displayed at the top of the html report

• Product Name
• Product Version
• Git Commit
• Git Branch
• Git url
• CI url

obviously this information needs to be passed to cucumber when its ran (via command line options or environment variables) for it to be added to the report, but there seems no way at present to do this.

[aslakhellesoy]

In the meta message there are 4 fields of type Meta.Product:

• implementation
• runtime
• os
• cpu

The Meta.Product type is a struct with a name and version. You can see an example of what this message looks like here (this is an example message we use for testing purposes).

Here it is, prettified:

{
  "meta": {
    "protocolVersion": "13.0.1",
    "implementation": {
      "name": "fake-cucumber",
      "version": "8.0.0"
    },
    "runtime": {
      "name": "node.js",
      "version": "12.16.2"
    },
    "os": {
      "name": "linux",
      "version": "4.19.76-linuxkit"
    },
    "cpu": {
      "name": "x64"
    },
    "ci": {
      "name": "GitHub Actions",
      "url": "https://github.com/cucumber-ltd/shouty.rb/actions/runs/154666429",
      "git": {
        "remote": "https://github.com/cucumber-ltd/shouty.rb.git",
        "revision": "99684bcacf01d95875834d87903dcb072306c9ad",
        "branch": "master"
      }
    }
  }
}

As you can see, there is no product field - it is just a type with a name and an optional version to describe:

• the Cucumber implementation (cucumber-ruby, cucumber-js, cucumber-jvm etc)
• the underlying runtime (JVM, Node.js, Ruby etc)
• the Operating sustem os (Windows, Linux, MacOS etc)
• the CPU architecture cpu (x86, amd64 etc)

The ci field is a slightly more complex struct of type Meta.CI with information about the CI server and the Git remote/revision/branch.

The construction of this entire meta message is handled by the create-meta library, which is used by Cucumber-JVM and other Cucumber implementations.

The ci field is populated from well-known environment variables defined by the most common CI servers. That's how the Git information is detected as well, as we don't want to shell out to git or bundle a native git client.

If no CI environment is detected, the ci field will be null, but all the other fields of meta should still be defined.

In other words, you have to run Cucumber on one of the supported CI servers in order for the ci field to be populated.

You could define those environment variables in your terminal manually, but that's error-prone and cumbersome.

In order to simplify generation and publishing of reports we're also offering the Cucumber Reports service, which we recommend using with your favourite CI server.

Does this answer your questions?

[aslakhellesoy]

The report screenshot you attached displays all the information you could expect from a non-CI execution. If you run on a CI server you should also see the CI/Git information.

[a1dutch]

Thanks that is a lot of information.

We are using jenkins which does not seem supported, you can tell if its running on jenkins by the JENKINS_URL environment variable. There are no environment variables for the git information though so reading from a generic environment variable would allow these to be passed in before execution of cucumber.

Being able to display custom fields (product name and version under test, environment under test, profiles, etc...) would make these reports more detailed. We use the https://github.com/jenkinsci/cucumber-reports-plugin that allows you to do this but we like the single html file that cucumber produces.

The cucumber report service looks nice, but not something we would be able to use due to sensitive data in the reports.

[aslakhellesoy]

@a1dutch what's needed in order to close this issue?

[a1dutch]

@aslakhellesoy i think the following

• jenkins support cucumber/common#1253
• overriding of default meta data
• adding custom meta data

[aslakhellesoy]

@a1dutch how would you like to specify additional metadata? How/where would you like it rendered?

[a1dutch]

@aslakhellesoy maybe move the search/filter down onto its own line and span the whole width, you could then add them in their own box on the top right (where the filter is).

Another option is to add them to the middle section (this might be pretty crowded, so it could be displayed in two columns and you could move the filter down as above).

you could pass these in as command line arguments

-Dcucumber.plugin.react.info=App Version=1.2.3, App Commit=12da3fa2, Tested By=Andy

or evironment variables

CUCUMBER_PLUGIN_REACT_INFO=App Version=1.2.3, App Commit=12da3fa2, Tested By=Andy

Cheers

Andy

[aslakhellesoy]

Ok, so if we use an environment variable like CUCUMBER_PLUGIN_REACT_INFO=App Version=1.2.3, App Commit=12da3fa2, Tested By=Andy, would you expect the value of that variable to be split on , and then rendered independently?

[a1dutch]

yes seperated by , you might want to support quoted strings like below (but i could live without that), and then rendered indepently.

CUCUMBER_PLUGIN_REACT_INFO=App Version="1f23,1d53", App Commit='1f23,1d53', Tested By="Holland, Andy"

i dont know if there is a standard way to pass parameters to a plugin, cucumber-jvm only seems to accept one parameter as part of the plugin and this is the report location

[davidjgoss]

cucumber-js supports --format-options as a JSON literal but I think it might be the only implementation that does.

[mpkorstanje]

I don't think we should try to squeeze this in through an environment variable or through the --plugin CLI option. Depending on the OS and CI system, providing complex strucutured information this way is impossible without also having to derive one or two layers of escape characters. This doesn't make for great UX.

Additionally it also limits our future options for expansion and doesn't allow others to use our code to create their own solutions. This makes the whole thing a dead end.

Rather, I think we should take a programatic approach first. That is, we create a API that can create the html report from the messages. And using that API add additional information. This API can then be used by tools like jenkinsci/cucumber-reports-plugin.

The current API of the html report is messages. So to add this custom information a new message is needed. And this does bring us back to the old problem. Our current message format is not extensible.