Cumulative HTML report for multiple features

[athiradamodaran]

Hi,

If I have to run multiple features and want to generate a single HTML report for all of the features, will it work using this command?

behave -f behave_html_pretty_formatter:PrettyHTMLFormatter -o <destination_path> <features folder path>

When I tried with the above command, I am getting a 404 not found error when opening the HTML file in the browser.

[modehnal]

Hello, yes that works. I have tested it locally with your command and the page is generated without issue.

Maybe double check your destination_path when you are opening the html log within browser,but other than that I see nothing wrong.

I suppose you are not getting any error with the tests that are running? It would make sense that the page was not there if something happened in the behave run.

[athiradamodaran]

1. I tried running for a single feature as well as multiple features. But I get the 404 error when opening the HTML file in the browser. Also, when I open the HTML file in the IDE, it throws the below error:

Could this be the reason I am getting a 404? The destination path is correct and the reports gets generated correctly there.

2. I tried giving the config parameters in the behave.ini as:

# — FILE: behave.ini
[behave.formatters]
html = behave_html_pretty_formatter:PrettyHTMLFormatter

If I run using the command

$ behave -f html -o reports/ei/rel.html -D runner.continue_after_failed_step=true (feature-path) --no-capture

I get the error:

behave: error: format=html is unknown

In order to run, I am using the command:

behave -f behave_html_pretty_formatter:PrettyHTMLFormatter -o reports/ei/rel.html -D runner.continue_after_failed_step=true (feature-path) --no-capture

[modehnal]

I followed your setup and commands and even generated single test, single feature and more features. But once again all is fine on my end. I even verified I have the same duplicate id reference like you but I get the html file to open without any issue.

But I think you might have gotten a hint what might be wrong. The behave: error: format=html is unknown is not very descriptive on the latest released behave and we had issues debugging it in the past also. But I think we are getting closer, somehow the behave does not know about the html formatter. So I think the issue lies in the behave.ini file.

To debug this please install the latest dev version of behave pip install git+https://github.com/behave/behave@v1.2.7.dev3which has more verbose error description, start the tests again and let me know what error you will get.

[jenisys]

It should not make any difference, if you specify the same formatter class on command-line or in the config-file.

Please try the following:

$ behave -f help

NOTE:

• This requires that you install the HEAD of the behave repository or behave v1.2.7.dev4 or v1.2.7.dev3 (mentioned above) for example (a newer version from the repo).
• The output of the help formatter indicates what kind of problem you have with the html formatter alias.

SEE:

• https://github.com/behave/behave

[modehnal]

Also when you are testing this. Try to fix the duplicate id reference by hand, change the id and try to open the html file in browser. We might be on different systems with different browsers so the issue might be as simple as that. We would than fix the duplicated id if the issue is there.

[athiradamodaran]

Thank you for the response. I am able to open the HTML logs from the folder location. Also, when I give the formatter class in the command line, it works properly.

[modehnal]

Thank you for the response. I am able to open the HTML logs from the folder location. Also, when I give the formatter class in the command line, it works properly.

That is good to hear.

Can you please share what you changed or what the issue was? Did the newer behave version help? Or was the issue somewhere else? I am curious as to what actually happened here, since I was unable to reproduce the issue that you were seeing.

[athiradamodaran]

I am still not able to open the HTML files directly from the Pycharm IDE. However, I can open it from the corresponding file location. I believe it must be an issue with the local path.

For the formatter class issue, I removed the config from behave.ini and gave it only in the command line. I didn't upgrade to the newer version since as part of the framework we are still continuing to use 1.2.6 and had some dependencies to change it.

[modehnal]

I removed the config from behave.ini

That would lead me to believe the behave.ini file was not placed in the correct directory? The file should be located on the same level as your features/directory (or at least that is where I have it, things could have changed):

- my_project
  - features/
  - behave.ini

While you have solved your issue I think having the file present will come in handy when you need to do a changes to tittle, date_format, summary default behavior and more. For example in my projects I am using this template:

# -- FILE: behave.ini
# Define ALIAS for PrettyHTMLFormatter.
[behave.formatters]
html-pretty = behave_html_pretty_formatter:PrettyHTMLFormatter

# Optional configuration of PrettyHTMLFormmater
# also possible to use "behave ... -D behave.formatter.html-pretty.{setting}={value}".
[behave.userdata]
behave.formatter.html-pretty.title_string = Behave Test Suite

# Example usecase, print {before/after}_scenarios as steps with attached data.
behave.formatter.html-pretty.pseudo_steps = false

# Structure of the result html page readable(pretty) or condensed.
behave.formatter.html-pretty.pretty_output = true

# The '%' must be escaped in ini format.
behave.formatter.html-pretty.date_format = %%d-%%m-%%Y %%H:%%M:%%S (%%s)

# Defines if the summary is expanded upon start.
behave.formatter.html-pretty.show_summary = false

# Defines if the user is interested in what steps are not executed.
behave.formatter.html-pretty.show_unexecuted_steps = true

So while I am still intrigued as to what your issue was, if the data provided is enough we can leave it there.

Good luck.

[fpokryvk]

Hello,

sorry for the late response, indeed, we have duplicate IDs in HTML, we should fix that, thanks for pointing that out. However, that issue is not critical and should not cause 404 error. Quick search resulted in this thread, maybe it will help, if you didn't solve it yet.

https://intellij-support.jetbrains.com/hc/en-us/community/posts/360000104764-Running-index-html-gives-a-webpage-that-says-404-not-found

Thank you for using the behave and html-pretty formatter!

[athiradamodaran]

Oh. I have multiple feature folders. So I created a common behave.ini file thinking it would be applicable for all. So, my folder structure looks like this:

+- src/
|  +-- proj1/
|  |  +-- features/
|  |     +-- steps/
|  +-- proj2/
|  |  +-- features/
|  |     +-- steps/
|
+- behave.ini

Do you suggest that I create a behave config file for each folder?

Also, I made the change to the IDE mentioned above and now it works. Thanks a lot!

[jenisys]

The problem are not the multiple features/ directories. But the behave test-runner supports currently only one step-directory. Therefore:

• You need two test runs, one for each directory
• You provide your own test-runner that supports multiple step-directories.

SEE ALSO:

• behave: features/directory_layout.advanced.feature

[athiradamodaran]

Thank you for the response. I tried giving the behave.ini inside the directory. So, it would be like:

+- src/
|  +-- proj1/
|  |  +-- features/
|  |     +-- steps/
|  |     +-- behave.ini

And I gave the content as :

# — FILE: behave.ini
[behave.formatters]
html-pretty = behave_html_pretty_formatter:PrettyHTMLFormatter

[behave.userdata]
runner.continue_after_failed_step = true

behave.formatter.html-pretty.title_string = Behave Test Suite

# Example usecase, print {before/after}_scenarios as steps with attached data.
behave.formatter.html-pretty.pseudo_steps = false

But when I ran the behave command below, I got the following error:

$ behave -f html-pretty -o <report_path> <features>   --no-capture
usage: behave [options] [ [DIR|FILE|FILE:LINE] ]+
behave: error: format=html-pretty is unknown

Can you help with this? I am unable to understand where to give the details.

[jenisys]

@athiradamodaran As stated above: Use behave -f help with a new behave version from the Git repository. It will show you, what kind of problems you have with your formatters (if any). When you know the problem, you can perform the necessary corrections.

SEE ALSO:

• behave repo:features/formatter.help.feature (BAD CASES)

IN ADDITION:

• behave.ini is normally not in the features/ directory, but one directory above under proj1.

[athiradamodaran]

I upgraded to 1.2.7.dev3 and used behave -f help but it still gives the list of formatters:

Anyway, I moved the behave.ini to the project folder and now it is reading the configuration. However, my report is printed in text format like shown below:

What can I do to get the previous format of report?

My behave.ini settings are:

[behave.formatters]
html-pretty = behave_html_pretty_formatter:PrettyHTMLFormatter

[behave.userdata]
runner.continue_after_failed_step = true
behave.formatter.html-pretty.title_string = Behave Test Suite
behave.formatter.html-pretty.date_format = %%d-%%m-%%Y %%H:%%M:%%S (%%s)

command used to run -

behave -f pretty -o <report-path> <features>  --no-capture

When I give the command, I get the below error:

behave -f html-pretty -o <report-path> <features>  --no-capture
usage: behave [options] [ [DIR|FILE|FILE:LINE] ]+
behave: error: BAD_FORMAT=html-pretty (problem: LookupError)

[jenisys]

• Does the format html-pretty show up if you run behave -f help ?
• If it does not, the format(ter) html-pretty is not registered (meaning your behave.ini is not found)
• LookupError means there is no such format(ter) registered.

ADVICE:

• Go into the directory where behave.ini is and run behave from there.
• Move behave.ini to the current directory where you try to run behave (and run it again)

[fpokryvk]

Hi, it seems, the webserver is sending HTTP header which forbids inline CSS and javascript (default security measure nowadays). I guess the easiest option is to open file locally (by double click in the folder). Or search how to change CSP header (Content Security Policy) in the application you use, into something less restrictive.

We had to adjust CSP in jenkins, where we store the HTML logs, in order to render them properly. That is the price to paid for having self-contained HTML logs.

[modehnal]

I believe this issue can be closed as there was not an issue in the formatter but in the user setup.

Feel free to open another issue if there is something we can help with.