Code Coverage view not rendering linked css / js files in artifact

[rrfenton]

I'm creating a javascript test coverage report using the karma-coverage plugin. It generates an HTML coverage report based on the Istanbul and a Cobertura report file.

I'm using the output directory for the html report as the report directory in the "Publish Code Coverage Results" task.

My build artifact then contains the full contents of the html folder, as well as the Cobertura report.

Build Artifact:

When I navigate to the Code Coverage page for the build, I see the results of my tests, and I'm able to navigate through the results, I just don't have any of the styling or scripts that are associated with the report.

Results:

Expected:

When I navigate to the coverage page in chrome, I get the following console error:

Blocked script execution in 'https://MYPROJECTHERE.visualstudio.com/_apis/Containers/321454/Code%20Coverage%20Report_2015/index.html' because the document's frame is sandboxed and the 'allow-scripts' permission is not set.

[nigurr]

@rrfenton, currently we have disabled the styling because of security constraints. team is working actively towards that. Thanks for the support

[nigurr]

@rrfenton it would be great if you can give some insight about Code Coverage usage. How often you generate CC HTML reports and what's the duration that you would like to store the data?

[beverts312]

@nigurr this is also a feature I would like to see (report is obviously much prettier with styling) In response to your questions directed to @rrfenton :
I generate a report on every CI build, I would like to be able to see the coverage statistics as long as I retain the build, however I would be satisfied with only being able to see the latest code coverage report (and maybe display it at the Build Definition level rather than the build level)

[rrfenton]

@nigurr Sorry for the delay, I had missed that you had asked some follow up questions.

We also generate a code coverage report on every build (for unit tests) / release (for integration tests). They are the primary testing artifact created along with the Cobertura report.

Ideally, retaining these reports should exist as long as the build artifact is available, as these reports will be used by the business to determine the validity of the build for release / promotion.

For our release flow, if we need to roll back to a previous release, it is beneficial for the results / coverage of these tests to be long running as their are the best identifier (beyond logging version numbers) as to what was a stable, valid release.

Thanks! Robert Fenton

[nigurr]

Thanks for the details. I will pass this information to Product team and trust me, we are trying to bring back the report with styles

[akankshagaur]

@rrfenton Can you tell me your PublishCodeCoverage task configuration. Its not able to find the path I gave. my structure is coverage/cobertura.xml for summary and coverage/html for report. Please help

[grantbowering]

I have problems displaying this type of report as well.
Like the OP, I'm using karma-coverage (or rather, isparta and karma-remap-istanbul, because typescript+webpack, but same result), generating a cobertura file to upload to VSTS as a summary and an html report to show on the code coverage tab.
Like the OP, the resulting artifact contains the full html folder nearly identical to theirs, and I can download that as a zip and extract it and see the coverage report... but unlike the OP, I don't even get anything displayed on my Code Coverage tab at ALL, just the link "Download Code Coverage Results". Clicking which gives me the .coverage file of the NUnit tests of the .NET classes, not anything to do with the client-side test coverage I also uploaded and included an attached html report:

Note that the Summary tab does show both sets, so it's definitely making use of the cobertura xml file, just not the html:

The log from the publish step doesn't indicate any problems either:

2017-01-14T00:46:34.0676187Z ##[section]Starting: Publish TS Code Coverage Results
2017-01-14T00:46:34.0676187Z ==============================================================================
2017-01-14T00:46:34.0676187Z Task         : Publish Code Coverage Results
2017-01-14T00:46:34.0676187Z Description  : Publish code coverage results to VSTS/TFS
2017-01-14T00:46:34.0676187Z Version      : 1.0.17
2017-01-14T00:46:34.0676187Z Author       : Microsoft Corporation
2017-01-14T00:46:34.0676187Z Help         : [More Information](https://go.microsoft.com/fwlink/?LinkID=626485)
2017-01-14T00:46:34.0676187Z ==============================================================================
2017-01-14T00:46:34.0676187Z Preparing task execution handler.
2017-01-14T00:46:34.4113949Z Executing the powershell script: C:\agent\_work\_tasks\PublishCodeCoverageResults_2a7ebc54-c13e-490e-81a5-d7561ab7cd97\1.0.17\PublishCodeCoverageResults.ps1
2017-01-14T00:46:34.6457503Z Starting 'Publish-CodeCoverage' cmdlet...
2017-01-14T00:46:35.2579513Z Publishing coverage summary data to TFS server.
2017-01-14T00:46:35.5392126Z Modifying Cobertura Index file
2017-01-14T00:46:35.5392126Z Publishing additional files to TFS server.
2017-01-14T00:46:35.8673343Z ##[section]Async Command Start: Upload Artifact
2017-01-14T00:46:35.8673343Z Uploading 173 files
2017-01-14T00:46:40.9441442Z Total file: 173 ---- Processed file: 64 (36%)
2017-01-14T00:46:48.0428676Z File upload succeed.
2017-01-14T00:46:48.0428676Z Upload 'C:\agent\_work\1\s\OnePlan-Front\Web\obj\coverage\html-report' to file container: '#/317683/Code Coverage Report_4585'
2017-01-14T00:46:48.6026504Z Associated artifact 2446 with build 4585
2017-01-14T00:46:48.6026504Z ##[section]Async Command End: Upload Artifact
2017-01-14T00:46:48.6026504Z ##[section]Finishing: Publish TS Code Coverage Results

I thought at first that maybe this tab just doesn't work with these kinds of results, but now that I've seen a screenshot of at least the text content of the report on the OP's code coverage tab, I'm wondering why I'm not enjoying the same output. Is it just a result of also having NUnit tests, and if so, how can I work around that, given that a huge amount of people must be in the same situation??

[jpsfs]

Hi!

I was also looking for the same feature described by @rrfenton. @nigurr Do you have any update on this?

Best,

[nigurr]

@jpsfs We have this work under our backlog and currently we don't have any timelines to share.

[yogendra-kaushik]

Hi @nigurr, same issue, please suggest if we have any available solution now ?

[nigurr]

You can download the coverage files and view it with rich HTML experience in offline. We still don't have timelines to share :(

[Yaash19]

@akankshagaur - any luck ? if it worked can you share how you did it ?

@rrfenton - can you share how you have setup PublishCodeCoverage task configuration , i have same setting as you. I have TFS 2015 and i am not even seeing "Code coverage" tab

my structure is coverage/cobertura.xml for summary and coverage/html for report. Please help

This is all i see in summary tab

[nigurr]

@Yaash19 This feature is not available to TFS OnPrem due to security constraints. We are working towards improving this experience and it's in our backlog

[shtratos]

@nigurr any news about bringing styles back?

[kohlikohl]

@nigurr We would also be interested in this feature.

[nigurr]

Hi,

@shtratos, @kohlikohl Please suggest/vote for this at https://visualstudio.uservoice.com/forums/330519-visual-studio-team-services We appreciate your feedback and please do consider that we prioritize based on feedback

[Blackbaud-TommyVernieri]

Direct link to a relevant User Voice item: https://visualstudio.uservoice.com/forums/330519-visual-studio-team-services/suggestions/15926806-code-coverage-report-on-build-should-not-strip-sty

[richardscholten73]

In Chrome the javascript files are also blocked. Blocked script execution in 'https://xxxx.visualstudio.com/xxxxx-e44870bd4e9b/_apis/test/CodeCoverage/browse/2095227/Code%20Coverage%20Report_114/index.html' because the document's frame is sandboxed and the 'allow-scripts' permission is not set.

Using karma-coverage-istanbul-reporter https://github.com/mattlewis92/karma-coverage-istanbul-reporter

[brownbl1]

We're also seeing this still. Would love to have scripts and css working.

[kohlikohl]

In the meantime, here is a little script that inlines the external CSS into the HTML pages generated by the coverage reporter. I run it as a post processing step in the build after the coverage has been collected.

https://gist.github.com/kohlikohl/ef77c751cfd3b731923ca74fec9443d5

[AndrewCraswell]

I'm also watching this. Without the CSS working, the viewer doesn't make much sense to us.

[brownbl1]

@kohlikohl Also, we tried the gist to inline the styles, and while we were able to generate the file, it still seems to be stripping out inline styles as well? Couldn't get it working that way either.

[AndrewCraswell]

@brownbl1, I've just managed to use @kohlikohl's gist successfully. It's currently my workaround to this issue. I've told the file to execute after my Jest unit tests, which produces the Istanbul code coverage HTML reports.

The results after uploading via the Publish Code Coverage Results task in VSTS. I suppose it might be behaving differently for you if you're on TFS or On Prem using an older version...

[shtratos]

I've got a Python 3.6 script that my team successfully uses for embedding CSS in code coverage reports. It depends on beautiful soup library for parsing HTML.

"""
Fixes VSTS coverage reports by inlining CSS files.
see https://github.com/Microsoft/vsts-tasks/issues/3027
Based on https://stackoverflow.com/a/23190429/7417402
"""
import os

import bs4

COVERAGE_REPORT_DIR = 'build/coverage/htmlcov/'
COVERAGE_REPORT = os.path.join(COVERAGE_REPORT_DIR, 'index.html')

def embed_css_in_html_file(html_file, css_dir):
    with open(html_file, 'r') as f:
        soup = bs4.BeautifulSoup(f.read(), "html.parser")

    stylesheets = soup.findAll("link", {"rel": "stylesheet"})
    for s in stylesheets:
        t = soup.new_tag('style')
        css_file = s["href"]
        print(f"found link to {css_file}")
        with open(os.path.join(css_dir, css_file), 'r') as f:
            c = bs4.element.NavigableString(f.read())
        t.insert(0, c)
        t['type'] = 'text/css'
        s.replaceWith(t)

    with open(html_file, 'w') as f:
        f.write(str(soup))

for file in os.listdir(COVERAGE_REPORT_DIR):
    if file.endswith(".html"):
        print(f"Embedding CSS in {file}")
        embed_css_in_html_file(os.path.join(COVERAGE_REPORT_DIR, file), COVERAGE_REPORT_DIR)

[kachkaev]

Thanks for the gist @kohlikohl! Here's my approach to the problem:

const fs = require("fs-extra");
const globby = require("globby");
const { resolve } = require("path");
const PQueue = require("p-queue");

(async () => {
  const rootDir = resolve(__dirname, "../../..");
  const queueConcurrency = 10;

  const cssToInject = (await Promise.all([
    fs.readFile(resolve(rootDir, "coverage/prettify.css"), "utf8"),
    fs.readFile(resolve(rootDir, "coverage/base.css"), "utf8"),
    `
body { font-family: "Segoe UI VSS (Regular)","Segoe UI","-apple-system",BlinkMacSystemFont,Roboto,"Helvetica Neue",Helvetica,Ubuntu,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol"; }
.wrapper { height: initial; background: rgba(248,248,248,1); }
.clearfix { display: inline-block; }
pre { background: #fff; padding: 10px 0; }
table { width: 1px; }
td { font-size: 14px; line-height: 18px; }
table.coverage { margin-top: 0px; }
pre, td.line-count, td.line-coverage, td.text { font-size: 12px; line-height: 16.8px; white-space: pre; }
    `,
  ])).join("\n");
  const queue = new PQueue({ concurrency: queueConcurrency });
  const coverageHtmlPaths = globby.sync("coverage/**/*.html", {
    pwd: rootDir
  });
  coverageHtmlPaths.forEach(coverageHtmlPath => {
    queue.add(async () => {
      coverageHtml = await fs.readFile(
        resolve(rootDir, coverageHtmlPath),
        "utf8"
      );
      const patchedCoverageHtml = coverageHtml
        .replace(/\n\s*<link rel="stylesheet" href=".*" \/>/g, "")
        .replace(/\n\s*<script(.|\n)*<\/script>/g, "")
        .replace(
          "<style type='text/css'>",
          `<style type='text/css'>\n${cssToInject}`
        );
      await fs.writeFile(
        resolve(rootDir, coverageHtmlPath),
        patchedCoverageHtml,
        "utf8"
      );
    });
  });
  await queue.onIdle();
})();

Pros:

• No dependency on inline-css, which makes a pretty significant impact on the size of node_modules
• Replacement rules are quite simple, which boosts the performance
• Promise queue makes the script scalable

Despite that this issue is VSTS-related, I believe that it needs to be solved on the istanbul side: https://github.com/gotwarlost/istanbul/issues/890

UPD: One more thing to watch out for: https://github.com/Microsoft/azure-pipelines-tasks/issues/9150 (partially solvable by adding .replace(/×/g, "x"))

[chris31389]

I followed this and it seemed to work really well 🥇 https://davidsekar.com/aspnetcore/code-coverage-html-reports-are-missing-styles-in-vsts

[davewilton]

I've got a Python 3.6 script that my team successfully uses for embedding CSS in code coverage reports. It depends on beautiful soup library for parsing HTML.

"""
Fixes VSTS coverage reports by inlining CSS files.
see https://github.com/Microsoft/vsts-tasks/issues/3027
Based on https://stackoverflow.com/a/23190429/7417402
"""
import os

import bs4

COVERAGE_REPORT_DIR = 'build/coverage/htmlcov/'
COVERAGE_REPORT = os.path.join(COVERAGE_REPORT_DIR, 'index.html')

def embed_css_in_html_file(html_file, css_dir):
    with open(html_file, 'r') as f:
        soup = bs4.BeautifulSoup(f.read(), "html.parser")

    stylesheets = soup.findAll("link", {"rel": "stylesheet"})
    for s in stylesheets:
        t = soup.new_tag('style')
        css_file = s["href"]
        print(f"found link to {css_file}")
        with open(os.path.join(css_dir, css_file), 'r') as f:
            c = bs4.element.NavigableString(f.read())
        t.insert(0, c)
        t['type'] = 'text/css'
        s.replaceWith(t)

    with open(html_file, 'w') as f:
        f.write(str(soup))

for file in os.listdir(COVERAGE_REPORT_DIR):
    if file.endswith(".html"):
        print(f"Embedding CSS in {file}")
        embed_css_in_html_file(os.path.join(COVERAGE_REPORT_DIR, file), COVERAGE_REPORT_DIR)

This is great and worked well for me. Except the encoding had to be set on the output otherwise I ended up with encoding issues on the nbsp generated by coverage with open(html_file, 'w', encoding="UTF8") as f:

and just to make it easier for anyone else here is my yaml

- bash: |
    pip install beautifulsoup4
    python emded_css_for_coverage.py
  displayName: 'Embed css'
  workingDirectory: 'Toolbox'

- task: PublishCodeCoverageResults@1
  condition: succeededOrFailed()
  inputs:
    codeCoverageTool: Cobertura
    summaryFileLocation: '$(System.DefaultWorkingDirectory)/**/coverage.xml'
    reportDirectory: '$(System.DefaultWorkingDirectory)/**/htmlcov'

[vutkin]

Hi, install NuGetToolInstaller@0 task first.

here is my azure-pipeline template that works like a charm:

parameters:
  reportDirectory: $(Build.SourcesDirectory)/CodeCoverage
  reportFile: 'coverage.cobertura.xml'

steps:
- task: DotNetCoreCLI@2
  displayName: Install ReportGenerator Global Tool
  inputs:
    command: custom
    custom: tool
    arguments: install dotnet-reportgenerator-globaltool --tool-path .

- script: |
    ./reportgenerator \
        "-reports:**/${{ parameters.reportFile }}" \
        "-targetdir:${{ parameters.reportDirectory }}" \
        "-reporttypes:HtmlInline_AzurePipelines;Cobertura" \
        "-tag:$(Build.BuildNumber)"
  displayName: Generate Code Coverage HTML Report

- task: PublishCodeCoverageResults@1
  displayName: Publish Code Coverage Results
  inputs:
    codeCoverageTool: cobertura
    summaryFileLocation: ${{ parameters.reportDirectory }}/Cobertura.xml
    reportDirectory: ${{ parameters.reportDirectory }}
  condition: succeededOrFailed()

[vcliment89]

Had to change it a bit since it was still not fully working for me with latest Angular app.

"""
Fixes VSTS coverage reports by inlining CSS files.
see https://github.com/Microsoft/vsts-tasks/issues/3027
Based on https://stackoverflow.com/a/23190429/7417402
"""
import os

import bs4

COVERAGE_REPORT_DIR = './coverage/'

def embed_css_in_html_file(html_file, css_dir):
    with open(html_file, 'r') as f:
        soup = bs4.BeautifulSoup(f.read(), "html.parser")

    stylesheets = soup.findAll("link", {"rel": "stylesheet"})
    for s in stylesheets:
        t = soup.new_tag('style')
        css_file = s["href"]
        print(f"found link to {css_file}")
        with open(os.path.join(css_dir, css_file), 'r') as f:
            c = bs4.element.NavigableString(f.read())
        t.insert(0, c)
        t['type'] = 'text/css'
        s.replaceWith(t)

    with open(html_file, 'w', encoding="UTF8") as f:
        f.write(str(soup))

for dp, dn, fn in os.walk(os.path.expanduser(COVERAGE_REPORT_DIR)):
  for file in fn:
    if file.endswith(".html"):
        print(f"Embedding CSS in {file}")
        embed_css_in_html_file(os.path.join(
            dp, file), dp)

Then in my Azure pipeline used it like this

- task: UsePythonVersion@0
    displayName: 'Use Python 3.x'

- bash: |
    pip install beautifulsoup4
    python tools/fix-vsts-coverage-styles.py
  displayName: 'Fix Code Coverage HTML Report'

[soham-dongargaonkar]

Any updates on this?

[sladg]

Why is the issue closed? It is currently not working.

[alenzo-pinc]

I am also having issues with pytest coverage. Coverage tab is not rendering with CSS

[kotborealis]

Any updates on this? Inlining CSS and scripts seems like a wrong solution, and I would like to avoid it, as it duplicates a lot of resources.

[GaTechThomas]

IIRC, this limitation is by design to avoid various security issues. If you want to avoid duplicating the resources, then creating a custom extension may be a better option. However, if it's a large amount of resources repeatedly copied then it may be an indicator that ADO is being used for something beyond ADO's intent. I've seen much of such misuse at work, and it makes for problems down the road if the approach is not quickly changed to be done in a more measured way.