Open carllp opened 4 years ago
This doesn't look like a problem related to the handlebars template. The error is related to the scanner not being able to publish the result files. Normally this happens on a self-hosted agent that has a version of docker installed via the SNAP package manager. Are you by any chance running a similar self-hosted agent @carllp ?
@doymturner The agent the task is being run on is the Ubuntu 18.04 Microsoft-hosted agent.
Can you share your full pipeline (cleansed)? At least all of the bits related to zap scanner?
Sure. I believe it's been sufficiently cleansed of all unique identifiers.
As a side note, if you could point me to some documentation (via DM or here) to speed up the cleansing of pipeline logs it would be much appreciated.
Took a second look at the original error you posted. Did you make any changes to the handlebar template when adding it to your pipeline? Could you share the bits of your pipeline YAML related to the extension and report publishing (including the template)?
FYI: We are working on additional reporting options for the extension, but plan to leave the options for custom reporting such as with the handlebars template so I'm keen to figure out the issue.
Sorry for the late reply @doymturner.
Fixed quoted text below by including only the bash scripting code and not displayName and condition lines. Please ignore.
We did edit the bash script convert report.json to test-results.xml slightly as it was throwing errors (see log below). I assume that this is the culprit as to why the nunit test report doesn't work. The two removed lines work fine in the handlebars template script so we're not sure why they throw errors in this step.
We removed the following two lines so that the conversion would run: displayName: 'owasp nunit template' condition: always()
The error report if the lines are included:
========================================================= Generating script. ========================== Starting Command Output ==== /home/vsts/work/_temp/: line 2: displayName:: command not found /home/vsts/work/_temp/: line 3: syntax error near unexpected token
(' /home/vsts/work/_temp/: line 3:
condition: always()'[error]Bash exited with code '2'.
[section]Finishing: Bash Script - Convert report.json to test-results.xml
Here's how it looks in our pipeline. If you know what's syntactically wrong, please let me know.
Appreciate your help with this.
As requested, 1st snippet is the handlebars YAML and the 2nd is the report conversion. 3rd is the Publish Test Results YAML.
steps:
bash: | sudo npm install -g handlebars-cmd
cat <
<test-run id="2" name="Owasp test" start-time="{{../[@generated]}}" > <test-suite id="{{@index}}" type="Assembly" name="{{[@name]}}" result="Failed" failed="{{alerts.length}}">
{{#each alerts}}<test-case
id="{{@index}}"
name="{{alert}}"
result="Failed"
fullname="{{alert}}"
time="1">
<failure>
<message>
<![CDATA[{{{desc}}}]]>
</message>
<stack-trace>
<![CDATA[
Solution: {{{solution}}}
Reference: {{{reference}}}
instances:{{#each instances}}
steps:
bash: | handlebars owaspzap/report.json < owaspzap/nunit-template.hbs > owaspzap/test-results.xml
displayName: 'Bash Script - Convert report.json to test-results.xml'
steps:
Error in the publish test results task is slightly different now.
It now shows:
Instead of /home/vsts/work/r1/a/owaspzap/test-results.xml. Error : Data at the root level is invalid. Line 22, position 4..
Full log:
` ##[section]Starting: Publish Test Results test-results.xml
Task : Publish Test Results Description : Publish test results to Azure Pipelines Version : 2.161.1 Author : Microsoft Corporation Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/test/publish-test-results
[command]/usr/bin/dotnet --version 3.0.101
`
I have the same issue as carllp and I have solved it by creating whole task in yaml instead of DevOps separate components.
@carllp Take a look in your test-results.xml file (copy it to artifacts so you can download it), it likely has 2 <test-run>
nodes which is invalid xml format, and Nunit only allows one per file
I've opened an issue for it as the script itself is written to do this specifically for each site that is scanned, in my case it was http vs https
In my case, this was resolved by moving my final EOF over two spaces. In the spacing adjustments in the YAML file it became offset thus causing a problem as shown here. Once it was aligned, the warning didn't show up anymore. To be clear - the task was succeeding but a warning was being output on the PublishTestResults task.
@carllp Take a look in your test-results.xml file (copy it to artifacts so you can download it), it likely has 2
<test-run>
nodes which is invalid xml format, and Nunit only allows one per fileI've opened an issue for it as the script itself is written to do this specifically for each site that is scanned, in my case it was http vs https
So how did you resolve the issue? NM. Thanks!
The sample shows the following
sudo npm install -g handlebars-cmd
cat <
... snip
EOF
When running in a bash task this causes failure. Fix by moving everything from cat to EoF two spaces to the left.
hi I am getting same issue now. I have tried all the options but nothing works (yaml,azure devops tasks) , removing EOF and other stuff.. Result Attachments will be stored in LogStore Run Attachments will be stored in LogStore
Async Command Start: Publish test results Async Command End: Publish test results Finishing: Publish Test Results owaspzap/test-results.xml
I am not able to see test tab eventhough build is successful
trigger:
pool: vmImage: 'ubuntu-latest'
steps:
task: owaspzap@1 inputs: threshold: '150' scantype: 'targetedScan' url: 'example.com' port: '443'
task: CopyFiles@2 inputs: SourceFolder: 'owaspzap/' Contents: '**' TargetFolder: '$(Build.ArtifactStagingDirectory)'
task: PublishBuildArtifacts@1 condition: always() inputs: ArtifactName: 'owasp_zap_reports'
bash: | sudo npm install -g handlebars-cmd
cat <
<test-run id="2" name="Owasp test" start-time="{{@generated}}"> {{#each site}}<test-suite id="{{@index}}" type="Assembly" name="{{[@name]}}" result="Failed" failed="{{alerts.length}}">
{{#each alerts}}<test-case
id="{{@index}}"
name="{{alert}}"
result="Failed"
fullname="{{alert}}"
time="1">
<failure>
<message>
<![CDATA[{{{desc}}}]]>
</message>
<stack-trace>
<![CDATA[
Solution: {{{solution}}}
Reference: {{{reference}}}
instances:{{#each instances}}
bash: ' handlebars owaspzap/report.json < owaspzap/nunit-template.hbs > owaspzap/test-results.xml' displayName: 'generate nunit type file' condition: always()
task: PublishTestResults@2 inputs: testResultsFormat: 'Nunit' testResultsFiles: 'owaspzap/test-results.xml'
Can anyone help me
I've found solution for this, I'll just paste it from marketplace reviews.
Sithu Kyaw 9/23/2020 I tried it just now and it works fine except not able to attach "owaspzap/report.html" for Publish Test Result task.
After a couple of troubleshooting, I found Handlebars script needs minor adjustment by adding $(System.DefaultWorkingDirectory)/ in front of 'owasp/report.html' inside
<attachment>
<filePath>$(System.DefaultWorkingDirectory)/owaspzap/report.html</filePath>
</attachment>
In Step 1 - Remove the EOF statement at the end. Keep only the below code. It will work fine
sudo npm install -g handlebars-cmd
cat <<EOF > $(System.DefaultWorkingDirectory)/owaspzap/nunit-template.hbs
{{#each site}}
<test-run
id="2"
name="Owasp test"
start-time="{{../[@generated]}}" >
<test-suite
id="{{@index}}"
type="Assembly"
name="{{[@name]}}"
result="Failed"
failed="{{alerts.length}}">
<attachments>
<attachment>
</attachment>
</attachments>
{{#each alerts}}<test-case
id="{{@index}}"
name="{{alert}}"
result="Failed"
fullname="{{alert}}"
time="1">
<failure>
<message>
<![CDATA[{{{desc}}}]]>
</message>
<stack-trace>
<![CDATA[
Solution:
{{{solution}}}
Reference:
{{{reference}}}
instances:{{#each instances}}
* {{uri}}
- {{method}}
{{#if evidence}}- {{{evidence}}}{{/if}}
{{/each}}]]>
</stack-trace>
</failure>
</test-case>
{{/each}}
</test-suite>
</test-run>
{{/each}}
I have had the same problem, tried all different possibilites. But always get error file not found..
##[warning]File at
$(Build.SourcesDirectory)/OWASP-ZAP-Report.xml
was not found
Failed to upload run logs to LogStore, Trying to upload to FileService
##[warning]Skipping attachment as it is not available on disk:
$(Build.SourcesDirectory)/OWASP-ZAP-Report.xml
also by injecting env variable to xslt
##[warning]File at /home/vsts/work/1/s/OWASP-ZAP-Report.xml
was not found
Failed to upload run logs to LogStore, Trying to upload to FileService
##[warning]Skipping attachment as it is not available on disk: /home/vsts/work/1/s/OWASP-ZAP-Report.xml
also added echo to confirm the location of file
echo `pwd`/`ls OWASP-ZAP-Report.xml`
/home/vsts/work/1/s/OWASP-ZAP-Report.xml
In my case the pipeline runs successfully. However the Passed test results are not shown in the test report. I only see the failed or others. This is the template I am using.
stages:
stage: OWASP_ZAP_Stage # !CHANGE! per env jobs:
job : OWASP_ZAP
variables:
websiteurl: 'https://qatest.gdc.net/status'
pool:
vmImage: 'ubuntu-latest'
steps:
task: Bash@3 displayName: 'Prepare OWASPZAP output directory for Docker' inputs: targetType: 'inline' script: | sudo mkdir $(System.DefaultWorkingDirectory)/owaspzap sudo chmod -R 777 $(System.DefaultWorkingDirectory)/owaspzap
task: Bash@3 displayName: 'Run OWASPZAP Docker Image and scan $(websiteurl)' inputs: targetType: 'inline' script: | docker run -v $(System.DefaultWorkingDirectory)/owaspzap:/zap/wrk:rw -t owasp/zap2docker-stable zap-baseline.py -t $(websiteurl) -J report.json -i --autooff
bash: | sudo npm install -g handlebars-cmd
cat <
<test-run id="1" name="Owasp test" start-time="{{@generated}}"> {{#each site}}<test-suite id="{{@index}}" type="Assembly" name="{{[@name]}}" result="Failed" failed="{{alerts.length}}">
{{#each alerts}}<test-case
id="{{@index}}"
name="{{alert}}"
fullname="{{alert}}"
time="1">
<result>
<message>
<![CDATA[{{{desc}}}]]>
</message>
<stack-trace>
<![CDATA[
Solution: {{{solution}}}
Reference: {{{reference}}}
instances:{{#each instances}}
bash: ' handlebars owaspzap/report.json < owaspzap/nunit-template.hbs > owaspzap/test-results.xml' displayName: 'Generate NUnit type file for $(websiteurl)'
Hello everyone! I'm don't use this extension for ZAP testing, but got the same issue - unable to upload test results. So my pipeline yaml looks like this:
trigger: none
pr: none
pool:
vmImage: ubuntu-latest
steps:
- script: docker run --rm --user root -v $(System.DefaultWorkingDirectory):/zap/wrk/:rw -i owasp/zap2docker-stable zap-full-scan.py -d -j -I -t https://blah-blah -J owasp-zap/zap-report.json
displayName: 'Run dynamic testing'
- bash: |
sudo npm install -g handlebars-cmd
handlebars owasp-zap/zap-report.json < owasp-zap/nunit-template.hbs > owasp-zap/test-results.xml
python3 owasp-zap/splitter.py test-results.xml
displayName: 'Generate test result files'
- task: PublishTestResults@2
displayName: 'Publish rest results'
inputs:
testResultsFormat: NUnit
testResultsFiles: 'owasp-zap/test-results-*.xml'
mergeTestResults: true
So, at the root of my repo i had directory owasp-zap that contains file nunit-template.hbs:
{{#each site}}
<test-run
id="2"
name="Owasp test"
start-time="{{../[@generated]}}" >
<test-suite
id="{{@index}}"
type="Assembly"
name="{{[@name]}}"
result="Failed"
failed="{{alerts.length}}">
{{#each alerts}}<test-case
id="{{@index}}"
name="{{alert}}"
result="Failed"
fullname="{{alert}}"
time="1">
<failure>
<message>
<![CDATA[{{{desc}}}]]>
</message>
<stack-trace>
<![CDATA[
Solution:
{{{solution}}}
Reference:
{{{reference}}}
instances:{{#each instances}}
* {{uri}}
- {{method}}
{{#if evidence}}- {{{evidence}}}{{/if}}
{{/each}}]]>
</stack-trace>
</failure>
</test-case>
{{/each}}
</test-suite>
</test-run>
{{/each}}
And file splitter.py:
import xml.dom.minidom as minidom
import argparse
def parse_file():
parser = argparse.ArgumentParser(description='File path parser')
parser.add_argument('-f', '--file', type=str, help='A required string - path to report file.')
args = parser.parse_args()
return args.file
def get_xml(document: str) -> list:
with open(document, 'r') as file:
return [line for line in file]
def to_file(line: str, file_name: str):
with open(file_name, 'a') as f:
f.write(line)
def extract_separate_documents(lines: list):
count = 0
for line in lines:
line: str
if '<test-run' in line:
count += 1
print('report:', count)
if count > 0:
to_file(line, f'test-results-{count}.xml')
document = parse_file()
if not document:
print("Exception! Report file not specified.")
exit(1)
lines = get_xml(document=document)
extract_separate_documents(lines)
My ZAP testing run generates json file, that should be procced by handlebars tool and I got test-results.xml in owasp-zap dir. In my case test-results.xml file contains two
Hello,
Receiving the following error in the Publish Test Results task after successfully running both Bash scripts as found on your reporting documentation (https://marketplace.visualstudio.com/items?itemName=CSE-DevOps.zap-scanner). Test results are not shown on the test tab even though the task is marked as having succeeded.:
2019-12-05T14:36:03.5747957Z ##[debug]OS type: Linux 2019-12-05T14:36:04.4503434Z ##[debug]Reading test results from file '/home/vsts/work/r1/a/owaspzap/test-results.xml' 2019-12-05T14:36:04.4569568Z ##[warning]Failed to read /home/vsts/work/r1/a/owaspzap/test-results.xml. Error : Data at the root level is invalid. Line 22, position 4.. 2019-12-05T14:36:04.4591686Z ##[warning]Failed to publish test results: Object reference not set to an instance of an object. 2019-12-05T14:36:04.4717111Z ##[debug]Processed: ##vso[results.publish type=NUnit;mergeResults=false;publishRunAttachments=true;resultFiles=/home/vsts/work/r1/a/owaspzap/test-results.xml;failTaskOnFailedTests=false;testRunSystem=VSTS - PTR;] 2019-12-05T14:36:04.4718265Z ##[debug]task result: Succeeded 2019-12-05T14:36:04.4718634Z ##[debug]Processed: ##vso[task.complete result=Succeeded;]
As a possible alternative solution if handlebars is causing issues, perhaps outputting the test results file as an .xml and then using the template found on this azuredevblog might work?
https://devblogs.microsoft.com/premier-developer/azure-devops-pipelines-leveraging-owasp-zap-in-the-release-pipeline/
Thanks for your time! Appreciate your work.