DefectDojo SonarQube Integration

[rjimgal]

First of all, thank you for this amazing tool, I'm just starting to use it.

I would like to use this recent Integration, however I am unable to generate HTML report in SonarQube.

Sample HTML file can be obtained at https://github.com/DefectDojo/sample-scan-files/blob/master/sonarqube/sonarqube_v6.7.5.html and it works like a charm.

I am also using SonarQube 6.7.5 version, on its Community Edition.

Having a look to HTML sample code, it contains some banner about @AdrienGuillerme employer, which leads me to think, that provided HTML report is an internal report generated by them. If so, would you mind to share how you generate it? Maybe some template / SonarQube API calls.

Thanks in advance.

[AdrienGuillerme]

I will take a look at it

[aaronweaver]

Hi @AdrienGuillerme just wanted to checkin to see how the file is generated. Thanks!

[AdrienGuillerme]

The tool used to generate the HTML is an internal tool. I am waiting for an official answer from my company.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[rjimgal]

Hi @AdrienGuillerme, any update on this?

[W1R3D-Code]

also interested, for what it's worth :)

[agigleux]

Hello,

I'm working for SonarSource the company making SonarQube. We are looking at the options to provide an official integration with DefectDojo that will be supported by SonarSource.

Do you know if DefectDojo is supporting a generic import format? Or shall we define our own export format and provide the importer in the DefectDojo repo?

Thanks

[aaronweaver]

@agigleux That's fantastic news, we do have a generic csv or if it's easier you can define a json export and we can create it. Thanks!

[agigleux]

@aaronweaver I tried to follow what is documented here and generate a file corresponding to it but no luck so far. Can you provide a sample CSV file so I compare with what I generated?

I used https://defectdojo.herokuapp.com/ and selected "Generic Findings Import". Do you confirm this instance allows upload of files, maybe that's the reason why I can't import it?

[aaronweaver]

I'll take a look and get back to you shortly. Thanks!

[aaronweaver]

Hi @agigleux, I updated the csv field here. If you need any additional fields let me know and I can add that to this importer. Thx

[agigleux]

I'm able to use the "Generic Findings Import" thanks to your sample.

I have a couple of remarks:

• the generic CSV format doesn't allow to upload Findings with Active=True and Verified=True: once the import is done, even by setting these columns to "True", the findings are always Inactive and Verified is unticked. Is that expected? In that case, what's the point to have these columns in the CSV format?
• the CSV format is import Findings as "Dynamic finding" by default and it's not possible to specify that our Findings are generated by our SAST feature - so I can't set by default "Static finding" on Findings.
• is there any field in DefectDojo where we can feed the Vendor URL where the user can look at the findings in the Vendor's UI? SonarQube is providing the full execution flow steps, and I believe it would be valuable to provide a direct access to such information from DefectDojo.
• this is not related to this CSV import, but I'm very surprised it's possible to edit the Hash of a Finding - I believe this is what is used to guess that a Finding has been already imported and avoid too many duplicated entries. If I'm correct, this Hash data should be hidden.
• Findings imported as visible as "Found by Generic Findings Import" : it means SonarQube name is not visible and that's a problem from a product promotion point of view, so I'm thinking more and more that we will have to define our own JSON format and have a dedicated importer developed.

Thanks

[aaronweaver]

Hi @agigleux, Thanks for the comments. I'll file a bug report and work on those. You are correct hash shouldn't be editable. It would probably be a better integration with a custom importer via json. I agree that deep linking to the issue would be important as well.

[jpescalona]

@agigleux, regarding your last point, I've been working on some improvements under the PR https://github.com/DefectDojo/django-DefectDojo/pull/843, many of them were basically extending the API functionalities to enable CRUD for some models, but the one of the most important features implemented on the API was to be able import findings specifying differents test_type and scan_type, for example, one of the problem we handled was we wanted to import findings using the Generic Findings Import, but we wanted to create a SonarQube test type instead of Generic Findings Import. I guess this feature can be also extended for any other kind of test_type, regardless of the scan_type you want to use. These feature was implemented on the API only, i do not recall if this is available through the GUI.

I'm able to use the "Generic Findings Import" thanks to your sample.

I have a couple of remarks:

• the generic CSV format doesn't allow to upload Findings with Active=True and Verified=True: once the import is done, even by setting these columns to "True", the findings are always Inactive and Verified is unticked. Is that expected? In that case, what's the point to have these columns in the CSV format?
• the CSV format is import Findings as "Dynamic finding" by default and it's not possible to specify that our Findings are generated by our SAST feature - so I can't set by default "Static finding" on Findings.
• is there any field in DefectDojo where we can feed the Vendor URL where the user can look at the findings in the Vendor's UI? SonarQube is providing the full execution flow steps, and I believe it would be valuable to provide a direct access to such information from DefectDojo.
• this is not related to this CSV import, but I'm very surprised it's possible to edit the Hash of a Finding - I believe this is what is used to guess that a Finding has been already imported and avoid too many duplicated entries. If I'm correct, this Hash data should be hidden.
• Findings imported as visible as "Found by Generic Findings Import" : it means SonarQube name is not visible and that's a problem from a product promotion point of view, so I'm thinking more and more that we will have to define our own JSON format and have a dedicated importer developed.

Thanks

[agigleux]

Hello @aaronweaver,

Finally SonarSource won't have the possibility to add a dedicated API to extract the data for DefectDojo. Instead, we would like to try to reuse the issues/search API that is used to display the issues in SonarQube and also to extract issues in JSON format.

I wrote the specification of the mapping between SonarQube and DefectDojo here: https://jira.sonarsource.com/browse/MMF-1672

Can you take the lead on it and implement a parser.py dedicated to the SonarQube output? I know that some data are missing like the CWE-ID but we can think about adding them later.

Thanks

[ptrovatelli]

@agigleux @aaronweaver @AdrienGuillerme sorry it didn't come earlier, I was able to get the agreement from our hierarchy to disclose the sonar export source code. The code is in javascript; it's depoyed as an npm package It's using sonar API to get the required information, notably

• /api/rules/search : get the rules
• /api/issues/search : get the vulnerabilities

It exports the list of vulnerabilities and rules to html, which can be parsed by the sonarQube parser that was previously commited to DefectDojo. Not all the fields from https://jira.sonarsource.com/browse/MMF-1672 are implemented yet but it can be a starting point. Currently we have for each vuln:

• Rule
• Severity
• Description
• Message

Are you still interested? The main difference with what Alexandre proposed is that we export html instead of json. It might be cleaner to use json, however it's handy to have a human readable format too. I don't think there's any free plugin to export to html from sonar. Last time I did that was with sonar preview mode - which may still work although it's been deprecated for some time. Here's what the export looks like:

[ptrovatelli]

Hi, I've released our sonar html export code here: https://github.com/soprasteria/sonar-report Feedback is welcome

[praveendvd]

Hi, I've released our sonar html export code here: https://github.com/soprasteria/sonar-report Feedback is welcome

Hi @ptrovatelli @aaronweaver i can't find SOnarQube in my supported scan type, how to get it.

{"importscan": {"scan_type": ["scan_type must be one of the following: , Netsparker Scan, Burp Scan, Nessus Scan, Nmap Scan, Nexpose Scan, AppSpider Scan, Veracode Scan, Checkmarx Scan, Crashtest Security Scan, ZAP Scan, Arachni Scan, VCG Scan, Dependency Check Scan, Retire.js Scan, Node Security Platform Scan, NPM Audit Scan, Qualys Scan, Qualys Webapp Scan, OpenVAS CSV, Snyk Scan, Generic Findings Import, Trustwave Scan (CSV), SKF Scan, Clair Klar Scan, Bandit Scan, SSL Labs Scan, Acunetix Scan, Fortify Scan, Gosec Scanner, MobSF Scan, Trufflehog Scan, Nikto Scan, Clair Scan, Brakeman Scan, SpotBugs Scan, AWS Scout2 Scan, AWS Prowler Scan, IBM AppScan DAST, PHP Security Audit v2, PHP Symfony Security Check, Safety Scan, DawnScanner Scan, Anchore Engine Scan, Bundler-Audit Scan, Twistlock Image Scan, Kiuwan Scan, Blackduck Hub Scan, Openscap Vulnerability Scan, Wapiti Scan, Immuniweb Scan, Sonatype Application Scan, Cobalt.io Scan, Mozilla Observatory Scan, Whitesource Scan, Contrast Scan, Microfocus Webinspect Scan, Wpscan, Sslscan, JFrog Xray Scan, Sslyze Scan, Testssl Scan"]}}

i am using following defectdojo code:

908118a: Merge pull request #1412 from Maffooch/dev [2019-07-30 08:47:59 -0500]

[ptrovatelli]

@praveendvd good question :) I need to re-activate it. Right now it's just commented out. I was waiting for some reactions on this thread, if it's fine by everyone I will reactivate it in dev branch and throw in a bit of documentation

[praveendvd]

@praveendvd good question :) I need to re-activate it. Right now it's just commented out. I was waiting for some reactions on this thread, if it's fine by everyone I will reactivate it in dev branch and throw in a bit of documentation

Hi @ptrovatelli thanks for your huge effort, the report looks great but i am facing one problem. I am getting only those bugs which have a vulnerability tag.

SonarQube with filter:

SonarQube without filter:

Sonar-html Report:

The html report shows only the 4 vulnerabilities , is it the expected behavior ?. Could you please let me know about any work around to make full report ?.

Thank you so much @ptrovatelli

[ptrovatelli]

@praveendvd thank you; I didn't code it myself though :) about your question, that project was done specifically to export vulnerabilities, not code smells and other things you'll find in sonar. If you want to export everything, you can tune the filters and remove "types=VULNERABILITY" (see index.js).

[praveendvd]

@praveendvd thank you; I didn't code it myself though :) about your question, that project was done specifically to export vulnerabilities, not code smells and other things you'll find in sonar. If you want to export everything, you can tune the filters and remove "types=VULNERABILITY" (see index.js).

@ptrovatelli !!! Thank you so much, it worked perfectly. Does defectdojo sonarqube integration supports uploading the entire result like the one i have provided below?

For any one who wants to do the same , below are the steps :

1) search for the sonar-report installation :

sudo find / -name sonar-report

2) navigate to the lib directory from the search result

cd <path>/node-v10.16.0-linux-x64/lib/node_modules/sonar-report

3) Edit the index.js

vi index.js

4) search for 'types=VULNERABILITY &' and remove it

Now run the command :

[ptrovatelli]

@praveendvd yes it should work as well. right now it's broken on the dev branch due to python3; i'm looking into it. If you're using a python2 branch (master or legacy-python2.7) it should work after uncommenting ("SonarQube Scan", "SonarQube Scan"), in dojo/forms.py (and rebuilding the images if using docker)

[praveendvd]

@praveendvd yes it should work as well. right now it's broken on the dev branch due to python3; i'm looking into it. If you're using a python2 branch (master or legacy-python2.7) it should work after uncommenting ("SonarQube Scan", "SonarQube Scan"), in dojo/forms.py (and rebuilding the images if using docker)

I am getting below error for all the uploads in master branch through importscan API. It seems like master branch upload api is broken.

In dev branch everything is working fine

.

Will wait for the dev branch fix :) thank you @ptrovatelli

[ptrovatelli]

@praveendvd I have just made a PR with the fix for python3 and other improvements. See https://github.com/DefectDojo/django-DefectDojo/pull/1445 You'll be glad to see that I have also added a parameter to export all sonar bugs, not only vulnerabilities in the sonar-report export code :) Last version can be found here https://github.com/soprasteria/sonar-report (the npm registry is not up to date yet)

[ptrovatelli]

@praveendvd feel free to try this PR too that brings a different approach, with a 2-way integration with snoarqube: https://github.com/DefectDojo/django-DefectDojo/pull/1444

[praveendvd]

@praveendvd feel free to try this PR too that brings a different approach, with a 2-way integration with snoarqube:

1444

sure @ptrovatelli will give it a look and try to give you the feed back at the earliest as i can. Thank you so much.

[praveendvd]

@praveendvd feel free to try this PR too that brings a different approach, with a 2-way integration with snoarqube:

1444

sure @ptrovatelli will give it a look and try to give you the feed back at the earliest as i can. Thank you so much.

For #1445 :

Hi @ptrovatelli ,

I cloned and build the below repository:

https://github.com/twsagarcia/django-DefectDojo.git

and did below checkout:

git checkout reactivate-sonar-2

and installed new sonar-report using below command:

npm install -g https://github.com/soprasteria/sonar-report.git

I was able to upload default report which takes in only vulnerabilities:

File used :

sonar-report_sonar-report.html.zip

Output:

But for full report , it throws parsing error:

File used:

sonar-report_sonar-report_Full.html.zip

Output:

Below are the commands i used to create reports:

Vulnerability report:

sonar-report --sonarurl="http://localhost:9000" / --sonarcomponent="Juice" / --project="Sonar Report" / --application="sonar-report" / --release="1.0.0" / --sinceleakperiod="false" / --allbugs="false" > /home/praveen/Downloads/sonar-report_sonar-report_Full.html

Full Report:

sonar-report --sonarurl="http://localhost:9000" / --sonarcomponent="Juice" / --project="Sonar Report" / --application="sonar-report" / --release="1.0.0" / --sinceleakperiod="false" / --allbugs="true" > /home/praveen/Downloads/sonar-report_sonar-report_Full.html

[ptrovatelli]

@praveendvd the parser wasn't meant to import non-vuln bugs. there may be a field missing on those. i reproduced with the tomcat report; i'll have a look.

[praveendvd]

@praveendvd the parser wasn't meant to import non-vuln bugs. there may be a field missing on those. i reproduced with the tomcat report; i'll have a look.

Thank you so much @ptrovatelli , i also checked the second solution you suggested #1444

but i am getting below error message

my sonar server is at localhost:9000 and my DefectDojo is at localhost:8000

i also tried giving url field in the tool configuration as 'http://localhost:9000/api' but still gets same error.

I am able to access the http://localhost:9000/api/projects/search?q=Juice , through browser

I tried with both API and username/password authentication

[ptrovatelli]

@praveendvd I haven't tried this one yet. have you tried the last version? (last commit 3 days ago). If it's still not working i advise to continue the discussion here https://github.com/DefectDojo/django-DefectDojo/pull/1444 or on slack if @twsagarcia is there

[praveendvd]

@ptrovatelli

I tested your fix , it works perfectly. I was able to upload both the vulnerability and full report. I will try with API on monday. Thank you so much for the help

[ptrovatelli]

sonarqube was reactivated in dev branch: see https://github.com/DefectDojo/django-DefectDojo/pull/1445 There is also another solution in development with 2-way integration with sonar: see https://github.com/DefectDojo/django-DefectDojo/pull/1444