Error when uploading a .CSV-file using importScan API

[davidabayon]

Issue Type

• [x] Bug
• [ ] Enhancement

Description

The importScan API has return this error message on the response when we run the .jar file we created: Server returned HTTP response code: 500 for URL: http://[Server address]/api/v1/importscan/

So after encountering the error we tried posting the request directly to DefectDojo. We are using the importScan API to import a scanned report (.csv file) from OpenVAS but this error appeared when we tried posting it through the DefectDojo testing of API.

Operating System

Linux: Ubuntu 16.04

Install

• [x] setup.bash
• [ ] docker

Steps to Reproduce

To detail out the steps we did to come up with the error: 1. Have an OpenVAS CSV file in directory (sample file name: report-be9610c8-ddd2-4c25-af18-2b53ad5a2e18.csv)

2. Build the request for the importScan API: (Sample request) { "minimum_severity" : "Info", "scan_date" : "2018-03-22", "verified" : false, "file" : "/home/user/report-be9610c8-ddd2-4c25-af18-2b53ad5a2e18.csv", "lead" : "/api/v1/users/8/", "tags" : "", "active" : false, "engagement" : "/api/v1/engagements/62/", "resource_uri" : "/api/v1/engagements/62/", "scan_type" : "OpenVAS CSV" }

3. Post the request

4. This response body was given by DefectDojo: { "error_message": "'unicode' object has no attribute 'read'", "traceback": "Traceback (most recent call last):\n\n File \"/home/cltungcul/appsec/defectdojo/dojoenv/local/lib/python2.7/site-packages/tastypie/resources.py\", line 221, in wrapper\n response = callback(request, *args, kwargs)\n\n File \"/home/cltungcul/appsec/defectdojo/dojoenv/local/lib/python2.7/site-packages/tastypie/resources.py\", line 461, in dispatch_list\n return self.dispatch('list', request, kwargs)\n\n File \"/home/cltungcul/appsec/defectdojo/dojoenv/local/lib/python2.7/site-packages/tastypie/resources.py\", line 493, in dispatch\n response = method(request, kwargs)\n\n File \"/home/cltungcul/appsec/defectdojo/dojoenv/local/lib/python2.7/site-packages/tastypie/resources.py\", line 1401, in post_list\n updated_bundle = self.obj_create(bundle, self.remove_api_resource_names(kwargs))\n\n File \"/home/cltungcul/appsec/defectdojo/django-DefectDojo/dojo/api.py\", line 1153, in obj_create\n parser = import_parser_factory(bundle.data['file'], t)\n\n File \"/home/cltungcul/appsec/defectdojo/django-DefectDojo/dojo/tools/factory.py\", line 65, in import_parser_factory\n parser = OpenVASUploadCsvParser(file, test)\n\n File \"/home/cltungcul/appsec/defectdojo/django-DefectDojo/dojo/tools/openvas_csv/parser.py\", line 313, in init\n content = filename.read()\n\nAttributeError: 'unicode' object has no attribute 'read'\n" }

Screenshots

The DefectDojo showing the error when using the importScan API to import the OpenVAS CSV file . . . This is the screenshot of our own logs using the .jar file

[aaronweaver]

@davidabayon two questions.

1. Does this work if you manually upload the file through the web interface?
2. Would you be able to provide me with a sanitized version of the file? I'd only need a couple rows in the file.

Thanks!

[davidabayon]

Hi @aaronweaver , yes, the .csv file gets uploaded successfully when we upload it through the web interface. Please take note that:

• when we upload the .csv file through API, it is placed in the server (Ubuntu 16.04)
• when we upload the .csv file through web interface, it is located in the local server (Windows 8)

I'll work on the file and check which data can be removed for the safety of the data

[davidabayon]

Hi @aaronweaver , here's the sample .csv file Sample .CSV File.zip

I tried removing most of the data and left some of the things I needed to run the .jar file, but

• with the .csv file being almost empty and
• I have checked if there's some special character that could be causing the "error_message": "'unicode' object has no attribute 'read'",

the same error message still appears, as shown in the screenshot below...

Screenshot of error from DefectDojo

[aaronweaver]

@davidabayon I tried it on the latest version of DefectDojo and it works. Here is the request if you'd like to try it on your instance or test environment.

Just substitute your API key and engagement id. I verified that it correctly submits the issue.

POST /api/v1/importscan/ HTTP/1.1 Host: localhost:8000 Connection: close Accept-Encoding: gzip, deflate Accept: / User-Agent: DefectDojo_api/1.1.3 Authorization: ApiKey Content-Length: 1368 Content-Type: multipart/form-data; boundary=3f3faabb083d4d24bf696e1d859f2c87

--3f3faabb083d4d24bf696e1d859f2c87 Content-Disposition: form-data; name="build_id"; filename=""

test --3f3faabb083d4d24bf696e1d859f2c87 Content-Disposition: form-data; name="minimum_severity"; filename=""

Info --3f3faabb083d4d24bf696e1d859f2c87 Content-Disposition: form-data; name="scan_date"; filename=""

2018-03-22 --3f3faabb083d4d24bf696e1d859f2c87 Content-Disposition: form-data; name="file"; filename="openvas_2018-01-31-17-08-15_449fc880-58be-434d-b224-cbe9cf0a0002.csv"

IP,Hostname,Port,Port Protocol,CVSS,Severity,Solution Type,NVT Name,Summary,Specific Result,NVT OID,CVEs,Task ID,Task Name,Timestamp,Result ID,Impact,Solution,Affected Software/OS,Vulnerability Insight,Vulnerability Detection Method,Product Detection Result,BIDs,CERTs,Other References 9.9.9.9,www.example.com,80,,,,High,,Test,,,,,PRODUCT_TEST_ONLY,2018-03-21T10:19:16+08:00,,,,,,,,,,

--3f3faabb083d4d24bf696e1d859f2c87 Content-Disposition: form-data; name="tags"; filename=""

test --3f3faabb083d4d24bf696e1d859f2c87 Content-Disposition: form-data; name="active"; filename=""

true --3f3faabb083d4d24bf696e1d859f2c87 Content-Disposition: form-data; name="engagement"; filename=""

/api/v1/engagements/200/ --3f3faabb083d4d24bf696e1d859f2c87 Content-Disposition: form-data; name="scan_type"; filename=""

OpenVAS CSV --3f3faabb083d4d24bf696e1d859f2c87--

[devGregA]

@davidabayon , closing. Please feel free to reopen if you continue to have issues after updating.

[davidabayon]

Hi @aaronweaver , thanks. Will be checking this on Monday and update you if we also gets it resolved once we have the latest DefectDojo.

[davidabayon]

Hi @aaronweaver , we are still encountering the same error after pulling the latest version of DefectDojo. We get the same error message.

We both tried this request to be sent using our .jar file and the DefectDojo itself, please see the screenshots below: . Request body for the above picture: { "minimum_severity" : "Info", "scan_date" : "2018-03-26", "verified" : false, "file" : "/home/dabayon/report-bf0610c8-ddd2-4c17-af09-2b53ca5a2e18.csv", "lead" : "/api/v1/users/8/", "tags" : "", "active" : false, "engagement" : "/api/v1/engagements/65/", "resource_uri" : "/api/v1/engagements/65/", "scan_type" : "OpenVAS CSV" } . . . . . Request headers and body for the above picture:

User-Agent: DefectDojo_api/1.1.3 Accept-Encoding: gzip, deflate Accept-Language: en-US,en;q=0.9 Authorization: ApiKey Content-Type: application/json

{ "minimum_severity" : "Info", "scan_date" : "2018-03-26", "verified" : false, "file" : "/home/dabayon/report-bf0610c8-ddd2-4c17-af09-2b53ca5a2e18.csv", "lead" : "/api/v1/users/8/", "tags" : "", "active" : false, "engagement" : "/api/v1/engagements/65/", "resource_uri" : "/api/v1/engagements/65/", "scan_type" : "OpenVAS CSV" }

[aaronweaver]

@davidabayon In order for it to work using the Swagger UI, you'll have to pass in the contents of the file as the form doesn't have access to that file. I'd recommend trying this curl command to verify. (Substitute your api key.)

Is the jar file a custom upload you have written?

curl -i -s -k -X $'POST' -H $'User-Agent: DefectDojo_api/1.1.3' -H $'Authorization: ApiKey admin:19fc9859df610b6a8608ae2df8783ee20f892755' -H $'Content-Type: multipart/form-data; boundary=3f3faabb083d4d24bf696e1d859f2c87' --data-binary $'--3f3faabb083d4d24bf696e1d859f2c87\x0d\x0aContent-Disposition: form-data; name=\"build_id\"; filename=\"\"\x0d\x0a\x0d\x0atest\x0d\x0a--3f3faabb083d4d24bf696e1d859f2c87\x0d\x0aContent-Disposition: form-data; name=\"minimum_severity\"; filename=\"\"\x0d\x0a\x0d\x0aInfo\x0d\x0a--3f3faabb083d4d24bf696e1d859f2c87\x0d\x0aContent-Disposition: form-data; name=\"scan_date\"; filename=\"\"\x0d\x0a\x0d\x0a2018-03-22\x0d\x0a--3f3faabb083d4d24bf696e1d859f2c87\x0d\x0aContent-Disposition: form-data; name=\"file\"; filename=\"nmap_2018-01-31-17-08-15_449fc880-58be-434d-b224-cbe9cf0a0002.csv\"\x0d\x0a\x0d\x0aIP,Hostname,Port,Port Protocol,CVSS,Severity,Solution Type,NVT Name,Summary,Specific Result,NVT OID,CVEs,Task ID,Task Name,Timestamp,Result ID,Impact,Solution,Affected Software/OS,Vulnerability Insight,Vulnerability Detection Method,Product Detection Result,BIDs,CERTs,Other References\x0d\x0a9.9.9.9,www.example.com,80,,,,High,,Test,,,,,PRODUCT_TEST_ONLY,2018-03-21T10:19:16+08:00,,,,,,,,,,\x0a\x0d\x0a--3f3faabb083d4d24bf696e1d859f2c87\x0d\x0aContent-Disposition: form-data; name=\"tags\"; filename=\"\"\x0d\x0a\x0d\x0atest\x0d\x0a--3f3faabb083d4d24bf696e1d859f2c87\x0d\x0aContent-Disposition: form-data; name=\"active\"; filename=\"\"\x0d\x0a\x0d\x0atrue\x0d\x0a--3f3faabb083d4d24bf696e1d859f2c87\x0d\x0aContent-Disposition: form-data; name=\"engagement\"; filename=\"\"\x0d\x0a\x0d\x0a/api/v1/engagements/1/\x0d\x0a--3f3faabb083d4d24bf696e1d859f2c87\x0d\x0aContent-Disposition: form-data; name=\"scan_type\"; filename=\"\"\x0d\x0a\x0d\x0aOpenVAS CSV\x0d\x0a--3f3faabb083d4d24bf696e1d859f2c87--\x0d\x0a' $'http://localhost:8000/api/v1/importscan/'

[davidabayon]

Yes the jar file is a custom program I have written.

So how can we import the OpenVAS CSV file using the importScan API? Do I need to change the format of the request from application/json to this format multipart/form-data?

I'm sorry I'm not getting the whole picture of what you have said

[aaronweaver]

multipart/form-data is what I would recommend. That's how the python api wrapper does all the file uploads. If you need reference to find out take a look at how that does the uploads.

https://github.com/aaronweaver/defectdojo_api

[davidabayon]

Okay, I will be changing the string request into a multipart/form-data format. But why is it not also working with the JSON format when I POST it through the DefectDojo UI when I use the importScan API? Isn't the request should be in JSON format when sent through the UI?

I also would like to point out that this is how the written program works:

1. Use GET to retrieve the list of existing product/s
2. If the product (based on the CSV file) is not yet existing, create one (use POST method of products API, the request is in JSON format)
3. Use GET to retrieve the list of existing engagement/s within each existing product/s (including the newly created product/s)
4. If the engagement (based on the CSV file) is not yet existing, create one (use POST method of engagements API, the request is in JSON format)
5. Use POST method of importScan to import the CSV file based on the engagement and product indicated in the file itself (request is in JSON format).

So what I would do is just change the format of the request being sent to importScan API's POST method because it's where we only get the error from.

[davidabayon]

Now after changing the format into form-data, I am now getting this error after successfully parsing the contents of the file:

[davidabayon]

Hi @aaronweaver , I have managed to change the POST request format for importScan API from application/json to multipart/form-data. Here is the complete request being sent to the API:

POST /api/v1/importscan/ HTTP/1.1 User-Agent: DefectDojo_api/1.1.3 Accept-Encoding: gzip, deflate Accept-Language: en-US,en;q=0.9 Authorization: ApiKey Content-Type: multipart/form-data Content-Length: 2137 Host: localhost:8000 Connection: Keep-Alive

--b8O3BXv6VeFxXtpKdEnuB24pb7hJQU-189U9 Content-Disposition: form-data; name="file"; filename="report-bf0610c8-ddd2-4c17-af09-2b53ca5a2e18.csv" Content-Type: text/plain; charset=ISO-8859-1

IP,Hostname,Port,Port Protocol,CVSS,Severity,Solution Type,NVT Name,Summary,Specific Result,NVT OID,CVEs,Task ID,Task Name,Timestamp,Result ID,Impact,Solution,Affected Software/OS,Vulnerability Insight,Vulnerability Detection Method,Product Detection Result,BIDs,CERTs,Other References 172.19.62.70,,4848,tcp,7.5,High,Workaround,Oracle GlassFish Admin Default Credentials,"The remote Oracle GlassFish is prone to a default account authentication bypass vulnerability.","Vulnerable url: https://localhost:4848/common/index.jsf

It was possible to login using the following credentials:

admin:admin",1.3.6.1.4.1.12345.1.0.111073,NOCVE,2145d93c-b8cf-45f6-bb11-8be5732572b2,SAMPLE_TEST_PRODUCT,2018-03-21T10:19:16+08:00,b01b43a1-17c8-4f2f-a12c-c47dab389e82,"This issue may be exploited by a remote attacker to gain access to sensitive information.",Change the password.,,"It was possible to login with default credentials ""admin:admin"" or ""admin:""","Try to login with default credentials. Details: Oracle GlassFish Admin Default Credentials (OID: 1.3.6.1.4.1.12345.1.0.111073) Version used: $Revision: 6211 $ ",,,,

---b8O3BXv6VeFxXtpKdEnuB24pb7hJQU-189U9 Content-Disposition: form-data; name="minimum_severity"

Info

---b8O3BXv6VeFxXtpKdEnuB24pb7hJQU-189U9 Content-Disposition: form-data; name="scan_date"

2018-03-28

---b8O3BXv6VeFxXtpKdEnuB24pb7hJQU-189U9 Content-Disposition: form-data; name="verified"

false

---b8O3BXv6VeFxXtpKdEnuB24pb7hJQU-189U9 Content-Disposition: form-data; name="lead"

/api/v1/users/1/

---b8O3BXv6VeFxXtpKdEnuB24pb7hJQU-189U9 Content-Disposition: form-data; name="tags"

---b8O3BXv6VeFxXtpKdEnuB24pb7hJQU-189U9 Content-Disposition: form-data; name="active"

false

---b8O3BXv6VeFxXtpKdEnuB24pb7hJQU-189U9 Content-Disposition: form-data; name="engagement"

/api/v1/engagements/3/

---b8O3BXv6VeFxXtpKdEnuB24pb7hJQU-189U9 Content-Disposition: form-data; name="resource_uri"

/api/v1/engagements/3/

---b8O3BXv6VeFxXtpKdEnuB24pb7hJQU-189U9 Content-Disposition: form-data; name="scan_type"

OpenVAS CSV

---b8O3BXv6VeFxXtpKdEnuB24pb7hJQU-189U9--

But after doing so, the same response code was sent back to me HTTP/1.0 500 Internal Server Error with an error message "Invalid boundary in multipart: None". Here is the complete response headers and body:

HTTP/1.0 500 Internal Server Error Date: Wed, 28 Mar 2018 00:31:08 GMT Server: WSGIServer/0.1 Python/2.7.12 Vary: Cookie X-Frame-Options: SAMEORIGIN Content-Type: application/ Content-Length: 1657

{ "error_message": "Invalid boundary in multipart: None", "traceback": "Traceback (most recent call last):\n\n File \"/usr/local/lib/python2.7/dist-packages/tastypie/resources.py\", line 221, in wrapper\n response = callback(request, *args, kwargs)\n\n File \"/usr/local/lib/python2.7/dist-packages/tastypie/resources.py\", line 461, in dispatch_list\n return self.dispatch('list', request, kwargs)\n\n File \"/usr/local/lib/python2.7/dist-packages/tastypie/resources.py\", line 493, in dispatch\n response = method(request, **kwargs)\n\n File \"/usr/local/lib/python2.7/dist-packages/tastypie/resources.py\", line 1398, in post_list\n deserialized = self.deserialize(request, request.body, format=request.META.get('CONTENT_TYPE', 'application/json'))\n\n File \"/home/dabayon/django-DefectDojo-master/dojo/api.py\", line 124, in deserialize\n data = request.POST.copy()\n\n File \"/usr/local/lib/python2.7/dist-packages/django/core/handlers/wsgi.py\", line 126, in _get_post\n self._load_post_and_files()\n\n File \"/usr/local/lib/python2.7/dist-packages/django/http/request.py\", line 299, in _load_post_and_files\n self._post, self._files = self.parse_file_upload(self.META, data)\n\n File \"/usr/local/lib/python2.7/dist-packages/django/http/request.py\", line 257, in parse_file_upload\n parser = MultiPartParser(META, post_data, self.upload_handlers, self.encoding)\n\n File \"/usr/local/lib/python2.7/dist-packages/django/http/multipartparser.py\", line 78, in init\n raise MultiPartParserError('Invalid boundary in multipart: %s' % boundary)\n\nMultiPartParserError: Invalid boundary in multipart: None\n" }

[davidabayon]

Hi @aaronweaver , I have found the cause of the error, it seems that I should not have explicitly added a header for "Content-type" since the request is already providing it and that explicitly adding my own header causes the "Invalid boundary in multipart: None" error

Thanks a lot for all the help!

[aaronweaver]

@davidabayon, glad to hear you have it working!