Generic Parser No Longer Parsing Previously Working Generic CSV File

[desebjohnston]

We have recently migrated to version 2.22.3 from 2.18.0 due to a failing apt dependency issue we encountered on deployment. Looking through the code in the importer and generic parser it appears that a test step prior to get_findings has been added that is causing the failure. When we upload findings through the import-scan API our CSV fails to import with this warning in the dojo log:

django-defectdojo-2223-uwsgi-1 | [24/May/2023 15:20:01] WARNING [dojo.importers.importer.importer:260] Expecting value: line 1 column 1 (char 0) django-defectdojo-2223-uwsgi-1 | [24/May/2023 15:20:01] WARNING [django.request:241] Bad Request: /api/v2/import-scan/

The CSV file we're uploading is just test data from a contrived example using a fuzzer so it's not sensitive and I've uploaded the contents below: generic_HONGGFUZZ.REPORT.csv

We're using the python wrapper version 2 for the defect dojo API import-scan request and can confirm that the upload works with 2.18.0 but not 2.22.3. I didn't see anything in the changelogs that would suggest the CSV format or API request needs to change between these minor version revisions and suspect the changes to the generic parser and importer in 2.19.0 may have triggered this without test coverage picking it up.

Thank you for your time and consideration!

[kiblik]

I tested your CSV on 2.22.3. I uploaded CSV over API to empty DD instance (I pre-created only product and engagement) and everything works well on my side. Can you check your logs - looks you sent only part of it - maybe some important part is missing. Can you double-check that you used Generic Findings Import?

[desebjohnston]

I can confirm Generic Findings Import. I tested with curl instead of using the deprecated python wrapper for the API (https://github.com/DefectDojo/defectdojo_api/blob/master/defectdojo_api/defectdojo_apiv2.py - still worked post-deprecation using version 2 on 2.18.0). It looked like the verified param isn't set in the wrapper, but I dropped that from the curl post and it still worked. I don't see any required params that are missing otherwise. I tried figuring out what was different in the request without much luck. My suspicion is there is something about the way the filedata is read in and composed in the wrapper and sent through the python request that the API interface doesn't like anymore.

For reference here is the curl post I used that works:

curl -X POST "http://" -H "accept: application/json" -H "Content-Type: multipart/form-data" -H "User-Agent: DefectDojo_api/1.2.0." -H "Authorization: Token " -F "minimum_severity=Info" -F "active=true" -F "scan_type=Generic Findings Import" -F "push_to_jira=false" -F "file=@generic_HONGGFUZZ.REPORT.csv" -F "product_name=Test" -F "scan_date=2023-05-24" -F "engagement=7"

I probably need to bump the log level in the docker container to get more information, here is a little more, but may not be useful:

ango-defectdojo-2223-uwsgi-1 | [pid: 37|app: -|req: -/-] 172.18.0.1 (-) {30 vars in 366 bytes} [Wed May 24 18:51:54 2023] GET / => generated 0 bytes in 28 msecs (HTTP/1.0 500) 0 headers in 0 bytes (0 switches on core 0) django-defectdojo-2223-uwsgi-1 | [pid: 37|app: -|req: -/-] 172.18.0.1 (-) {40 vars in 657 bytes} [Wed May 24 18:53:16 2023] GET //api/v2/users/?limit=20&username=dojoadmin => generated 254 bytes in 37 msecs (HTTP/1.1 200) 8 headers in 246 bytes (1 switches on core 1) django-defectdojo-2223-uwsgi-1 | [pid: 37|app: -|req: -/-] 172.18.0.1 (-) {42 vars in 644 bytes} [Wed May 24 18:53:17 2023] POST //api/v2/engagements/ => generated 809 bytes in 131 msecs (HTTP/1.1 201) 8 headers in 251 bytes (1 switches on core 0) django-defectdojo-2223-uwsgi-1 | [24/May/2023 18:53:17] WARNING [dojo.importers.importer.importer:260] Expecting value: line 1 column 1 (char 0) django-defectdojo-2223-uwsgi-1 | [24/May/2023 18:53:17] WARNING [django.request:241] Bad Request: /api/v2/import-scan/

[kiblik]

Looks like issue in defectdojo_api. So I suppose, it should be reported in https://github.com/DefectDojo/defectdojo_api/issues

[desebjohnston]

Okay I reported it there. Not sure if it'll get much traction since it's deprecated. Are there any future plans to support a new API wrapper for any language?

[kiblik]

API is the set of HTTP endpoints. It is "language-independent". defectdojo_api is one of many implementation of libraries or wrappers which somebody implemented or can implement in future - doesn't matter in which language.

[desebjohnston]

Gotcha. Yeah, it would just be a nice-to-have if there was a language-specific wrapper that was officially maintained by the community. We would certainly adopt our use-case to whatever language that happened to be.

I do have one more datum point that adds to this mystery. This issue appears to be specific to generic parsing. I can import sonarqube and bandit reports all day through the python wrapper on 2.22.3. This was initially why my rationale was something has changed with Dojo handling of generic reports on the endpoint in conjunction with the logs that point to the newly added code.

[manuel-sommer]

I believe this can be closed @mtesauro as the discussion is about defectdojo_api python wrapper and the Generic parser works according to comments.