Burpsuite normalizer cannot parse xml file

[stephanie0x00]

Describe the bug A community member ran into the issue that Burpsuite XML files are not parsed properly by the burpsuite-normalizer.

Uploading a raw file with the mime-type xml/burp-export works, however when the raw file is being parsed by the scheduler an error is triggered.

Reproducing this issue also showed that it's currently unclear how a user can create the XML file (standard export format is a binary .burp file). Thus an update on the documentation would also be required to make this more clear.

To Reproduce Steps to reproduce the behavior:

1. In BurpSuitePro create an XML file by right-click on one or more hosts (e.g. Targets page) and then click 'Issues > Report issues for this host'. Choose the XML export and continue with the wizard.
2. In Openkat click on Objects.
3. Select Upload raw file and use the mime-type 'xml/burp-export'.
4. Observe the error in the logs from the normalizer.

Expected behavior Under Object you upload a raw file with the xml/burp-export mime-type. The raw file is uploaded and should automatically be parsed by the scheduler.

Screenshots The error message:

OpenKAT version commit 8b29e48f229d872174dd56b7b9e4e007f490e393 (HEAD -> main, origin/main, origin/HEAD)

[underdarknl]

Couple of options as discussed:

• Assume all uploads are declarations, this would mean they are no longer cleaned up by the garbage collector, and there is no clear path towards 'refreshing' the data if a new raw file is uploaded, Do we clear all objects from the previous manual run for this normalizer, or do we somehow decide on a number which designates two runs are 'the same'?
• Ask the user for an input-ooi, and use that. This assumes every Uploaded raw file in associable with an input_ooi. Some tools might however use hundreds of input_ooi's.
• Don't raise and issue when no input-ooi is present for manually triggered normalizers. This would mean that objects that do need to reference an object also need to be yielded in conjunction with that referenced (input) OOI

[dekkers]

As discussed, the most suitable option seems to be the following:

We create a new Scan object type that can be used by the user to relate an uploaded raw file to a certain scan. For example there can be a Scan object "network 1" and "network 2". If the users uploads a new raw file with the contents of the scan of network 1, the user can select the "network 1" Scan object and that object will be used as input OOI. If the previous raw file that was linked to this Scan object had an object that the new raw file doesn't have, the garbage collector can then detect this and delete the object .

In the upload interface the user should have the option to either select existing Scans or create a new Scan object.

[noamblitz]

So these objects are mostly placeholder OOIs so the garbage collector can remove old results? I assume these Scan OOIs only have a name as attribute? @dekkers

[noamblitz]

I started the implementation of this issue in this PR: https://github.com/minvws/nl-kat-coordination/pull/3169. For testing, I found it easy to relate uploaded raw files to existing OOIs. Is this something we want to keep? Then there are three choices:

1. relate raw to existing ooi
2. relate raw to existing scan
3. relate raw to a new scan

[underdarknl]

The Scan objects mentioned by @dekkers are 'special' OOI's which designate just that a scan. The scan would get a unique (user given) name, and the user can create these Scan OOI's when doing their upload, or select a previous one. If a previous one is selected that Scan-OOI + the upload 'boefje' and normalizers would form the combination for the garbage collector to invalidate any OOI's resulting from the previous run of those combinations.

Selecting an existing OOI could work too, but then we would need to make sure that we can differentiate between the Tool used on that OOI. For example:, running Burpsuite on ipX, and running Nmap locally on ipX would point at the same input-ooi, might both create an XML raw file, and thus be parsed by the same normalizer. Resulting in Nmap overwriting any info from Burpsuite, and Burpsuite's exta info being deleted by garbage collection.

[noamblitz]

Great, I discussed with @Donnype that the worry about differentiating between boefje+normalizer and manual+normalizer will be solved. For now, I will finalize the backend (, merge?), and then talk to the designers about communicating to users. In the current POC garbage collection already works on scan OOIs and other OOIs.

I think it is a great step forward that we can now share raw files of all boefjes and upload them to our own KAT install (and not only raw files for normalizers that were specifically designed for that).