Test against real world applications

[TACIXAT]

This will fill uncover bugs in Sienna and fill in functionality gaps. Hopefully find some cool bugs in the targeted software too.

[ehennenfent]

We should run SL2 overnight on low-hanging-fruit software projects that will provide new insight into things that can go wrong or enhancements we can make. Additionally, this will give us something to point to in the upcoming quarterly reports (and final report) to indicate that SL2 produces useful results on things other than just Blender. We should start doing this ASAP, but since it's an ongoing process, we'll move it to the Polish milestone after we deliver the beta version.

Some potential targets:

• [ ] https://www.sandboxie.com/DownloadSandboxie (interesting!)
• [ ] https://github.com/pmq20/node-packer
• [x] https://github.com/FreeCAD/FreeCAD
• [ ] https://github.com/x64dbg/x64dbg
• [x] https://www.bandisoft.com/bandizip/
• [x] http://www.peazip.org/
  • [x] 7z.exe from PeaZip
• [ ] http://www.freeoffice.com/en/
• [x] https://www.libreoffice.org/
• [x] https://www.gonitro.com/pdf-reader
• [ ] https://www.openoffice.org/
• [x] https://www.sumatrapdfreader.org/free-pdf-reader.html
• [ ] https://www.wps.com/office-free
• [ ] https://calibre-ebook.com/
• [x] https://www.blender.org/
• [x] https://inkscape.org/en/
• [ ] https://handbrake.fr/
• [x] https://github.com/keepassxreboot/keepassxc
• [ ] https://github.com/lc-soft/LCUI
• [ ] https://github.com/dokan-dev/dokany
• [ ] https://github.com/mpc-hc/mpc-hc
• [ ] https://github.com/mixxxdj/mixxx
• [ ] https://github.com/alexfru/SmallerC
• [x] Internet Explorer
• [x] https://en.bandisoft.com/honeyview/
• [ ] Pretty much every NirSoft tool: https://www.nirsoft.net/
• [x] http://www.dependencywalker.com/
• [ ] http://sqlitebrowser.org/

Taken out of consideration:

• http://luci.criosweb.ro/riot/ (32 bit)
• https://www.kobo.com/us/en/p/desktop (32 bit)
• https://github.com/leozide/leocad (32 bit)
• http://www.angusj.com/resourcehacker/ (32 bit)

[woodruffw]

https://github.com/trailofbits/pe-parse might also be a good candidate.

I'll give LibreOffice a shot (it takes a lot of ancient document formats, so there's lots of interesting space to explore there).

[woodruffw]

Testing LibreOffice helped discover and resolve #171 and #172.

I'm testing SumatraPDF right now (with a DjVu input, since that's probably less tested).

[woodruffw]

Testing KeePassXC revealed #180, which is relatively low priority (since users can re-run the harness as admin to obtain the permissions needed).

Neither SumatraPDF nor KeePassXC yielded any interesting crashes, which makes some sense: SumatraPDF is built on the relatively solid muPDF library, and KeePassXC performs substantial integrity checks.

[ehennenfent]

Issues with NitroPDF have been fixed in https://github.com/trailofbits/sienna-locomotive/commit/78ad3716228bae982411ae70d704f13203ce85fe. However, I'm still not sure why the wrap context pointer is null in some cases.

[haxmeadroom]

The wizard with Inkscape 0.92.3 isn't finding any functions

[woodruffw]

The wizard with Inkscape 0.92.3 isn't finding any functions

Is it possible that Inkscape is 32-bit? I haven't fixed #184 yet, so 32-bit wizard runs currently just fail silently.

[haxmeadroom]

It's 64bit.

[woodruffw]

Hm. Any interesting output from running the wizard directly?

[haxmeadroom]

@woodruffw I put some more info in #188

[haxmeadroom]

peazip spawns 7z.exe, then the PeaZip gui hangs with PeaZip (Not Responding)

[haxmeadroom]

Fuzzing c:/prorgram files/PeaZip/res/7z/7z.exe directly works, but not inside the gui. No crashes on 7z.exe after running for a night on a zip.

[woodruffw]

Honeyview runs like a charm under SL2, except for a strange case where handle_coverage_info will fail after a timeout. I made it slightly more resilient by changing the hard server failure into a warning in https://github.com/trailofbits/sienna-locomotive/commit/edec8af4ad9410fc0dc930be01a82879f1d3484f, but there's probably an underlying issue there that needs to be fixed.

[ehennenfent]

@woodruffw Can you make a separate issue for that? It's high priority, so I don't want to lose track of it.

[woodruffw]

Yep, I'll do so in a moment.

[woodruffw]

Opened as #355.

[woodruffw]

Going to try some NirSoft tools, since it looks like there are some 64-bit builds available and they deal with all sorts of weird Microsoft formats internally.

[woodruffw]

Looks like most of the NirSort tools don't take an input file on the command line, which means that we don't have an easy way to feed them our initial seed input 😞

[ehennenfent]

Hmm, that's not good. I take it they do file selection in the GUI, then? Perhaps if the client asks for it, we'll have to find a quick way to attach an AutoIt script to each fuzzing run. Hopefully it doesn't come to that.

[woodruffw]

Yep, in the GUI. I think NirSoft is an exception to the rule -- most Windows applications expect to be fed their input as the first argument for interoperability with the shell/explorer (since file registrations require that, IIRC). Most of his tools do things behind the scenes and as such don't need file registrations, so it makes some sense.

AutoIt or AutoHK would probably be a good fit for these edge cases, but yeah, it'd be great if it doesn't come to that.

[woodruffw]

Going to try this, since it specifies a CLI: https://www.nirsoft.net/utils/resources_extract.html

[woodruffw]

ResourcesExtract.exe appears to be statically linked to the CRT, so the wizard can't pick up any of its I/O.

[woodruffw]

ResourceHacker.exe would have been a great candidate, but it's 32-bit only.

[woodruffw]

depends.exe caused an interesting condition in the fuzzer:

    // NOTE(ww): The wizard should weed these failures out for us; if it happens
    // here, there's not much we can do.
    if (!GetMappedFileName(GetCurrentProcess(), info->lpBuffer, hash_ctx.fileName, MAX_PATH)) {
        SL2_DR_DEBUG("Fatal: Couldn't get filename for memory map (GLE=%d)! Aborting.\n", GetLastError());
        crashed = false;
        dr_exit_process(1);
    }

[woodruffw]

GLE reports error 1006, or ERROR_FILE_INVALID:

The volume for a file has been externally altered so that the opened file is no longer valid.

CreateFileMapping will return ERROR_FILE_INVALID when told to map an empty file, but this error occurs after GetMappedFileName and notepad.exe is definitely not empty 😕

[woodruffw]

366 will fix the bugs uncovered by depends.exe. So far, I've found one access violation while fuzzing it.

[woodruffw]

"DB Browser for SQLite" doesn't produce any wizard results, much less output: not even module loads get registered. It's definitely 64-bit, though.

[woodruffw]

Fuzzing FreeCAD now, via FreeCADCmd.exe.

[ehennenfent]

Moving to Polish since this will never fully be done.