Provide optional support for emulation without `apisetschema.dll`

[mewmew]

Hi @kgwinnup and @jholowczak!

I stumbled upon Binee today, and what a pleasure it has been to start diving into it. You've essentially managed to capture an idea I've been playing around with myself for quite some time, and made it into a beautiful working system. Thanks for sharing Binee with the open source community!

As I wanted to take binee out for a spin, I started with a simple "hello world" sample (see foo.exe and foo.go below).

The first issue I ran into was file 'apisetschema.dll' not found, which is expected, as I have not (yet) downloaded the docker image. I'm currently travelling so downloading 10 GB would be limiting.

$ binee foo.exe
2019/11/21 21:01:44 file 'apisetschema.dll' not found

I know the rationale for implementing support for apisetschema.dll as there may exist several versions of a given DLL. However, as there are quite a few PE binaries that are capable of running without apisetschema.dll present, it would seem preferable to also add optional support using binee without requiring apisetschema.dll to be present.

On the system I'm currently running, I have access to all DLLs used by the sample set of binaries I'd like to analyze, but I do not have access to apisetschema.dll.

How much effort would be require to allow binee to analyze PE executables without requiring apisetschema.dll to be present?

I may peek around a bit in the code and see if I can make this optional, or if it would require a redesign of the DLL loader.

Wish you all the best and happy coding!

Cheers, Robin

---

Contents of foo.go:

package main

func main() {
    println("foo")
}

Command used to compile foo.go:

GOARCH=386 GOOS=windows go build -o foo.exe foo.go

[kgwinnup]

Agree with you 100% on removing the apisetschema.dll requirement. Just a quick glance at the code and I don't believe it is a trivial change, but it is certainly possible. I think the challenge will just be going through the few places in the loader and a couple hooks and figure out a clean way of checking if the apiset exists or not. I believe there are a couple places where we (wrongly) assume values to be in the apiset object.

[mewmew]

@kgwinnup, wow. Quick response time :)

I started looking at it, and it seems that making it optional was quite easy. Just needed to change 5 or so lines of code. Now, I'm going about fixing a few nil-deref panics.

edit: not at all a finished patch, but this works and does make loading of apisetschema.dll optional. I also added a few log statements to handle errors where otherwise I would run into nil-deref panics.

diff --git a/windows/loader.go b/windows/loader.go
index d34b487..dd1bea0 100644
--- a/windows/loader.go
+++ b/windows/loader.go
@@ -1,6 +1,7 @@
 package windows

 import "fmt"
+import "log"
 import "os"
 import "bytes"
 import "strings"
@@ -377,6 +378,10 @@ func retrieveDllFromDisk(cur map[string]*pefile.PeFile, apiset *pefile.PeFile, s
    // get realDll name on disk
    // for apiset recurse through each real dll in the apisets list
    if strings.Compare(name[:4], "api-") == 0 {
+       if apiset == nil {
+           fmt.Fprintf(os.Stderr, "error loading dll %s; unable to locate \"apisetschema.dll\"\n", name)
+           return
+       }
        apiset_len := len(apiset.Apisets[name[0:len(name)-6]]) - 1
        if apiset_len >= 0 {
            realDll = apiset.Apisets[name[0:len(name)-6]][apiset_len]
@@ -757,11 +762,11 @@ func (emu *WinEmulator) initPe(pe *pefile.PeFile, path string, arch, mode int, a
    }

    // load Apisetschema dll for mapping to real dlls
-   apisetPath, err := util.SearchFile(emu.SearchPath, "apisetschema.dll")
-   if err != nil {
-       return err
+   var apiset *pefile.PeFile
+   if apisetPath, err := util.SearchFile(emu.SearchPath, "apisetschema.dll"); err == nil {
+       // only load apisetschema.dll if present.
+       apiset, _ = pefile.LoadPeFile(apisetPath)
    }
-   apiset, _ := pefile.LoadPeFile(apisetPath)

    // create the main map to hold all name/realdll mappings to actual PeFile object
    peMap := make(map[string]*pefile.PeFile)
@@ -813,9 +818,11 @@ func (emu *WinEmulator) initPe(pe *pefile.PeFile, path string, arch, mode int, a
    ldrEntry := emu.createLdrEntry(pe, 0)
    emu.writeLdrEntry(ldrEntry, "Memory")
    emu.writeLdrEntry(ldrEntry, "Initialization")
-   var lpe *pefile.PeFile
    for i, key := range ldrList {
-       lpe = peMap[key]
+       lpe, ok := peMap[key]
+       if !ok {
+           log.Printf("unable to locate DLL %q", key)
+       }
        ldrEntry = emu.createLdrEntry(lpe, uint64(i+1))
        emu.writeLdrEntry(ldrEntry, "Load")
        emu.writeLdrEntry(ldrEntry, "Memory")
diff --git a/windows/winemulator.go b/windows/winemulator.go
index 8eeb753..5194079 100644
--- a/windows/winemulator.go
+++ b/windows/winemulator.go
@@ -1,5 +1,6 @@
 package windows

+import "log"
 import "gopkg.in/yaml.v2"
 import "os"
 import "io/ioutil"
@@ -280,9 +281,14 @@ func New(path string, arch, mode int, args []string, verbose int, config string,
    }

    //load the PE
-   pe, _ := pefile.LoadPeFile(emu.Binary)
+   pe, err := pefile.LoadPeFile(emu.Binary)
+   if err != nil {
+       log.Println("unable to load PE file:", err)
+   }
    err = emu.initPe(pe, path, arch, mode, args, calldllmain)
-
+   if err != nil {
+       log.Println("unable to load PE file:", err)
+   }
    emu.Cpu = core.NewCpuManager(emu.Uc, emu.UcMode, emu.MemRegions.StackAddress, emu.MemRegions.StackSize, emu.MemRegions.HeapAddress, emu.MemRegions.HeapSize)
    emu.Scheduler = NewScheduleManager(&emu)

[mewmew]

While I have you on the phone, is emu.Opts.Root configurable? It seems like using temp works in most places for handling search paths for DLLs, but there are a few that are hard coded for emu.Opts.Root, e.g. from windows/ntdll.go:

data, err := ioutil.ReadFile(emu.Opts.Root + fmt.Sprintf("windows/system32/c_%d.nls", in.Args[1]))

The default value seem to be emu.Opts.Root = "os/win10_32/", so if not, I can just create a dummy os/win10_32 directory for now and symlink towards my real root.

[kgwinnup]

Yes, that is one of the values configurable with a yaml file, which can be provided via command line flags.

Other configurable options exist as well, the main struct definition is https://github.com/carbonblack/binee/blob/master/windows/winemulator.go#L24

[mewmew]

I tried the config file, but for some reason it doesn't seem to work. Using the dummy os/win10_32 symlink workaround worked however.

Contents of win.yaml:

$ cat win.yaml 
root: "/home/u/.wine/drive_c/"

Using win.yaml config:

$ binee -c win.yaml foo_32.exe
error finding file ntdll.dll
error finding file kernel32.dll
error finding file kernel32.dll
2019/11/21 21:56:35 unable to locate DLL "ntdll.dll"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x55b090]

goroutine 1 [running]:
github.com/carbonblack/binee/windows.(*WinEmulator).createLdrEntry(0xc000148900, 0x0, 0x1, 0x1)
    /home/u/Desktop/binee/windows/loader.go:291 +0x70
...

Using os/win10_32 symlink workaround:

$ mkdir os
$ ln -s /home/u/.wine/drive_c/ os/win10_32
$ binee foo_32.exe
panic: interface conversion: interface {} is *pefile.OptionalHeader32P, not *pefile.OptionalHeader32

goroutine 1 [running]:
github.com/carbonblack/binee/windows.(*WinEmulator).createLdrEntry(0xc000114900, 0xc000138100, 0x1, 0x9)
    /home/u/Desktop/binee/windows/loader.go:303 +0x3ec
...

Notice, the workaround gets further (crashes in a different place).

[kgwinnup]

ah, nice find. I think I found the bug, the search path never gets updated with the new Root if there is a config provided. https://github.com/carbonblack/binee/blob/master/windows/winemulator.go#L267

Does that sound like what your seeing? Just below that line, the config file is loaded and all the default values are overwritten with the configuration. However, the search path is never updated beyond line 267 it looks like.

[mewmew]

Hmm, from a quick glance at main.go, it seems like the config path of -c is never used (unless you also specify -a or -A).

    if options.ApisetDump {
        rootFolder := "os/win10_32/"
        if options.Config != "" {
            conf, err := util.ReadGenericConfig(options.Config)

...

    if options.ApisetLookup != "" {
        rootFolder := "os/win10_32/"
        if options.Config != "" {
            conf, err := util.ReadGenericConfig(options.Config)

edit: also, the way of handling command line arguments in binee looks very C-like. To clean up handling of command line arguments a bit, I'd suggest looking into using the flag package of the Go standard library.

[kgwinnup]

The config file is still used by Winemulator, however, with the call to New https://github.com/carbonblack/binee/blob/master/main.go#L192

At least in the early stage, this was originally poc'd in C. I'll look into flag though, thanks

[mewmew]

Ah, you are right.

Yes, so the issue you pointed out is indeed the cause. The search path should be set after reading the yml config.

-   emu.SearchPath = []string{"temp/", emu.Opts.Root + "windows/system32/", "c:\\Windows\\System32"}

    var buf []byte
    if buf, err = ioutil.ReadFile(config); err == nil {
        _ = yaml.Unmarshal(buf, &emu.Opts)
    }

+   emu.SearchPath = []string{"temp/", emu.Opts.Root + "windows/system32/", "c:\\Windows\\System32"}

[kgwinnup]

just pushed that "hotfix"

[mewmew]

I'll look into flag, this was my first project in Golang, and it was (at least a very early stage of it) originally done in C.

No worries, I'm glad you decided to write it in Go, even if it started out as a learning experiment :) And things like these are easy to fix. As you go further, there will be more things you'll bump into and learn that would help make the code more idiomatic Go.

One such thing would be to group imports. An automatic way to do so is to run goimports.

-import "log"
-import "gopkg.in/yaml.v2"
-import "os"
-import "io/ioutil"
-import "time"
-import "github.com/carbonblack/binee/pefile"
-import "encoding/binary"
-
-//import "regexp"
-import cs "github.com/kgwinnup/gapstone"
-import uc "github.com/unicorn-engine/unicorn/bindings/go/unicorn"
-import "sort"
-import core "github.com/carbonblack/binee/core"

+import (
+   "encoding/binary"
+   "io/ioutil"
+   "log"
+   "os"
+   "sort"
+   "time"
+
+   "github.com/carbonblack/binee/core"
+   "github.com/carbonblack/binee/pefile"
+   cs "github.com/kgwinnup/gapstone"
+   uc "github.com/unicorn-engine/unicorn/bindings/go/unicorn"
+   "gopkg.in/yaml.v2"
+)

Note, in the above, I also removed the local package name core as binee/core was already called core, so no need for a rename.

To use goimports, do as follows.

go get golang.org/x/tools/cmd/goimports
goimports -w foo.go

I'll prepare a PR for you :)

[kgwinnup]

thank you, saw all the PR's. This is great!