Build system improvements

hillu commented 3 years ago

So, here's the follow-up to my promise from #3.

I have make some changes to the Makefile that make the build a bit more robust, add a 64bit Windows target, and add cross-building YARA for Windows. This is still a lot simpler than the 3rdparty.mk from Spyre but I have changed quite a lot, so I feel that a high-level explanation might be useful:

All the main targets are now based on files generated during the build stage and they depend on the source files. This utilizes Make's main features -- inferring what needs to be (re)built from source and binary file timestamps.
the original targets are kept around as "phony" targets that are not tied to files. For example, windows-386 depends on kraken.exe and kraken-launcher.exe in the build/windows-386 subdirectory. Once these two files exist and there haven't been edits to the source files since, the windows-386 target is considered as built and re-running make windows-386 causes a message "Nothing to be done for 'windows-386'".

It would probably make sense to also do private builds of the static YARA library for non-Windows architectures.

hillu commented 3 years ago

If anything is unclear, feel free to ask. :-)

botherder commented 3 years ago

My apologies for ghosting this. With holidays and all, I haven't yet had the time to look at this. I promise I will do so soon. Thank you already for the contribution.

botherder commented 3 years ago

I might be missing something, but are check-env and rules-compiler orphans now?

hillu commented 3 years ago

I might be missing something, but are check-env and rules-compiler orphans now?

No, you can still run make check-env and make rules-compiler and get the same results as before.

It would make sense to split the compiler target into two steps, though:

build the compiler itself (which has no external rule dependencies),
compile + convert the ruleset

I'm going to do another PR.

botherder commented 3 years ago

Thanks!

I guess the main thing to account for is to allow to re-compile if the source code hasn't changed but the Yara rules we want to compile with did. The rules are then build in a go-bindata "rules" file which scanner.go tries to load with Asset (.. I should find a way to make optional, in case one wants to a build without any embedded rules). Another situation is if we want a build with a different BACKEND env var.

Other than that, I'm currently having some issues with building the various targets, I suppose because of statically linking. For example, for darwin:

# os/user
/usr/lib/go-1.15/src/os/user/getgrouplist_darwin.go: In function ‘mygetgrouplist’:
/usr/lib/go-1.15/src/os/user/getgrouplist_darwin.go:16:11: warning: implicit declaration of function ‘getgrouplist’; did you mean ‘mygetgrouplist’? [-Wimplicit-function-declaration]
   16 |  int rv = getgrouplist(user, (int) group, buf, ngroups);
      |           ^~~~~~~~~~~~
      |           mygetgrouplist
# github.com/shirou/gopsutil/mem
../../go/pkg/mod/github.com/shirou/gopsutil@v2.17.12+incompatible/mem/mem_darwin_cgo.go:7:10: fatal error: mach/mach_host.h: No such file or directory
    7 | #include <mach/mach_host.h>
      |          ^~~~~~~~~~~~~~~~~~
compilation terminated.
# github.com/shirou/gopsutil/cpu
../../go/pkg/mod/github.com/shirou/gopsutil@v2.17.12+incompatible/cpu/cpu_darwin_cgo.go:10:10: fatal error: mach/mach_init.h: No such file or directory
   10 | #include <mach/mach_init.h>
      |          ^~~~~~~~~~~~~~~~~~
compilation terminated.
# net
/usr/lib/go-1.15/src/net/cgo_bsd.go:15:72: could not determine kind of name for C.AI_MASK
# github.com/mattn/go-sqlite3
/usr/bin/ld: $WORK/b073/_x011.o: in function `unixDlError':
../../go/pkg/mod/github.com/mattn/go-sqlite3@v1.9.0/sqlite3-binding.c:37959: undefined reference to `dlerror'
/usr/bin/ld: $WORK/b073/_x011.o: in function `unixDlClose':
../../go/pkg/mod/github.com/mattn/go-sqlite3@v1.9.0/sqlite3-binding.c:37990: undefined reference to `dlclose'
/usr/bin/ld: $WORK/b073/_x011.o: in function `unixDlSym':
../../go/pkg/mod/github.com/mattn/go-sqlite3@v1.9.0/sqlite3-binding.c:37986: undefined reference to `dlsym'
/usr/bin/ld: $WORK/b073/_x011.o: in function `unixDlOpen':
../../go/pkg/mod/github.com/mattn/go-sqlite3@v1.9.0/sqlite3-binding.c:37945: undefined reference to `dlopen'
collect2: error: ld returned 1 exit status
make: *** [Makefile:120: /home/user/kraken/build/darwin/kraken] Error 2

For windows-386, for example, I get:

/usr/bin/i686-w64-mingw32-ld: libyara/.libs/libyara.a(pe.o):pe.c:(.bss+0x0): multiple definition of `yr_cryptprov'; libyara/.libs/libyara.a(libyara.o):libyara.c:(.bss+0x208): first defined here
/usr/bin/i686-w64-mingw32-ld: libyara/.libs/libyara.a(hash.o):hash.c:(.bss+0x0): multiple definition of `yr_cryptprov'; libyara/.libs/libyara.a(libyara.o):libyara.c:(.bss+0x208): first defined here

I should probably try to build on a clean system.

hillu commented 3 years ago

I ran into the multiple definition ofyr_cryptprov'` issue with GCC 10+ for Windows targets before and got it fixed in YARA. See #14.

I can't say anything useful about the MacOSX problem since I don't have a MacOSX system myself.

hillu commented 3 years ago

The go-sqlite-related errors can likely be avoided by passing the sqlite_omit_load_extension tag to the compiler. This causes SQLITE_OMIT_LOAD_EXTENSION to be defined in C code so the code referencing the external symbols is not compiled. (You aren't going to need SQLite plugin support anyway.)

botherder / kraken

Build system improvements #12