search

Yamato-Security / takajo

Takajō (鷹匠) is a Hayabusa results analyzer.
https://yamato-security.github.io/takajo/
GNU Affero General Public License v3.0
89 stars 7 forks source link

Replace Puppy with standard HTTP client? #160

Open YamatoSecurity opened 5 months ago

YamatoSecurity commented 5 months ago

@fukusuket What about replacing Puppy with https://nim-lang.org/docs/httpclient.html in order to be able to compile static linux binaries?

fukusuket commented 5 months ago

@YamatoSecurity Yes, I think it's good! My only concern is that a dll is required on Windows, but is that OK?

YamatoSecurity commented 5 months ago

@fukusuket Since we already require separate dlls to be bundled I don't think there is any problem to add one more. As long as it can compile statically on linux and macos.

fukusuket commented 5 months ago

@YamatoSecurity I see, that's fine then, I agree with replacing it with HTTP client!

fukusuket commented 5 months ago

I'll try implementing it!

fukusuket commented 5 months ago

@YamatoSecurity I tried the httpclient.html, but unforunatelly it seems that a similar issue also occurs :( It seems difficult to resolve, I'll put it hold on now.

YamatoSecurity commented 5 months ago

@fukusuket Thanks so much for implementing it. It seems the problem is that libpcre is still being compiled dynamically. This person figured out how to compile it statically: https://forum.nim-lang.org/t/5160 Also, nimpcre might work: https://github.com/genotrance/nimpcre

fukusuket commented 5 months ago

@YamatoSecurity Thank you for information! I tried, but got a compile error :( It was last committed 4 years ago, so it may not be compatible with recent versions of Nim...?😭

fukusuke@ub:~/takajo$ nimble build -d:release --threads:on --gcc.exe:"musl-gcc" --gcc.linkerexe:"musl-gcc" --passL="-static" -d:nimDebugDlOpen
  Verifying dependencies for takajo@2.5.0
     Info:  Dependency on cligen@>= 1.5 already satisfied
  Verifying dependencies for cligen@1.7.0
     Info:  Dependency on suru@#f6f1e607c585b2bc2f71309996643f0555ff6349 already satisfied
  Verifying dependencies for suru@0.3.1
     Info:  Dependency on termstyle@any version already satisfied
  Verifying dependencies for termstyle@0.1.0
     Info:  Dependency on nancy@any version already satisfied
  Verifying dependencies for nancy@0.1.1
     Info:  Dependency on ansiparse@>= 0.2.0 already satisfied
  Verifying dependencies for ansiparse@0.4.0
     Info:  Dependency on jsony@>= 1.1.5 already satisfied
  Verifying dependencies for jsony@1.1.5
     Info:  Dependency on bearssl@any version already satisfied
  Verifying dependencies for bearssl@0.2.2
     Info:  Dependency on unittest2@any version already satisfied
  Verifying dependencies for unittest2@0.2.2
     Info:  Dependency on nimpcre@any version already satisfied
  Verifying dependencies for nimpcre@0.2.1
     Info:  Dependency on nimterop@>= 0.4.4 already satisfied
  Verifying dependencies for nimterop@0.6.13
     Info:  Dependency on regex@>= 0.15.0 already satisfied
  Verifying dependencies for regex@0.25.0
     Info:  Dependency on unicodedb@>= 0.7.2 already satisfied
  Verifying dependencies for unicodedb@0.12.0
     Info:  Dependency on cligen@>= 1.0.0 already satisfied
  Verifying dependencies for cligen@1.7.0
   Building takajo/takajo using c backend
stack trace: (most recent call last)
nimpcre.nim(7, 11)       nimpcre
assertions.nim(41, 14)   failedAssertImpl
assertions.nim(36, 13)   raiseAssert
fatal.nim(53, 5)         sysFatal
/home/fukusuke/.choosenim/toolchains/nim-2.0.2/lib/system/fatal.nim(53, 5) Error: unhandled exception: nimpcre.nim(7, 5) `false` Error: -d:usePcreHeader is required [AssertionDefect]
       Tip: 36 messages have been suppressed, use --verbose to show them.
nimble.nim(229)          buildFromDir

    Error:  Build failed for the package: takajo
fukusuke@ub:~/takajo$ nimble build -d:release --threads:on --gcc.exe:"musl-gcc" --gcc.linkerexe:"musl-gcc" --passL="-static" -d:nimDebugDlOpen -d:usePcreHeader
  Verifying dependencies for takajo@2.5.0
     Info:  Dependency on cligen@>= 1.5 already satisfied
  Verifying dependencies for cligen@1.7.0
     Info:  Dependency on suru@#f6f1e607c585b2bc2f71309996643f0555ff6349 already satisfied
  Verifying dependencies for suru@0.3.1
     Info:  Dependency on termstyle@any version already satisfied
  Verifying dependencies for termstyle@0.1.0
     Info:  Dependency on nancy@any version already satisfied
  Verifying dependencies for nancy@0.1.1
     Info:  Dependency on ansiparse@>= 0.2.0 already satisfied
  Verifying dependencies for ansiparse@0.4.0
     Info:  Dependency on jsony@>= 1.1.5 already satisfied
  Verifying dependencies for jsony@1.1.5
     Info:  Dependency on bearssl@any version already satisfied
  Verifying dependencies for bearssl@0.2.2
     Info:  Dependency on unittest2@any version already satisfied
  Verifying dependencies for unittest2@0.2.2
     Info:  Dependency on nimpcre@any version already satisfied
  Verifying dependencies for nimpcre@0.2.1
     Info:  Dependency on nimterop@>= 0.4.4 already satisfied
  Verifying dependencies for nimterop@0.6.13
     Info:  Dependency on regex@>= 0.15.0 already satisfied
  Verifying dependencies for regex@0.25.0
     Info:  Dependency on unicodedb@>= 0.7.2 already satisfied
  Verifying dependencies for unicodedb@0.12.0
     Info:  Dependency on cligen@>= 1.0.0 already satisfied
  Verifying dependencies for cligen@1.7.0
   Building takajo/takajo using c backend
/home/fukusuke/.nimble/pkgs2/nimpcre-0.2.1-3377b148d8e5e1c87e45259d2d4ebb1495ba354c/nimpcre.nim(24, 10) template/generic instantiation of `getHeader` from here
/home/fukusuke/.nimble/pkgs2/nimterop-0.6.13-a93246b2ad5531db11e51de7b2d188c42d95576a/nimterop/build/getheader.nim(421, 65) Error: type mismatch
Expression: true == 1
  [1] true: bool
  [2] 1: int literal(1)

Expected one of (first mismatch at [position]):
[1] proc `==`(a, b: NimIdent): bool
[1] proc `==`(a, b: NimNode): bool
[1] proc `==`(a, b: NimSym): bool
[1] proc `==`(err1, err2: OSErrorCode): bool
[1] proc `==`(pkg1, pkg2: ConanPackage): bool
[1] proc `==`(pkg1, pkg2: JBBPackage): bool
[1] proc `==`(x, y: char): bool
[1] proc `==`(x, y: cstring): bool
[1] proc `==`(x, y: float): bool
[1] proc `==`(x, y: float32): bool
[1] proc `==`(x, y: int): bool
[1] proc `==`(x, y: int16): bool
[1] proc `==`(x, y: int32): bool
[1] proc `==`(x, y: int64): bool
[1] proc `==`(x, y: int8): bool
[1] proc `==`(x, y: pointer): bool
[1] proc `==`(x, y: string): bool
[1] proc `==`(x, y: uint): bool
[1] proc `==`(x, y: uint16): bool
[1] proc `==`(x, y: uint32): bool
[1] proc `==`(x, y: uint64): bool
[1] proc `==`(x, y: uint8): bool
[1] proc `==`[A, B](s, t: OrderedTableRef[A, B]): bool
[1] proc `==`[A, B](s, t: OrderedTable[A, B]): bool
[1] proc `==`[A, B](s, t: TableRef[A, B]): bool
[1] proc `==`[A, B](s, t: Table[A, B]): bool
[1] proc `==`[A](s, t: CountTableRef[A]): bool
[1] proc `==`[A](s, t: CountTable[A]): bool
[1] proc `==`[Enum: enum](x, y: Enum): bool
[1] proc `==`[I, T](x, y: array[I, T]): bool
[1] proc `==`[T: proc | iterator](x, y: T): bool
[1] proc `==`[T: tuple | object](x, y: T): bool
[1] proc `==`[T](x, y: openArray[T]): bool
[1] proc `==`[T](x, y: ptr T): bool
[1] proc `==`[T](x, y: ref T): bool
[1] proc `==`[T](x, y: seq[T]): bool
[1] proc `==`[T](x, y: set[T]): bool
[2] proc `==`(x, y: bool): bool

       Tip: 28 messages have been suppressed, use --verbose to show them.
nimble.nim(229)          buildFromDir

    Error:  Build failed for the package: takajo
fukusuket commented 1 month ago

I was able to solve the pcre compile error, but now I have a sqlite compile error ... :(

git clone https://github.com/kaushalmodi/hello_musl.git
cd hello_musl/
nim musl -d:pcre src/hello_musl_pcre.nim
cd ../
git clone https://github.com/Yamato-Security/takajo.git
cd takajo
git checkout 160-replace-puppy-httpclient
cp ../hello_musl/pcre/8.42/include/pcre.h /usr/include/x86_64-linux-musl/
nimble build -d:release --threads:on -d:nimDebugDlOpen --gcc.exe:musl-gcc --gcc.linkerexe:musl-gcc --passL=-static --passL:"../hello_musl/pcre-8.42/.libs/libpcre.a" -d:usePcreHeader
./takajo
Dynamic loading not supported
Dynamic loading not supported
could not load: libsqlite3.so(|.0)

memo:

sudo apt install libsqlite3-dev
fukusuket commented 1 month ago

sqlite static compile error memo

wget https://www.sqlite.org/2024/sqlite-autoconf-3460100.tar.gz
tar zxvf sqlite-autoconf-3460100.tar.gz
cd sqlite-autoconf-3460100
CFLAGS="-O2" LDFLAGS="-static" ./configure
make
ls .libs/libsqlite3.a

musl-gcc  -O2           \
  shell.c sqlite3.c     \
    -static -lm         \
    -o sqlite3

nimble build -d:release --threads:on -d:nimDebugDlOpen --gcc.exe:musl-gcc --gcc.linkerexe:musl-gcc --passL=-static --passL:"../hello_musl/pcre-8.42/.libs/libpcre.a" -d:usePcreHeader --dynlibOverride:sqlite3 --passL:"../sqlite-autoconf-3460100/.libs/libsqlite3.a"
$ nimble build -d:release --threads:on -d:nimDebugDlOpen --gcc.exe:musl-gcc --gcc.linkerexe:musl-gcc --passL=-static --passL:"../hello_musl/pcre-8.42/.libs/libpcre.a" -d:usePcreHeader --dynlibOverride:sqlite3 --passL:"../sqlite-autoconf-3460100/.libs/libsqlite3.a" --mm:refc -d:nimDebugDlOpen --verbose
     Info:  Nimble data file "/home/fukusuke/.nimble/nimbledata2.json" has been loaded.
  Verifying dependencies for takajo@2.6.0
    Reading official package list
   Checking for cligen@>= 1.5
     Info:  Dependency on cligen@>= 1.5 already satisfied
  Verifying dependencies for cligen@1.7.0
    Reading official package list
   Checking for suru@#f6f1e607c585b2bc2f71309996643f0555ff6349
     Info:  Dependency on suru@#f6f1e607c585b2bc2f71309996643f0555ff6349 already satisfied
  Verifying dependencies for suru@0.3.1
    Reading official package list
   Checking for termstyle@any version
     Info:  Dependency on termstyle@any version already satisfied
  Verifying dependencies for termstyle@0.1.0
    Reading official package list
   Checking for nancy@any version
     Info:  Dependency on nancy@any version already satisfied
  Verifying dependencies for nancy@0.1.1
    Reading official package list
   Checking for ansiparse@>= 0.2.0
     Info:  Dependency on ansiparse@>= 0.2.0 already satisfied
  Verifying dependencies for ansiparse@0.4.0
    Reading official package list
   Checking for jsony@>= 1.1.5
     Info:  Dependency on jsony@>= 1.1.5 already satisfied
  Verifying dependencies for jsony@1.1.5
    Reading official package list
   Checking for db_connector@>= 0.1.0
     Info:  Dependency on db_connector@>= 0.1.0 already satisfied
  Verifying dependencies for db_connector@0.1.0
    Reading official package list
   Checking for bearssl@any version
     Info:  Dependency on bearssl@any version already satisfied
  Verifying dependencies for bearssl@0.2.2
    Reading official package list
   Checking for unittest2@any version
     Info:  Dependency on unittest2@any version already satisfied
  Verifying dependencies for unittest2@0.2.2
   Building takajo/takajo using c backend
  Executing /home/fukusuke/.nimble/bin/nim c --colors:on --noNimblePath -d:release --threads:on -d:nimDebugDlOpen --gcc.exe:musl-gcc --gcc.linkerexe:musl-gcc --passL:-static --passL:../hello_musl/pcre-8.42/.libs/libpcre.a -d:usePcreHeader --dynlibOverride:sqlite3 --passL:../sqlite-autoconf-3460100/.libs/libsqlite3.a --mm:refc -d:nimDebugDlOpen -d:NimblePkgVersion=2.6.0 --path:/home/fukusuke/.nimble/pkgs2/termstyle-0.1.0-4641c9f9e587d5cf04ccd3ce0bf1ef0263a16299 --path:/home/fukusuke/.nimble/pkgs2/cligen-1.7.0-300bd7fdb6e48d2d98e34ed0661206b50331e99c --path:/home/fukusuke/.nimble/pkgs2/nancy-0.1.1-626d5a8937046ed31841d8af728e98013c476768 --path:/home/fukusuke/.nimble/pkgs2/jsony-1.1.5-6aeb83e7481ca8686396a568096054bc668294df --path:/home/fukusuke/.nimble/pkgs2/suru-0.3.1-06ecd8dfb1294b5469d919c83e14db111c1346be --path:/home/fukusuke/.nimble/pkgs2/unittest2-0.2.2-6ac7f7ea74df812a25be5d6d36a59d23789eb508 --path:/home/fukusuke/.nimble/pkgs2/ansiparse-0.4.0-49411d0e4b4e90df4020386b6f14be8c54809a63 --path:/home/fukusuke/.nimble/pkgs2/db_connector-0.1.0-d68319e3785fa937f0465ea915e942b61b6b5442 --path:/home/fukusuke/.nimble/pkgs2/bearssl-0.2.2-d238564ab1e29087395561368505753e4c25ccde -o:/home/fukusuke/takajo/takajo /home/fukusuke/takajo/src/takajo.nim
Hint: used config file '/home/fukusuke/.choosenim/toolchains/nim-2.0.8/config/nim.cfg' [Conf]
Hint: used config file '/home/fukusuke/.choosenim/toolchains/nim-2.0.8/config/config.nims' [Conf]
Hint: used config file '/home/fukusuke/takajo/config.nims' [Conf]
...............................................................................................................................................................................................................................
/home/fukusuke/takajo/src/takajo.nim(16, 8) Warning: use the nimble packages `malebolgia`, `taskpools` or `weave` instead; threadpool is deprecated [Deprecated]
..........
/home/fukusuke/takajo/src/takajopkg/general.nim(452, 17) Hint: 'x' is declared but not used [XDeclaredButNotUsed]
/home/fukusuke/takajo/src/takajopkg/general.nim(437, 9) Hint: 'length' is declared but not used [XDeclaredButNotUsed]
.................
/home/fukusuke/takajo/src/takajopkg/htmlReport.nim(6, 8) Hint: duplicate import of 'os'; previous import here: /home/fukusuke/takajo/src/takajo.nim(18, 8) [DuplicateModuleImport]
/home/fukusuke/takajo/src/takajopkg/htmlReport.nim(6, 12) Hint: duplicate import of 'strutils'; previous import here: /home/fukusuke/takajo/src/takajo.nim(11, 8) [DuplicateModuleImport]
/home/fukusuke/takajo/src/takajopkg/htmlReport.nim(188, 42) Hint: 'e' is declared but not used [XDeclaredButNotUsed]
..
/home/fukusuke/takajo/src/takajopkg/splitJsonTimeline.nim(33, 13) Hint: 'inputFile' is declared but not used [XDeclaredButNotUsed]
.
/home/fukusuke/takajo/src/takajopkg/stackCmdlines.nim(17, 18) Hint: 'otherColumn' is declared but not used [XDeclaredButNotUsed]
.
/home/fukusuke/takajo/src/takajopkg/stackComputers.nim(18, 18) Hint: 'otherColumn' is declared but not used [XDeclaredButNotUsed]
..
/home/fukusuke/takajo/src/takajopkg/stackIpAddresses.nim(18, 18) Hint: 'otherColumn' is declared but not used [XDeclaredButNotUsed]
..
/home/fukusuke/takajo/src/takajopkg/stackProcesses.nim(17, 18) Hint: 'otherColumn' is declared but not used [XDeclaredButNotUsed]
...
/home/fukusuke/takajo/src/takajopkg/stackUsers.nim(48, 18) Hint: 'otherColumn' is declared but not used [XDeclaredButNotUsed]
..
/home/fukusuke/takajo/src/takajopkg/sysmonProcessTree.nim(54, 13) Hint: 's' is declared but not used [XDeclaredButNotUsed]
/home/fukusuke/takajo/src/takajopkg/sysmonProcessTree.nim(124, 17) Hint: 'timeStamp' is declared but not used [XDeclaredButNotUsed]
.
/home/fukusuke/takajo/src/takajopkg/timelineLogon.nim(171, 17) Hint: 'eventId' is declared but not used [XDeclaredButNotUsed]
..
/home/fukusuke/takajo/src/takajopkg/timelineSuspiciousProcesses.nim(20, 14) Hint: 'pidInt' is declared but not used [XDeclaredButNotUsed]
/home/fukusuke/takajo/src/takajopkg/timelineSuspiciousProcesses.nim(20, 5) Hint: 'eventId' is declared but not used [XDeclaredButNotUsed]
/home/fukusuke/takajo/src/takajopkg/timelineSuspiciousProcesses.nim(21, 5) Hint: 'channel' is declared but not used [XDeclaredButNotUsed]
....
/home/fukusuke/takajo/src/takajopkg/ttpVisualizeSigma.nim(45, 9) Hint: 'x' is declared but not used [XDeclaredButNotUsed]
.
/home/fukusuke/takajo/src/takajopkg/vtDomainLookup.nim(12, 9) Hint: 'malicious' is declared but not used [XDeclaredButNotUsed]
.
/home/fukusuke/takajo/src/takajopkg/vtIpLookup.nim(9, 9) Hint: 'malicious' is declared but not used [XDeclaredButNotUsed]
.
/home/fukusuke/takajo/src/takajopkg/vtHashLookup.nim(11, 9) Hint: 'malicious' is declared but not used [XDeclaredButNotUsed]
.
Hint:  [Link]
/usr/bin/ld: ../sqlite-autoconf-3460100/.libs/libsqlite3.a(sqlite3.o): in function `fts5TriTokenize':
sqlite3.c:(.text+0xc041): undefined reference to `__memmove_chk'
/usr/bin/ld: ../sqlite-autoconf-3460100/.libs/libsqlite3.a(sqlite3.o): in function `sqlite3VdbeMemStringify':
sqlite3.c:(.text+0x307d2): undefined reference to `__memcpy_chk'
/usr/bin/ld: ../sqlite-autoconf-3460100/.libs/libsqlite3.a(sqlite3.o): in function `fts5MergePrefixLists':
sqlite3.c:(.text+0x69bab): undefined reference to `__memset_chk'
/usr/bin/ld: ../sqlite-autoconf-3460100/.libs/libsqlite3.a(sqlite3.o): in function `balance_nonroot':
sqlite3.c:(.text+0x7c110): undefined reference to `__memset_chk'
/usr/bin/ld: sqlite3.c:(.text+0x7c258): undefined reference to `__memset_chk'
/usr/bin/ld: ../sqlite-autoconf-3460100/.libs/libsqlite3.a(sqlite3.o): in function `unixOpen':
sqlite3.c:(.text+0x8a5e5): undefined reference to `__memcpy_chk'
/usr/bin/ld: ../sqlite-autoconf-3460100/.libs/libsqlite3.a(sqlite3.o): in function `whereLoopAddOr':
sqlite3.c:(.text+0x9aea3): undefined reference to `__memcpy_chk'
/usr/bin/ld: sqlite3.c:(.text+0x9af8a): undefined reference to `__memcpy_chk'
/usr/bin/ld: ../sqlite-autoconf-3460100/.libs/libsqlite3.a(sqlite3.o):(.data.rel+0x1f10): undefined reference to `fcntl64'
collect2: error: ld returned 1 exit status
Error: execution of an external program failed: 'musl-gcc  @takajo_linkerArgs.txt'
nimble.nim(304)          buildFromDir

    Error:  Build failed for the package: takajo
     Info:  Nimble data file "/home/fukusuke/.nimble/nimbledata2.json" has been saved.