brexhq / substation

Substation is a toolkit for routing, normalizing, and enriching security event and audit logs.
https://substation.readme.io
MIT License
322 stars 16 forks source link

Unable to run locally on macOS - missing substation-file directory #229

Closed Guled closed 1 week ago

Guled commented 1 week ago

Describe the bug Following the instructions located here https://substation.readme.io/docs/try-substation-on-macos-linux and it appears to not be working correctly. I am getting the following error:

substation $ go build -o ./examples/substation-file ./cmd/development/substation-file/
stat /Users/guled.abdilahi/sourcecode/substation/cmd/development/substation-file: directory not found

To Reproduce Steps to reproduce the behavior:

  1. Open a terminal
  2. Copy/Paste the quick install/run instruction
  3. See error

Expected behavior I expected to the example program to comply and run successfully,

Screenshots Here is my output with additional validation steps to check for files existing

sourcecode $ git clone https://github.com/brexhq/substation.git 
Cloning into 'substation'...
remote: Enumerating objects: 9956, done.
remote: Counting objects: 100% (3874/3874), done.
remote: Compressing objects: 100% (1696/1696), done.
remote: Total 9956 (delta 2593), reused 2963 (delta 2019), pack-reused 6082 (from 1)
Receiving objects: 100% (9956/9956), 6.18 MiB | 7.15 MiB/s, done.
Resolving deltas: 100% (5940/5940), done.
sourcecode $ cd substation 
substation $ ls -la
total 248
drwxr-xr-x   29 guled.abdilahi  staff    928 Sep 17 19:17 .
drwxr-xr-x   56 guled.abdilahi  staff   1792 Sep 17 19:17 ..
drwxr-xr-x    5 guled.abdilahi  staff    160 Sep 17 19:17 .devcontainer
-rw-r--r--    1 guled.abdilahi  staff    201 Sep 17 19:17 .dockerignore
drwxr-xr-x   12 guled.abdilahi  staff    384 Sep 17 19:17 .git
drwxr-xr-x    6 guled.abdilahi  staff    192 Sep 17 19:17 .github
-rw-r--r--    1 guled.abdilahi  staff   2403 Sep 17 19:17 .gitignore
-rw-r--r--    1 guled.abdilahi  staff   5870 Sep 17 19:17 .golangci.yml
drwxr-xr-x    3 guled.abdilahi  staff     96 Sep 17 19:17 .vscode
-rw-r--r--    1 guled.abdilahi  staff  26229 Sep 17 19:17 CHANGELOG.md
-rw-r--r--    1 guled.abdilahi  staff     27 Sep 17 19:17 CODEOWNERS
-rw-r--r--    1 guled.abdilahi  staff   5495 Sep 17 19:17 CODE_OF_CONDUCT.md
-rw-r--r--    1 guled.abdilahi  staff   5915 Sep 17 19:17 CONTRIBUTING.md
-rw-r--r--    1 guled.abdilahi  staff   1061 Sep 17 19:17 LICENSE
-rw-r--r--    1 guled.abdilahi  staff  14433 Sep 17 19:17 README.md
-rw-r--r--    1 guled.abdilahi  staff    177 Sep 17 19:17 SECURITY.md
-rw-r--r--    1 guled.abdilahi  staff   1265 Sep 17 19:17 VERSIONING.md
drwxr-xr-x    6 guled.abdilahi  staff    192 Sep 17 19:17 build
drwxr-xr-x    5 guled.abdilahi  staff    160 Sep 17 19:17 cmd
drwxr-xr-x   73 guled.abdilahi  staff   2336 Sep 17 19:17 condition
drwxr-xr-x    3 guled.abdilahi  staff     96 Sep 17 19:17 config
drwxr-xr-x    7 guled.abdilahi  staff    224 Sep 17 19:17 examples
-rw-r--r--    1 guled.abdilahi  staff   2395 Sep 17 19:17 go.mod
-rw-r--r--    1 guled.abdilahi  staff  12917 Sep 17 19:17 go.sum
drwxr-xr-x   17 guled.abdilahi  staff    544 Sep 17 19:17 internal
drwxr-xr-x    4 guled.abdilahi  staff    128 Sep 17 19:17 message
-rw-r--r--    1 guled.abdilahi  staff   1912 Sep 17 19:17 substation.go
-rw-r--r--    1 guled.abdilahi  staff   5417 Sep 17 19:17 substation_test.go
drwxr-xr-x  149 guled.abdilahi  staff   4768 Sep 17 19:17 transform
substation $ ls examples/cmd/client/file/substation 
config.jsonnet  data.json   event.libsonnet main.go     send.libsonnet
substation $ sh build/scripts/config/compile.sh 
substation $ ls examples/cmd/client/file/substation
config.json config.jsonnet  data.json   event.libsonnet main.go     send.libsonnet
substation $ go build -o ./examples/substation-file ./cmd/development/substation-file/
stat /Users/guled.abdilahi/sourcecode/substation/cmd/development/substation-file: directory not found
substation $ ls ./examples/ 
Makefile    README.md   cmd     config      terraform

Build (please complete the following information):

Additional context N.A.

jshlbrd commented 1 week ago

Hey @Guled, thanks for stopping by (and hope you're doing well)! I think what happened here is that the v1.7 documentation was mistakenly updated with changes from the upcoming v2.0 release. These commands should work if you run them from the root directory:

$ sh build/scripts/config/compile.sh
$ cd examples/cmd/client/file/substation
$ go build -o .
$ ./substation -config ./config.json -file ./data.json

{"foo":"bar","qux":"quux"}

If you want to try v2.0, then the branch is v2 and you can follow these steps from the root directory:


$ sh build/scripts/config/compile.sh && \
$ go build -o ./examples/substation-file ./cmd/development/substation-file/ && \
$ ./examples/substation-file -config ./examples/transform/aggregate/summarize/config.json -file ./examples/transform/aggregate/summarize/data.jsonl

{"client":"10.1.1.3","server":"8.8.4.4","timestamp":1674429064,"count":2,"bytes_total":40}
{"client":"10.1.1.2","server":"8.8.4.4","timestamp":1674429065,"count":6,"bytes_total":112}
{"client":"10.1.1.3","server":"8.8.8.8","timestamp":1674429066,"count":6,"bytes_total":52}
{"client":"10.1.1.4","server":"1.2.3.4","timestamp":1674429054,"count":1,"bytes_total":2400}
{"client":"10.1.1.2","server":"8.8.8.8","timestamp":1674429067,"count":4,"bytes_total":72}

I'll update the readme.io docs so it's fixed there, too. I think this v1.7 recipe is still accurate, and it walks through the local testing process (if you're interested).

Guled commented 1 week ago

Thanks @jshlbrd , now I am able to run substation locally using the updated instructions. Can't wait to explore this framework more!

jshlbrd commented 1 week ago

@Guled No problem! btw, v2.0 should be released next week.