search

Security-Onion-Solutions / securityonion

Security Onion is a free and open platform for threat hunting, enterprise security monitoring, and log management. It includes our own interfaces for alerting, dashboards, hunting, PCAP, detections, and case management. It also includes other tools such as osquery, CyberChef, Elasticsearch, Logstash, Kibana, Suricata, and Zeek.
https://securityonion.net
3.28k stars 507 forks source link

FIX: steno multiple thread collision #7412

Closed dougburks closed 1 year ago

dougburks commented 2 years ago

Originally posted by @jtgreen-cse in https://github.com/Security-Onion-Solutions/securityonion/discussions/7372

I'd like to use more than one thread with steno. My storage performs better if multiple processes are writing to it. I've added:

steno:
  threads: 8

to global.sls. Things work well for a while until eventually I get this in stenographer.log:

2022-03-01T17:15:55.954502Z T:442717 [stenotype.cc:567] Starting, page size is 4096
2022-03-01T17:15:55.954645Z T:442717 [stenotype.cc:594] Setting up AF_PACKET sockets for packet reading
panic: runtime error: slice bounds out of range [-822726196:]

goroutine 20 [running]:
github.com/golang/leveldb/table.block.seek.func1(0xc45bd06, 0xc000a46000)
        /home/mreeves/rpmbuild/BUILD/stenographer-e8db1ee8f6b62790ebc8081e464dcedd7c961766/src/github.com/golang/leveldb/table/reader.go:60 +0x20a
sort.Search(0x188b7a0d, 0xc00007d930, 0xc0001db530)
        /usr/lib/golang/src/sort/search.go:66 +0x58
github.com/golang/leveldb/table.block.seek(0xc000a46000, 0xd25ec, 0xd25f1, 0xbd6d20, 0x16fc060, 0xc0008c8012, 0x1, 0x1, 0xc00007da00, 0x40fb38, ...)
        /home/mreeves/rpmbuild/BUILD/stenographer-e8db1ee8f6b62790ebc8081e464dcedd7c961766/src/github.com/golang/leveldb/table/reader.go:59 +0x33a
github.com/golang/leveldb/table.(*tableIter).nextBlock(0xc0002be180, 0xc0008c8012, 0x1, 0x1, 0x0, 0xc0008c8012)
        /home/mreeves/rpmbuild/BUILD/stenographer-e8db1ee8f6b62790ebc8081e464dcedd7c961766/src/github.com/golang/leveldb/table/reader.go:197 +0x1d4
github.com/golang/leveldb/table.(*Reader).find(0xc000488820, 0xc0008c8012, 0x1, 0x1, 0x0, 0x0, 0x16fc060, 0xc00007db00)
        /home/mreeves/rpmbuild/BUILD/stenographer-e8db1ee8f6b62790ebc8081e464dcedd7c961766/src/github.com/golang/leveldb/table/reader.go:377 +0x1ea
github.com/golang/leveldb/table.(*Reader).Get(0xc000488820, 0xc0008c8012, 0x1, 0x1, 0x0, 0x1, 0x100000000000003, 0x0, 0x30, 0xc0004866f0)
        /home/mreeves/rpmbuild/BUILD/stenographer-e8db1ee8f6b62790ebc8081e464dcedd7c961766/src/github.com/golang/leveldb/table/reader.go:337 +0x8d
github.com/google/stenographer/indexfile.NewIndexFile(0xc0004866f0, 0x30, 0xc00021fb90, 0x3, 0xaff380, 0x3)
        /home/mreeves/rpmbuild/BUILD/stenographer-e8db1ee8f6b62790ebc8081e464dcedd7c961766/src/github.com/google/stenographer/indexfile/indexfile.go:67 +0x23d
github.com/google/stenographer/blockfile.NewBlockFile(0xc000486660, 0x30, 0xc00021fb90, 0xc000486660, 0x30, 0x1)
        /home/mreeves/rpmbuild/BUILD/stenographer-e8db1ee8f6b62790ebc8081e464dcedd7c961766/src/github.com/google/stenographer/blockfile/blockfile.go:63 +0x14b
github.com/google/stenographer/thread.(*Thread).trackNewFile(0xc000276aa0, 0xc0000c0020, 0x10, 0x10, 0x16fe000)
        /home/mreeves/rpmbuild/BUILD/stenographer-e8db1ee8f6b62790ebc8081e464dcedd7c961766/src/github.com/google/stenographer/thread/thread.go:172 +0xc9
github.com/google/stenographer/thread.(*Thread).syncFilesWithDisk(0xc000276aa0)
        /home/mreeves/rpmbuild/BUILD/stenographer-e8db1ee8f6b62790ebc8081e464dcedd7c961766/src/github.com/google/stenographer/thread/thread.go:139 +0x2af
github.com/google/stenographer/thread.(*Thread).SyncFiles(0xc000276aa0)
        /home/mreeves/rpmbuild/BUILD/stenographer-e8db1ee8f6b62790ebc8081e464dcedd7c961766/src/github.com/google/stenographer/thread/thread.go:363 +0x45
github.com/google/stenographer/env.(*Env).syncFiles(0xc0001761c0)
        /home/mreeves/rpmbuild/BUILD/stenographer-e8db1ee8f6b62790ebc8081e464dcedd7c961766/src/github.com/google/stenographer/env/env.go:267 +0x50
github.com/google/stenographer/env.(*Env).callEvery(0xc0001761c0, 0xc0002566a0, 0x37e11d600)
        /home/mreeves/rpmbuild/BUILD/stenographer-e8db1ee8f6b62790ebc8081e464dcedd7c961766/src/github.com/google/stenographer/env/env.go:180 +0x83
created by github.com/google/stenographer/env.New
        /home/mreeves/rpmbuild/BUILD/stenographer-e8db1ee8f6b62790ebc8081e464dcedd7c961766/src/github.com/google/stenographer/env/env.go:131 +0x3f8

I'm going to guess that:

https://github.com/Security-Onion-Solutions/securityonion/pull/5911#pullrequestreview-782014392

is still a thing? May I suggest salt create separate folders for each thread so that they don't clobber each others files. I didn't create an issue since steno threads probably aren't ready for prime time. Just letting you know that there's at least one person waiting for it :)

jtgreen-cse commented 2 years ago

Thanks Doug. I've actually implemented this locally but my nsm filled up because of the issue below (I am aware this is an unsupported config):

https://github.com/google/stenographer/issues/214

I had patched steno's config and init.sls to make the multiple folders and use them. It worked fine until stenotype tried to cleanup at 10% free disk space on /nsm .. because it's working on one disk, the threads don't clean up the same as if each thread had a separate disk. The above issue describes this.

jtgreen-cse commented 2 years ago

Not sure how you're building the rpm referenced in so-steno's Dockerfile, but maybe you can add a patch from https://github.com/rskjetlein/stenographer/commit/6f6d2cb258066a05d8c55f4548f5cfc06847a6bd to the source you use.

jtgreen-cse commented 2 years ago

Fork that includes what I did to get so-steno to write to multiple folders.

https://github.com/jtgreen-cse/securityonion/tree/jtgreen-cse-patch-1

jtgreen-cse commented 2 years ago

I've compiled stenotype and stenographer with the random thread picker commit as seen in the previous comments. I inserted the new binaries in one of my so-steno containers and manually applied the capabilities and correct perms. Everything now deletes correctly, no more filling up \nsm. I'm looking forward to seeing this is prod as I don't want to maintain this duct tape :)

dougburks commented 2 years ago

Thanks for the updates @jtgreen-cse !

We'll take a look as time allows.

TOoSmOotH commented 1 year ago

2.4 has the option to run multiple steno threads pointing to different locations.