Open MaxOrelus opened 5 years ago
Hi @MaxOrelus! Thanks for submitting an issue. I just updated the dependencies to make sure we're pulling in the latest PDK and client library. I pushed the updated lock file.
If you pull the latest master and then do a dep ensure
you should be good to go!
I will caution you though that this code is a proof of concept, and probably quite a bit of tuning is needed before we can ingest at a "very high rate" depending on what that means to you. Pilosa can support extremely high ingest rates though, so let me know what you find and we'd be happy to help optimize things.
Hi @jaffee,
I went ahead and pulled the latest code. I tested both CLI commands and got varying results. Using the interface method I was successful at ingesting the data! 👍
sudo picap -i en0 -n net
Using the pcap file method had the console throw an error. I've documented the steps I took below, but the interface method works for my research purpose, Thanks!
When I ran the picap -n net -f ./malware.pcap
:
go get -u github.com/pilosa/picap
dep ensure
go install ./cmd/picap
I then ran the following command:
picap -n eve_logs -f ./malware2.pcap
I received following error:
picap -n eve_logs -f ./malware2.pcap
2019/02/22 22:26:29 storing mapping data in /var/folders/_7/5jcrs23s6fn4c05b5pcksb6w0000gn/T/485677159
ingest.Record: 1 panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x45871f7]
goroutine 142 [running]:
github.com/pilosa/picap.reifyPacket(0x0, 0x0, 0x43aa11a, 0xc0000d4900, 0xc0001ec330)
/Users/maxorelus/.go/src/github.com/pilosa/picap/picap.go:170 +0x57
github.com/pilosa/picap.(*NetSource).Record(0xc0000daca0, 0x46f1fee, 0xf, 0x1, 0x3ff0000000000000)
/Users/maxorelus/.go/src/github.com/pilosa/picap/picap.go:134 +0xa3
github.com/pilosa/picap/vendor/github.com/pilosa/pdk.(*Ingester).Run.func1(0xc000310090, 0xc000860090)
/Users/maxorelus/.go/src/github.com/pilosa/picap/vendor/github.com/pilosa/pdk/ingest.go:91 +0x79
created by github.com/pilosa/picap/vendor/github.com/pilosa/pdk.(*Ingester).Run
/Users/maxorelus/.go/src/github.com/pilosa/picap/vendor/github.com/pilosa/pdk/ingest.go:85 +0x82
Hi,
I'm currently in search of a storage solution that can ingest at a very high rate. The use-case listed on your site, "Monitoring Network Traffic", has me very interested in possibly implementing Pilosa.
I've tried following the use-case to get both Pilosa and Picap setup, but unfortunately, I was not able to view the use-case in my local environment. Pilosa server was up and running, but I think the issue is Picap related.
I tried Picap both against the official Pilosa docker image and I also went ahead and created my own image so I can pass in a config.toml. Both were running fine, but against Picap was never able to reach Pilosa.
I tried:
Error Example:
Any guidance is appreciated.
Best regards, Max
config.toml for custom built docker image: