Picap unable to send data Pilosa

[MaxOrelus]

Hi,

I'm currently in search of a storage solution that can ingest at a very high rate. The use-case listed on your site, "Monitoring Network Traffic", has me very interested in possibly implementing Pilosa.

I've tried following the use-case to get both Pilosa and Picap setup, but unfortunately, I was not able to view the use-case in my local environment. Pilosa server was up and running, but I think the issue is Picap related.

I tried Picap both against the official Pilosa docker image and I also went ahead and created my own image so I can pass in a config.toml. Both were running fine, but against Picap was never able to reach Pilosa.

I tried:

• picap -f ./malware.pcap -n net
• picap -i en0 -n net

Error Example:

2019/02/22 14:42:39 setting up frame 'http-method': creating frame '{http-method ranked 100000  []}': Error: Server error (404)

Any guidance is appreciated.

Best regards, Max

config.toml for custom built docker image:

bind = ":10101"
data-dir = "/data"
log-path = ""
max-writes-per-request = 5000
verbose = true

[anti-entropy]
  interval = "10m0s"

[cluster]
  coordinator = true
  disabled = false
  hosts = []
  long-query-time = "1m0s"
  replicas = 0

[gossip]
  interval = "200ms"
  key = ""
  nodes = 3
  port = "14000"
  probe-interval = "1s"
  probe-timeout = "500ms"
  push-pull-interval = "30s"
  seeds = []
  stream-timeout = "10s"
  suspicion-mult = 4
  to-the-dead-time = "30s"

[handler]
  allowed-origins = []

[metric]
  diagnostics = true
  host = ""
  poll-interval = "0s"
  service = "none"

[tls]
  certificate = ""
  key = ""
  skip-verify = false

[tracing]
  agent-host-port = ""
  sampler-param = 0.001
  sampler-type = "remote"

[translation]
  map-size = 0
  primary-url = ""

[jaffee]

Hi @MaxOrelus! Thanks for submitting an issue. I just updated the dependencies to make sure we're pulling in the latest PDK and client library. I pushed the updated lock file.

If you pull the latest master and then do a dep ensure you should be good to go!

I will caution you though that this code is a proof of concept, and probably quite a bit of tuning is needed before we can ingest at a "very high rate" depending on what that means to you. Pilosa can support extremely high ingest rates though, so let me know what you find and we'd be happy to help optimize things.

[MaxOrelus]

Hi @jaffee,

I went ahead and pulled the latest code. I tested both CLI commands and got varying results. Using the interface method I was successful at ingesting the data! 👍

sudo picap -i en0 -n net

Using the pcap file method had the console throw an error. I've documented the steps I took below, but the interface method works for my research purpose, Thanks!

When I ran the picap -n net -f ./malware.pcap:

go get -u github.com/pilosa/picap
dep ensure
go install ./cmd/picap

I then ran the following command:

picap -n eve_logs -f ./malware2.pcap

I received following error:

picap -n eve_logs -f ./malware2.pcap
2019/02/22 22:26:29 storing mapping data in /var/folders/_7/5jcrs23s6fn4c05b5pcksb6w0000gn/T/485677159
ingest.Record: 1 panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x45871f7]

goroutine 142 [running]:
github.com/pilosa/picap.reifyPacket(0x0, 0x0, 0x43aa11a, 0xc0000d4900, 0xc0001ec330)
    /Users/maxorelus/.go/src/github.com/pilosa/picap/picap.go:170 +0x57
github.com/pilosa/picap.(*NetSource).Record(0xc0000daca0, 0x46f1fee, 0xf, 0x1, 0x3ff0000000000000)
    /Users/maxorelus/.go/src/github.com/pilosa/picap/picap.go:134 +0xa3
github.com/pilosa/picap/vendor/github.com/pilosa/pdk.(*Ingester).Run.func1(0xc000310090, 0xc000860090)
    /Users/maxorelus/.go/src/github.com/pilosa/picap/vendor/github.com/pilosa/pdk/ingest.go:91 +0x79
created by github.com/pilosa/picap/vendor/github.com/pilosa/pdk.(*Ingester).Run
    /Users/maxorelus/.go/src/github.com/pilosa/picap/vendor/github.com/pilosa/pdk/ingest.go:85 +0x82