search

cloudflare / logshare

logshare is a library & CLI client for retrieving logs from Cloudflare's Enterprise Log Share service.
https://support.cloudflare.com/hc/en-us/articles/216672448
BSD 3-Clause "New" or "Revised" License
80 stars 24 forks source link

"Error Streaming Data" appending to end of JSON response #40

Open snblitz opened 5 years ago

snblitz commented 5 years ago

I am getting a few hundred thousand records per some unit time fetching around a minutes worth of data at a time. Maybe once every three days or so I get a JSON response along the lines of

(a few thousand JSON records) {the last JSON record ..."WorkerSubrequestCount":0,"ZoneID":277273} Error Streaming Data

The 'Error Streaming Data' is outside the JSON blocks and just tacked on to the end. Any idea what it means and how my code should respond to it?

At the moment if I get a response with that string tacked on the end, I toss the block and ask for it again.

jacobbednarz commented 5 years ago

I've not experienced this and we pull similar amounts of data. Do you have network blips while this is in use? Perhaps investigating using a tool like wireshark will help you identify whether it's Cloudflare or the client having issues with the stream.

snblitz commented 5 years ago

Hi jacob. Do you filter your data for valid json? I just take the json in and hope it is all valid. My own suspicion is a library in the bowels of Go is tossing an error due to, as you suspect, a network error. Since I have a work around (retrying) I do not have much incentive to explore the problem further.

jacobbednarz commented 5 years ago

We only rely on the logshare-cli for adhoc searching these days. We're now using Cloudflare's Logpush service (still in beta) which dumps the log entries directly into a S3 bucket of our choosing and then it gets slurped into our log aggregation tool.

If you're an Enterprise customer, you can talk to your Technical Account Manager about whether this functionality is right for you otherwise you'll need to discuss it with Cloudflare Support.