search

rapid7 / leexportpy

Logentries Log Export application that gives you the ability to synchronize your Logentries data with 3rd parties.
MIT License
2 stars 7 forks source link

Missing data #13

Closed steven2308 closed 7 years ago

steven2308 commented 7 years ago

I'm trying to get 18 different metrics from LogEntries to Hosted graphite, but there's a lot of gaps in HG. I haven't been able to find the cause of this problem.

I attach the configuration and the last 2000 lines of the log. Also this is how it looks like in HG: leexportpy bug2 The graphic is displaying the metrics MyAppProd1HTTP2XX and MyAppProd2HTTP2XX. Which as you can see in the logs are being sent a few times between 22:05:00 and 22:07:50 but then are not sent until 22:12:06. (query period and query range are set to 50s). This can be seen easily in this processed log searching for the HTTP2XX metric with 10 lines of context: LeExportpyLogHTTP2XX.txt

This problems happens whether it's daemonized or not. I'm running on Ubuntu 16.10, Python 2.7.12 and leexportpy 0.1.8. I want to clarify that this is different to issues #11 and #12 since I have a fix for those locally.

Thank you. Full log: LeExportpyLog2000.txt Config: my-app-config.ini.txt

stopal-r7 commented 7 years ago

hey @steven2308, according to your logs, requests are being made properly in the time periods that the graph shows gaps. I don't think this is an leexportpy issue.

steven2308 commented 7 years ago

Query period and query range are set to 50s, so it's supposed to ask to LE for information about the last 50 seconds, every 50 seconds. It starts working correctly for a while, but as can be seen in the logs suddenly it stops asking for a metrics for a time of 4m16s, which produces the gap. I haven't found a reason for this behaviour.

stopal-r7 commented 7 years ago

Hey Steven, I am seeing some error codes returned by leserver - 404 and 503 specifically. Do you get these errors often? Apparently the gap happened just at that time period.

I am planning to create an internal ticket for this soon as it looks interesting and no matter what export fails, others should be proceeding as it is. In the meantime feel free to submit PRs for a possible fix, I will be more than happy to review.

steven2308 commented 7 years ago

I hadn't actually noticed those error codes, but I can see they're clearly related to the gaps. Also I haven't found a fix for this, sorry.

stopal-r7 commented 7 years ago

Hey @steven2308, this is not an leexportpy issue. I am following this up internally. In the meantime you can contact our support regarding 404 and 503's returned by REST api. Thanks.