Teevity / ice

AWS Usage Tool
2.85k stars 434 forks source link

Billing records stop #279

Closed klausfiend closed 6 years ago

klausfiend commented 6 years ago

This one has me scratching my head. I've looked over the detailed billing files from November and December, and all the information is in there and looks normal, but for some reason, our billing records stopped dead at 9 AM on November 19th (see screenshot attached).

Every configuration of the application I've tried -- standalone, on a VM, or in a container -- shows me the same dismal picture, and I can't find anything in the logs to explain what might have happened.

Has this happened to anyone else? Where should I dig around in the code to debug this?

aws_usage

nfonrose commented 6 years ago

The best way to debug this is to create a very small billing file that contains just two lines:

Don't forget the include header line. You can also disable RI polling. And then try to debug from there. This will reduce the log noise while you debug.

Nicolas

-- Nicolas Fonrose | Teevity | Founder +33.6.61.35.43.31 https://teevity.com - Cloud Costs Analytics built on NetflixOSS twitter - @nfonrose / @teevity

On Wed, Dec 13, 2017 at 8:27 AM, klausfiend notifications@github.com wrote:

This one has me scratching my head. I've looked over the detailed billing files from November and December, and all the information is in there and looks normal, but for some reason, our billing records stopped dead at 9 AM on November 19th (see screenshot attached).

Every configuration of the application I've tried -- standalone, on a VM, or in a container -- shows me the same dismal picture, and I can't find anything in the logs to explain what might have happened.

Has this happened to anyone else? Where should I dig around in the code to debug this?

[image: aws_usage] https://user-images.githubusercontent.com/16393974/33926680-efd26b66-df93-11e7-8e92-0d9641abb500.png

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Teevity/ice/issues/279, or mute the thread https://github.com/notifications/unsubscribe-auth/AACUpSR7p9qT0Pdncfou7ZZnS1hY-Vdnks5s_3xSgaJpZM4RAHWZ .

klausfiend commented 6 years ago

So I've spent the last few days poring over various permutations of the original billing data from the affected interval, and my hypothesis at this point is that one or more of the intermediate data files for 2017-11 used by the reader was somehow corrupted. I've moved them all aside for that month in the meantime; I'm waiting for the processor to regenerate them to confirm my theory, at which point I'll blame it on our PaaS environment, finish migrating the application to something less tetchy, and close this issue.

However, I am still curious as to how to disable RI polling as it doesn't seem obvious to me from the source how that's done. Is that documented somewhere?