search

squarewave / bhr.html

Mozilla Public License 2.0
5 stars 2 forks source link

"Past x days" links displays past x days in the past, not from today #26

Open ochameau opened 6 years ago

ochameau commented 6 years ago

Is that expected? Do we have a delay in data processing?

For example, when I open past 9 days for light hangs: https://arewesmoothyet.com/?mode=explore&durationSpec=128_512&historical=false I get results from Nov 19th to Nov 26th, whereas we are today Dec 5th. I would expect to see from Nov 27th to today, Dec 5th.

squarewave commented 6 years ago

Eh, it is a bit of a misnomer. It should probably read something like "9-day window". The problem is it's based on data from a job that runs once per day, but the job is somewhat flaky because it has to do transformations on the data that cause Spark to OOM. I'm hoping to talk to a Spark expert in Austin about it. In any case, I will kick it again to get it running properly.

ochameau commented 6 years ago

Ah ok. That makes sense. I imagine I was mostly disturbed by how outdated it is. It is around two weeks old data. If you resolve the OOM but still get a couple of days old data, I think it will be reasonable behavior.

squarewave commented 6 years ago

Should be current (minus today, because that data isn't really in yet) now.