Bug with UNIX timestamp conversion

[ocafebabe]

I think there's a bug with the UNIX timestamp to local time conversion because If I register an event with a now value of: 1406952000 (Sat Aug 2 00:00:00 EDT 2014) it shows up in the web interface with a value of: Fri Aug 01 2014 20:00:00 GMT-0400 (EDT)!

[o]

Hello again,

I'm at holiday, and i haven't a computer for reproducing this bug.

But i feel, probably there is a bug with d3/rickshaw. Have you looked at results of fetch command from cli?

And how is your environment? Os? Python version? 08 Ağu 2014 03:43 tarihinde "ocafebabe" notifications@github.com yazdı:

I think there's a bug with the UNIX timestamp to local time conversion because If I register an event with a now value of: 1406952000 (Sat Aug 2 00:00:00 EDT 2014) it shows up in the web interface with a value of: Fri Aug 01 2014 20:00:00 GMT-0400 (EDT)!

— Reply to this email directly or view it on GitHub https://github.com/o/simmetrica/issues/17.

[ocafebabe]

Hello Osman! No problem! Relax and enjoy your vacation ;)

I'll try with the CLI and let you know my findings!

But in the meantime here's my environment:

• openSUSE 12.3
• Python 2.7.3

[ocafebabe]

Ok just to give you a heads up, here's what I have tried!

Query foo events between 2014-08-01 to 2014-08-31 and aggregate them at the day level:

$ simmetrica-cli.py query foo 1406865600 1409457600 --resolution=day 1406851200 0 1406937600 0 1407024000 0 ... 1409356800 0

If you check the value of the first returned timestamp you can see that it's off by -4 hours:

$ date -d @1406851200 Thu Jul 31 20:00:00 EDT 2014

And it's the same for all timestamps!

I'll keep investigating...

[d0ugal]

I think I can sort-of replicate the issue. Is this what you mean? It's actually the overlay for the dividers that shows this and not the data points.

I think its technically correct, because UTC midnight would be 01:00 here in BST (british summer time). However, it isn't what you'd expect and the further you are from UTC the odder it looks :)

[ocafebabe]

I understand your point but still this is really annoying! Because of the time difference/conversion I have to wait 1 day to see today's events (e.g. if I upload a bunch of data today I'll have to wait until tomorrow to see them in the chart)!

[o]

Hello.

There is two time constraint in simmetrica:

Server time: Used for storing, retrieving data in redis. Your browser time: Used for showing graphs.

I suspect something is wrong with this time constraints.

Just trying on fresh Mac Book, installed simmetrica from pip (1.0.1) and redis from homebrew. (w/ python 2.7.5). And pushed two example event to simmetrica.

Looks like it works correctly. If you want i can try to install simmetrica to Debian and Ubuntu (for tomorrow)

Have you tried with different browser or server?

[ocafebabe]

Here's an example of the problem:

1) The last data on the right side of the chart is actually from yesterday (2014-08-24)! But it shows as it was generated on the "2014-08-23" (which seems to be a conversion from UTC midnight to my timezone: GMT-0400 EDT)...

2) And the other problem is that today's event "2014-08-25" doesn't even show up on the chart! It will appear only tomorrow...

Something is wrong here...

Could you try it on your side with a resolution of type "day" and and "timespan" of 1 month?

Thanks

[o]

Hello!

I understand (looks like same problem in graphite, it have a keepLastValue function as a workaround) and (hopefully) fixed your problem.

I created gap-last-value branch, and patched bin/simmetrica-app.py file. Can you try it? If it's ok, i'll merge this changes to master and i'll send a release to pypi (for tomorrow).

[ocafebabe]

Well it fixed the second problem but the first one is still there: all timestamps are converted from UTC midnight to my timezone which is kind of odd and misleading!

If I generate an event with this timestamp: date --date="2014-08-25 00:00:00 EDT" +"%s"

I get and epoch timestamp of: 1408939200

Which in returns translate back to exactly the same value: date -d @1408939200 Mon Aug 25 00:00:00 EDT 2014

So when I query the application (with a "day" resolution) it should return all the events for the "2014-08-25" grouped together (the time part should be ignored) and NOT convert the timestamp back from midnight UTC to the user's local timezone... Otherwise all timestamps are offsetted by the user's local timezone (like in my case -0400 hours)!

Maybe the aggregation logic should be applied after retrieving the data from the backend (instead of on the database itself)?