search

borevitzlab / timestreamlib

DEPRECATED. Please use the current version of the TimeStream tools at https://gitlab.com/appf-anu/pyts2.
https://gitlab.com/appf-anu/pyts2
GNU General Public License v3.0
7 stars 4 forks source link

Change csv timestamp format. #87

Closed TimeScience closed 9 years ago

TimeScience commented 10 years ago

Someone clarify (and create a standard) for the timestamp in all output files. Currently conversion of timestamps from Unix to Matlab yields nonsensical dates (i.e. either conversion is wrong our output values are inaccurate somehow or not in UNIX time)

Joelgranados commented 10 years ago

Currently we handle timestamps with ts = time.mktime(context.origImg.datetime.timetuple()) * 1000. This represents milliseconds after the epoc. Epoc can change whether you execute the pipeline in windows or linux. Given that we are executing only in linux, the Epoc is 1st Jan 1970.

You are getting strange values because you are considering seconds after Epoc instead of milliseconds after Epoc. Just divide by 1000 in the conversion to Matlab.

This is a non-issue

kdm9 commented 9 years ago

It would be good if we could use the inbuilt timestream format, i.e. "%Y%m%d%H%M%S%02d", which goes to 10 millisecond resolution. This should be plenty for all use cases we can think of, and in any case we would be limited by the timestamps on images...

Joelgranados commented 9 years ago

Not sure I understand.

1) Whats wrong with the current timestamps? Are they not general enough? in what cases have they failed? 2) Note that using another format will not remove the extra step of having to reformat the timestamp when you are creating graphs. What is the added value with this new format? 3) The csv files are not meant to be human readable. they are meant to go to the statistician and serve as input for further analysis. They are intermediate files. 4) What is needed is not a new timestamp format but an easy way to visualize the csv.

kdm9 commented 9 years ago

It's not a new format, it's using the exact same format for dates we use everywhere else. It's way less ambiguous than seconds/millisecs since "epoch". Epoch depends where you are. '2014_09_20_16_30_00_00' is always parseable (any worthwhile language has some implementation of strptime), and as a bonus is human readable. I insist that we use a consistent format, and the one I describe above is the most common throughout the project.

If you worried about extra code, look at timestream.parse.ts_parse_date and ts_format_date. This has all been written.

This library should be cross platform, even if we only ever do things on GNU/linux. Relying on operating system features is very bad practice.

Joelgranados commented 9 years ago

This is a duplicate from https://github.com/borevitzlab/timestreamlib/issues/89