search

exaxorg / accelerator

The Accelerator is a tool for fast and reproducible processing of eBay-scale datasets on a single computer.
https://exax.org
Apache License 2.0
3 stars 1 forks source link

Feature request: dataset_type: allow to merge "date" and "time" columns into a "datetime" one #3

Open pabloyoyoista opened 2 weeks ago

pabloyoyoista commented 2 weeks ago

We have a custom method to sort of workarounds this missing feature: https://gitlab.com/combine-control-systems/accelerator-toolbox/-/blob/main/combine_toolbox/methods/a_generate_beautiful_timestamp.py?ref_type=heads

The timestamp is nice to be generated in multiple situations. So maybe this is more of an option to move that method into the standard method. Or maybe this is not something relevant. However, I still believe that not being able to merge a "date" and "time" into a "datetime" is possibly a missing feature. It would still be the question of whether it is a feature that needs to be in the standard methods

drougge commented 1 week ago

I think a method joining date and time columns to a datetime column is a fine idea. However the method you linked seems to be for strings, which I don't think we want in the standard library. (I.e. I think it should essentially do ds.write(datetime.combine(date, time)).)

pabloyoyoista commented 1 day ago

I think it should essentially do ds.write(datetime.combine(date, time))

That would already be a big benefit, since it would allow to use that instead of our method in logs with date and time split. The method we wrote started with that goal, but then has also evolved into other use-cases. We are happy to carry a more specific set of methods that are not "standard" where we can experiment more freely. And then bring ideas like this from time to time