search

dysonance / Temporal.jl

Time series implementation for the Julia language focused on efficiency and flexibility
Other
100 stars 25 forks source link

Support DataStreams and/or IterableTables #3

Closed femtotrader closed 5 years ago

femtotrader commented 7 years ago

DataStreams is a fast, generic framework for transferring table-like data structures in Julia https://github.com/JuliaData/DataStreams.jl http://juliadata.github.io/DataStreams.jl/stable/ https://www.youtube.com/watch?v=N39V6JMWazo

Implementing a TS.Sink will help to convert easily DataFrame (from DataFrames.jl) or any other supported DataStream Source (CSV, SQLite, ODBC...) to Temporal.TS.

Implementing a TS.Source will help to convert a Temporal.TS to any DataStream Sink (DataFrame, CSV, SQLite, ODBC...)

dysonance commented 7 years ago

@femtotrader I will definitely keep an eye on this package as it matures. Currently I have implemented some convenience methods for tabular time series data I/O to get started down this road, though I agree in the future it would be good to create a more convenient source/sink framework so that TS objects can better interface with other types (DataFrames in particular would be good to get working soon). For now, take a look at the following functions, hopefully they will be of some help working with external data.

femtotrader commented 7 years ago

You might be interested by https://github.com/femtotrader/DataReaders.jl

femtotrader commented 7 years ago

Query.jl is a package for querying julia data sources. It can filter, project, join and group data from any iterable data source, including all the sources supported in IterableTables.jl

I don't know what are the pros and cons between IterableTables.jl and DataStreams.jl

Pinging @davidanthoff and @quinnj

femtotrader commented 7 years ago

Temporal.jl currently have no support for DataStreams.jl or IterableTables.jl. I'm pretty sure that integrating one of this library could be a good idea to provide better I/O support. It may also help plotting https://github.com/dysonance/Temporal.jl/issues/9

davidanthoff commented 7 years ago

I just added Temporal.jl integration with IterableTables.jl here: https://github.com/davidanthoff/IterableTables.jl/pull/37. I'll merge and tag a new version as soon as tests pass.

This gives you all the conversions from IterableTables (DataFrame etc.), plus plotting, GLM etc. integration. And you can use a TS as a source of a Query.jl query, and a sink. I have a number of cool things related to file IO coming for this whole space and you'll get all of that automatically as well once it is done.

davidanthoff commented 7 years ago

Oh, and you get some integration with DataStreams via the IterableTables PR. You'll be able to do TS(source) where source is any DataStreams.Source, and you can also write to DataStreams sinks. That will essentially give you the fields based streaming option, if you want the column based DataStreams story, you'll need to add that separately.

femtotrader commented 7 years ago

Thanks @davidanthoff If I understand well , I think it should also help https://github.com/dysonance/Temporal.jl/issues/8 Isn't it?

davidanthoff commented 7 years ago

Yes, once I merge davidanthoff/IterableTables.jl#33, which right now is waiting for JuliaDB.jl to be registered in METADATA.jl. I'm not entirely sure whether that will be the most efficient implementation, though, I just don't understand JuliaDB well enough at this point.

femtotrader commented 7 years ago

Depends on https://github.com/JuliaComputing/JuliaDB.jl/issues/45

davidanthoff commented 7 years ago

Just to be clear: the integration of Temporal with IterableTables is done and released to METADATA, just the integration of JuliaDB with IterableTables is not yet merged.

femtotrader commented 7 years ago

Yes @davidanthoff I'm aware of this. Sorry my link post wasn't very clear... It's just a link to urge guys from @JuliaComputing to release to METADATA JuliaDB as it will also help JuliaDB integration of Temporal through IterableTables.

dysonance commented 5 years ago

It seems like support for the interface between JuliaDB and Temporal has been added since this was opened. Closing this now. Thanks for the help with this guys, didn't even realize it was done until today!