operation: data: Add/Update operations to handle time series data

intel / dffml

The easiest way to use Machine Learning. Mix and match underlying ML libraries and data set sources. Generate new datasets or modify existing ones with ease.

https://intel.github.io/dffml/main/

MIT License

253 stars 138 forks source link

operation: data: Add/Update operations to handle time series data #1321

Open programmer290399 opened 2 years ago

programmer290399 commented 2 years ago

Pain Point

We do not have operations to work with time-series data

Proposed Solution

Implement/Update operations for the following tasks:

[ ] NaN Removal
[ ] Outlier Removal
[ ] Missing Timestamp Handling
[ ] Scaling
[ ] Imputation & Interpolation
[ ] Shifting/lag generation
[ ] Time Series Decomposition

Note: Some Topics may overlap with others in the list above, we need to make our operations as generic as possible. So that they can work with not only time-series data but other data formats as well if possible.

TirelessClock commented 2 years ago

Hey, @programmer290399 You mentioned earlier that you guys generally avoid using 3rd party libraries, does that mean we'll be writing the code for Timestamp Handling on our own or can we use the datetime library?

johnandersen777 commented 2 years ago

Per the docs: https://intel.github.io/dffml/contributing/git.html#issue-and-pull-request-title-formatting

operation(s): data: means this will live in dffml.git/operations/data which means you can use anything you want to get the job done.

If it's in dffml.git/dffml then we stick to Python STDLIB

TirelessClock commented 2 years ago

Per the docs: https://intel.github.io/dffml/contributing/git.html#issue-and-pull-request-title-formatting

operation(s): data: means this will live in dffml.git/operations/data which means you can use anything you want to get the job done.

If it's in dffml.git/dffml then we stick to Python STDLIB

@pdxjohnny okay great!