gov-ai / flowt

Forex and stock market predictions using transformer
32 stars 10 forks source link

Realtime prediction pipeline #9

Open INF800 opened 3 years ago

INF800 commented 3 years ago

Need to make realtime predictions using data coming out of StaticPageScraper

INF800 commented 3 years ago

Concentrate on DataCleaner first. Expected but not resticted to the following API

# `etl` stands for extract, transform, load
from flowt.etl import DataCleaner

cleaner = DataCleaner(remove_missing=True)

...

ret, data_sample = cleaner(scraped_data_sample)

if ret:
    realtime_values.push(data_sample)
    preds = predictor(realtime_values, pred_config)
INF800 commented 3 years ago

scraped_data_sample is an element of data coming out of scraper. How to use scraper is available in this test https://github.com/government-ai/flowt/blob/cb91b367bcdcdbfc54d481adb48064e724a4d9b6/tests/test_scraper/test_static_page_scraper.py#L30-L34

INF800 commented 3 years ago

Prototype code https://colab.research.google.com/drive/1TidxRMaU2E7LiVf8LvSCGUaYOoYAc_tn?usp=sharing