Open INF800 opened 3 years ago
Concentrate on DataCleaner
first. Expected but not resticted to the following API
# `etl` stands for extract, transform, load
from flowt.etl import DataCleaner
cleaner = DataCleaner(remove_missing=True)
...
ret, data_sample = cleaner(scraped_data_sample)
if ret:
realtime_values.push(data_sample)
preds = predictor(realtime_values, pred_config)
scraped_data_sample
is an element of data
coming out of scraper. How to use scraper is available in this test https://github.com/government-ai/flowt/blob/cb91b367bcdcdbfc54d481adb48064e724a4d9b6/tests/test_scraper/test_static_page_scraper.py#L30-L34
Need to make realtime predictions using data coming out of
StaticPageScraper