Issue description
When I tested local stats with parquet data as raw data, it failed because it was not supported. So I have added this part of the function.
In StatsModelProcessor, it will call AkkaStatsWorker to do stats locally, which will call ShifuFileUtils.getDataScanners to get java Scanner from user input raw data. However it doesn't support the parquet format as input, so it broke the local testing.
How to support load parquet data into the scanner:
Read Parquet Group from parquet file.
Convert the parquet group to String in CSV format, columns separator with Shifu output data delimiter.
Convert the string to ByteArraryInputStream for the input stream. The above 3 steps are processed during the stream reading, so will not cache much data in memory.
Tested with stats in ShifuCLITest, it can run successfully.
Issue description When I tested local stats with parquet data as raw data, it failed because it was not supported. So I have added this part of the function.
In StatsModelProcessor, it will call AkkaStatsWorker to do stats locally, which will call ShifuFileUtils.getDataScanners to get java Scanner from user input raw data. However it doesn't support the parquet format as input, so it broke the local testing.
How to support load parquet data into the scanner:
The above 3 steps are processed during the stream reading, so will not cache much data in memory.
Tested with stats in ShifuCLITest, it can run successfully.