Here are some results running locally with the Google Cloud Storage internal storage implementation.
The test read an ION file and wrote a file in the new format (scenario 'write'), then read the new format and write an ION fle (scenario read). Doing this for two kind of files:
small: 1 million rows with 4 columns
big: 100 thousand rows with approx. 150 columns
Excel and XMLS use fewer rows as they load the data in memory (small 100K and big 10K rows).
This PR use the new methods on the FileSerde to improve the performance of the read/write of files.
Fixes #102
Here are some results running locally with the Google Cloud Storage internal storage implementation. The test read an ION file and wrote a file in the new format (scenario 'write'), then read the new format and write an ION fle (scenario read). Doing this for two kind of files:
Excel and XMLS use fewer rows as they load the data in memory (small 100K and big 10K rows).