This PR brings speed up in downloading the SSOFT in parquet format
output format
download time
csv
21.28 seconds
json
16.61 seconds
parquet (old)
5.85 seconds
parquet (new)
4.84 seconds
json and csv are slow because the data needs to be decompressed, and formatted in pandas DataFrame before being sent to the user (that is: DO NOT USE THAT unless you really needs it). The parquet mode is now faster because I get rid of the pandas DataFrame step.
This PR brings speed up in downloading the SSOFT in parquet format
jsonandcsvare slow because the data needs to be decompressed, and formatted in pandas DataFrame before being sent to the user (that is: DO NOT USE THAT unless you really needs it). The parquet mode is now faster because I get rid of the pandas DataFrame step.