Conduct experiments to compare storage costs and compute costs

[heyodai]

Description: Conduct experiments to compare the storage costs and compute costs of FPHLM using the current data storage format (CSV) and the best-performing alternative data storage format (Parquet), as identified in Ticket #7

Tasks:

• Design and execute experiments to compare storage costs and compute costs, including the following:
  • Develop a methodology for measuring and comparing storage and compute costs, based on the performance considerations identified in Ticket #5
  • Obtain or generate representative FPHLM data in both the current CSV format and the alternative Parquet format
  • Set up an environment for running the experiments, including any necessary hardware and software
  • Run the experiments using the methodology and representative data
  • Record the results of the experiments and create charts or visualizations to display them
• Include the charts and a description of the experiments in the project documentation, including the following:
  • A summary of the experimental methodology and setup
  • Charts or visualizations showing the storage and compute costs for both formats
  • Analysis and interpretation of the results, including any insights or conclusions about the performance of the different formats

[sbrunton1]

The paper 2020-AComparisonofHDFSFileFormatsAvroParquetandORC does an excellent job setting up mock data to answer the following requests. They then import this data to HDFS with Hive and perform multiple tests to answer those requests.

RQ1: Which file format consumes less storage space of HDFS? RQ2: Which data structure format (Avro, Parquet, or ORC) supports high performance with regards to aggregated/scanned queries? RQ3: Which data format (Avro or Parquet or ORC) is more compact?

Would it be worth looking at these test cases and approach to help us establish our methodology and how to measure success in relation to our datasets?

[heyodai]

Paper, for reference: 2020-AComparisonofHDFSFileFormatsAvroParquetandORC-annotated.pdf

[heyodai]

@sbrunton1 - I've read the paper and I agree with you. These are good questions and the paper provides compelling evidence that ORC is the better choice in most cases.

I will say that I think RQ1 and RQ3 are basically the same question, unless I'm missing something.