TDL-12486: Added support of compressed files

[dbshah1212]

Description of change

Compressed File Support

• Nested Compression (ZIP, GZ) is not supported

• file having extension .tar.gz not supported

• Throwing error if file has error in gz file

  • Possible Error(s):
  • Magic 2 bytes are empty
  • Not a gzipped file
  • Unknown compression method
  • The name is not in the header according to the flag

• Max 5 files are taken for sampling (including inner files of ZIP)

  E.g.

  1. sample_compressed_file.zip (has 5 files inside it) | sample_csv_file_1.csv |__ sample_csv_file_2.csv | sample_exe_file_1.exe |__ sample_exe_file_2.exe |__ sample_gz_file.gz
  2. sample_csv_file_3.csv
  3. sample_jsonl_file.jsonl
  4. sample_csv_file_4.csv

  • In Above scenario the first 5 supporting files are taken

    • New List of Files:

      1. sample_csv_file_1.csv
      2. sample_csv_file_2.csv
      3. sample_gz_file.gz
      4. sample_csv_file_3.csv
      5. sample_jsonl_file.jsonl
      6. sample_csv_file_4.csv
      • Taken 5 first files for sampling
      1. sample_csv_file_1.csv
      2. sample_csv_file_2.csv
      3. sample_gz_file.gz
      4. sample_csv_file_3.csv
      5. sample_jsonl_file.jsonl

Manual QA steps

• Testes all above-mentioned scenarios and ran all integration and unit tests.

Risks

Rollback steps

• revert this branch