flow-php / flow

Flow PHP - data processing framework
https://flow-php.com
MIT License
404 stars 23 forks source link

Temporarily drop support for Avro file format #1101

Closed norberttech closed 5 days ago

norberttech commented 5 days ago

Change Log

Added

Fixed

Changed

Removed

  • Temporarily drop support for Avro file format

Deprecated

Security


Description

Avro file format will come back to Flow PHP at some point, however, due to the large amount of items in backlog we can't provide any ETA at this point

github-actions[bot] commented 5 days ago

Flow PHP - Benchmarks

Results of the benchmarks from this PR are compared with the results from 1.x branch.

Extractors ```shell +-----------------------+-------------------+------+-----+------------------+------------------+-----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-----------------------+-------------------+------+-----+------------------+------------------+-----------------+ | CSVExtractorBench | bench_extract_10k | 1 | 3 | 5.175mb -0.08% | 336.896ms +0.27% | ±0.53% +278.03% | | JsonExtractorBench | bench_extract_10k | 1 | 3 | 5.208mb -0.08% | 1.105s +3.47% | ±2.13% -6.23% | | ParquetExtractorBench | bench_extract_10k | 1 | 3 | 135.881mb -0.00% | 748.744ms -0.67% | ±0.77% -55.43% | | TextExtractorBench | bench_extract_10k | 1 | 3 | 4.965mb -0.08% | 36.397ms +1.46% | ±1.69% +163.48% | | XmlExtractorBench | bench_extract_10k | 1 | 3 | 4.971mb -0.08% | 436.817ms +0.81% | ±0.48% -25.29% | +-----------------------+-------------------+------+-----+------------------+------------------+-----------------+ ```
Transformers ```shell +-----------------------------+--------------------------+------+-----+------------------+-----------------+----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-----------------------------+--------------------------+------+-----+------------------+-----------------+----------------+ | RenameEntryTransformerBench | bench_transform_10k_rows | 1 | 3 | 116.272mb -0.00% | 58.840ms -1.09% | ±0.60% -54.99% | +-----------------------------+--------------------------+------+-----+------------------+-----------------+----------------+ ```
Loaders ```shell +--------------------+----------------+------+-----+------------------+-----------------+-----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +--------------------+----------------+------+-----+------------------+-----------------+-----------------+ | CSVLoaderBench | bench_load_10k | 1 | 3 | 55.254mb -0.01% | 69.492ms +2.38% | ±1.23% +287.54% | | JsonLoaderBench | bench_load_10k | 1 | 3 | 107.628mb -0.00% | 50.398ms -1.84% | ±0.27% -56.69% | | ParquetLoaderBench | bench_load_10k | 1 | 3 | 227.051mb -0.00% | 1.405s -0.06% | ±0.83% +236.39% | | TextLoaderBench | bench_load_10k | 1 | 3 | 18.009mb -0.02% | 38.311ms -1.46% | ±0.45% +66.32% | +--------------------+----------------+------+-----+------------------+-----------------+-----------------+ ```
Building Blocks ```shell +-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+ | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 116.821mb -0.00% | 492.932ms -2.87% | ±1.49% -40.00% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 60.299mb -0.01% | 262.868ms +4.30% | ±3.72% +211.86% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 15.233mb -0.03% | 52.504ms -3.26% | ±1.67% -8.91% | | TypeDetectorBench | bench_type_detector | 1 | 3 | 60.000mb -0.01% | 437.561ms +1.25% | ±1.37% +103.30% | | TypeDetectorBench | bench_type_detector | 1 | 3 | 14.539mb -0.03% | 86.139ms -0.74% | ±3.15% +743.06% | | RowsBench | bench_chunk_10_on_10k | 2 | 3 | 87.091mb -0.00% | 3.344ms -7.53% | ±1.28% -56.72% | | RowsBench | bench_diff_left_1k_on_10k | 2 | 3 | 102.689mb -0.00% | 185.760ms -0.79% | ±0.67% -17.75% | | RowsBench | bench_diff_right_1k_on_10k | 2 | 3 | 85.409mb -0.00% | 18.485ms -2.46% | ±0.37% -63.93% | | RowsBench | bench_drop_1k_on_10k | 2 | 3 | 88.331mb -0.00% | 1.693ms -10.51% | ±2.66% -1.60% | | RowsBench | bench_drop_right_1k_on_10k | 2 | 3 | 88.331mb -0.00% | 1.689ms -15.57% | ±3.19% +36.18% | | RowsBench | bench_entries_on_10k | 2 | 3 | 85.443mb -0.00% | 2.713ms -11.53% | ±2.57% -0.14% | | RowsBench | bench_filter_on_10k | 2 | 3 | 85.972mb -0.00% | 17.614ms +7.55% | ±1.27% -18.40% | | RowsBench | bench_find_on_10k | 2 | 3 | 85.972mb -0.00% | 17.592ms +2.63% | ±1.94% +64.07% | | RowsBench | bench_find_one_on_10k | 10 | 3 | 83.876mb -0.00% | 1.506μs -16.05% | ±3.07% +15.22% | | RowsBench | bench_first_on_10k | 10 | 3 | 83.876mb -0.00% | 0.300μs 0.00% | ±0.00% 0.00% | | RowsBench | bench_flat_map_on_1k | 2 | 3 | 93.226mb -0.00% | 12.186ms -1.13% | ±0.20% -63.24% | | RowsBench | bench_map_on_10k | 2 | 3 | 122.597mb -0.00% | 60.588ms -1.85% | ±0.56% +7.09% | | RowsBench | bench_merge_1k_on_10k | 2 | 3 | 86.492mb -0.00% | 1.213ms -23.22% | ±0.50% -85.61% | | RowsBench | bench_partition_by_on_10k | 2 | 3 | 89.838mb -0.00% | 61.489ms -4.93% | ±3.04% +263.31% | | RowsBench | bench_remove_on_10k | 2 | 3 | 88.593mb -0.00% | 4.024ms -8.34% | ±0.25% -86.71% | | RowsBench | bench_sort_asc_on_1k | 2 | 3 | 84.019mb -0.00% | 39.143ms -2.27% | ±0.30% -63.60% | | RowsBench | bench_sort_by_on_1k | 2 | 3 | 84.020mb -0.00% | 38.966ms -3.37% | ±0.23% -56.75% | | RowsBench | bench_sort_desc_on_1k | 2 | 3 | 84.019mb -0.00% | 39.168ms -2.26% | ±0.90% -68.97% | | RowsBench | bench_sort_entries_on_1k | 2 | 3 | 86.317mb -0.00% | 7.303ms -1.37% | ±0.21% -65.72% | | RowsBench | bench_sort_on_1k | 2 | 3 | 83.876mb -0.00% | 28.545ms -1.70% | ±0.56% -57.59% | | RowsBench | bench_take_1k_on_10k | 10 | 3 | 83.876mb -0.00% | 13.466μs -0.94% | ±3.15% +807.60% | | RowsBench | bench_take_right_1k_on_10k | 10 | 3 | 83.876mb -0.00% | 15.483μs -5.66% | ±1.38% +141.86% | | RowsBench | bench_unique_on_1k | 2 | 3 | 102.690mb -0.00% | 194.853ms +1.22% | ±1.90% +5.61% | +-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+ ```