flow-php / flow

Flow PHP - data processing framework
https://flow-php.com
MIT License
483 stars 28 forks source link

Pipelines refactoring #1056

Closed norberttech closed 6 months ago

norberttech commented 6 months ago

Change Log

Added

Fixed

Changed

  • GroupBy is now allowed only with aggregations
  • NestedPipeline was renamed to LinkedPipeline
  • All pipelines are now grouped by LinkedPipeline

Removed

  • Pipeline::closure
  • Pipeline::cleanCopy

Deprecated

Security


Description

github-actions[bot] commented 6 months ago

Flow PHP - Benchmarks

Results of the benchmarks from this PR are compared with the results from 1.x branch.

Extractors ```shell +-----------------------+-------------------+------+-----+------------------+------------------+-----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-----------------------+-------------------+------+-----+------------------+------------------+-----------------+ | AvroExtractorBench | bench_extract_10k | 1 | 3 | 35.290mb +0.00% | 857.570ms +1.77% | ±0.27% -64.70% | | CSVExtractorBench | bench_extract_10k | 1 | 3 | 5.010mb +0.00% | 347.386ms +1.03% | ±1.80% +740.46% | | JsonExtractorBench | bench_extract_10k | 1 | 3 | 5.165mb +0.00% | 1.078s +0.32% | ±0.32% +5.02% | | ParquetExtractorBench | bench_extract_10k | 1 | 3 | 135.837mb +0.00% | 932.028ms +2.09% | ±0.42% +878.64% | | TextExtractorBench | bench_extract_10k | 1 | 3 | 4.925mb +0.00% | 35.676ms +0.31% | ±0.11% -37.52% | | XmlExtractorBench | bench_extract_10k | 1 | 3 | 4.931mb +0.00% | 436.621ms +0.73% | ±0.46% -74.80% | +-----------------------+-------------------+------+-----+------------------+------------------+-----------------+ ```
Transformers ```shell +-----------------------------+--------------------------+------+-----+------------------+-----------------+-----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-----------------------------+--------------------------+------+-----+------------------+-----------------+-----------------+ | RenameEntryTransformerBench | bench_transform_10k_rows | 1 | 3 | 116.231mb +0.00% | 61.547ms +3.50% | ±1.28% +161.40% | +-----------------------------+--------------------------+------+-----+------------------+-----------------+-----------------+ ```
Loaders ```shell +--------------------+----------------+------+-----+------------------+------------------+-----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +--------------------+----------------+------+-----+------------------+------------------+-----------------+ | AvroLoaderBench | bench_load_10k | 1 | 3 | 96.676mb +0.00% | 462.814ms -0.36% | ±0.39% -36.76% | | CSVLoaderBench | bench_load_10k | 1 | 3 | 55.151mb +0.00% | 68.905ms +0.51% | ±0.72% -26.00% | | JsonLoaderBench | bench_load_10k | 1 | 3 | 107.585mb +0.00% | 52.758ms +4.67% | ±0.01% -97.86% | | ParquetLoaderBench | bench_load_10k | 1 | 3 | 227.005mb +0.00% | 1.455s +2.31% | ±0.43% -52.11% | | TextLoaderBench | bench_load_10k | 1 | 3 | 17.967mb +0.00% | 39.792ms +5.50% | ±0.55% +346.24% | +--------------------+----------------+------+-----+------------------+------------------+-----------------+ ```
Building Blocks ```shell +-------------------------+----------------------------+------+-----+------------------+------------------+------------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-------------------------+----------------------------+------+-----+------------------+------------------+------------------+ | RowsBench | bench_chunk_10_on_10k | 2 | 3 | 87.051mb -0.00% | 4.105ms +23.30% | ±2.14% +43.06% | | RowsBench | bench_diff_left_1k_on_10k | 2 | 3 | 102.650mb -0.00% | 195.433ms +4.41% | ±0.27% -43.21% | | RowsBench | bench_diff_right_1k_on_10k | 2 | 3 | 85.370mb -0.00% | 19.798ms +4.93% | ±0.60% -39.96% | | RowsBench | bench_drop_1k_on_10k | 2 | 3 | 88.291mb -0.00% | 1.943ms +15.38% | ±1.64% -47.75% | | RowsBench | bench_drop_right_1k_on_10k | 2 | 3 | 88.291mb -0.00% | 2.144ms +29.01% | ±2.45% +156.02% | | RowsBench | bench_entries_on_10k | 2 | 3 | 85.404mb -0.00% | 3.024ms +19.49% | ±2.22% +29.24% | | RowsBench | bench_filter_on_10k | 2 | 3 | 85.933mb -0.00% | 17.468ms +7.61% | ±1.32% +25.18% | | RowsBench | bench_find_on_10k | 2 | 3 | 85.933mb -0.00% | 17.502ms +8.21% | ±2.20% +182.21% | | RowsBench | bench_find_one_on_10k | 10 | 3 | 83.837mb -0.00% | 2.006μs +24.91% | ±2.32% -19.67% | | RowsBench | bench_first_on_10k | 10 | 3 | 83.837mb -0.00% | 0.400μs 0.00% | ±0.00% 0.00% | | RowsBench | bench_flat_map_on_1k | 2 | 3 | 93.187mb -0.00% | 14.490ms +14.15% | ±2.26% +53.45% | | RowsBench | bench_map_on_10k | 2 | 3 | 122.557mb -0.00% | 63.404ms +4.45% | ±1.39% +54.15% | | RowsBench | bench_merge_1k_on_10k | 2 | 3 | 86.452mb -0.00% | 1.744ms +45.60% | ±2.21% +216.02% | | RowsBench | bench_partition_by_on_10k | 2 | 3 | 89.799mb -0.00% | 67.973ms +10.25% | ±0.50% -76.52% | | RowsBench | bench_remove_on_10k | 2 | 3 | 88.554mb -0.00% | 4.204ms +9.77% | ±0.61% -67.89% | | RowsBench | bench_sort_asc_on_1k | 2 | 3 | 83.915mb -0.00% | 42.570ms +5.49% | ±0.77% +969.04% | | RowsBench | bench_sort_by_on_1k | 2 | 3 | 83.915mb -0.00% | 42.758ms +5.66% | ±1.65% +278.17% | | RowsBench | bench_sort_desc_on_1k | 2 | 3 | 83.915mb -0.00% | 42.612ms +4.07% | ±0.96% +76.46% | | RowsBench | bench_sort_entries_on_1k | 2 | 3 | 86.278mb -0.00% | 7.945ms +7.50% | ±0.75% -69.72% | | RowsBench | bench_sort_on_1k | 2 | 3 | 83.837mb -0.00% | 29.959ms +3.68% | ±0.72% +1484.72% | | RowsBench | bench_take_1k_on_10k | 10 | 3 | 83.837mb -0.00% | 14.100μs +7.81% | ±0.58% -39.02% | | RowsBench | bench_take_right_1k_on_10k | 10 | 3 | 83.837mb -0.00% | 16.930μs +7.83% | ±1.38% +32.78% | | RowsBench | bench_unique_on_1k | 2 | 3 | 102.651mb -0.00% | 202.150ms +4.88% | ±0.79% -41.30% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 116.729mb -0.00% | 528.323ms +1.66% | ±1.06% +997.89% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 60.207mb -0.00% | 263.052ms +1.34% | ±0.62% -17.59% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 15.141mb -0.00% | 58.218ms +7.69% | ±2.05% +43.32% | | TypeDetectorBench | bench_type_detector | 1 | 3 | 59.966mb -0.00% | 446.024ms +4.00% | ±0.71% -38.67% | | TypeDetectorBench | bench_type_detector | 1 | 3 | 14.505mb -0.00% | 89.809ms +5.50% | ±0.72% -0.10% | +-------------------------+----------------------------+------+-----+------------------+------------------+------------------+ ```