flow-php / flow

Flow PHP - data processing framework
https://flow-php.com
MIT License
404 stars 23 forks source link

Fixed edge case where evolving schema is brings new definition but it's missing some old ones #1028

Closed norberttech closed 3 months ago

norberttech commented 3 months ago

Change Log

Added

Fixed

  • Evolving schema matcher, when right side has the same number of definitions but different name

Changed

Removed

Deprecated

Security


Description

github-actions[bot] commented 3 months ago

Flow PHP - Benchmarks

Results of the benchmarks from this PR are compared with the results from 1.x branch.

Extractors ```shell +-----------------------+-------------------+------+-----+------------------+------------------+-----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-----------------------+-------------------+------+-----+------------------+------------------+-----------------+ | AvroExtractorBench | bench_extract_10k | 1 | 3 | 35.282mb +0.00% | 822.328ms +0.65% | ±0.97% +59.54% | | CSVExtractorBench | bench_extract_10k | 1 | 3 | 5.005mb +0.00% | 341.576ms -0.44% | ±0.70% +22.39% | | JsonExtractorBench | bench_extract_10k | 1 | 3 | 5.156mb +0.00% | 1.072s +3.21% | ±1.83% +1.12% | | ParquetExtractorBench | bench_extract_10k | 1 | 3 | 135.831mb +0.00% | 894.903ms -0.55% | ±0.63% +184.08% | | TextExtractorBench | bench_extract_10k | 1 | 3 | 4.913mb +0.00% | 35.425ms +0.09% | ±0.40% -51.48% | | XmlExtractorBench | bench_extract_10k | 1 | 3 | 4.919mb +0.00% | 434.652ms -3.50% | ±0.33% -83.40% | +-----------------------+-------------------+------+-----+------------------+------------------+-----------------+ ```
Transformers ```shell +-----------------------------+--------------------------+------+-----+------------------+-----------------+----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-----------------------------+--------------------------+------+-----+------------------+-----------------+----------------+ | RenameEntryTransformerBench | bench_transform_10k_rows | 1 | 3 | 110.619mb +0.00% | 64.209ms -0.71% | ±1.95% +89.43% | +-----------------------------+--------------------------+------+-----+------------------+-----------------+----------------+ ```
Loaders ```shell +--------------------+----------------+------+-----+------------------+------------------+----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +--------------------+----------------+------+-----+------------------+------------------+----------------+ | AvroLoaderBench | bench_load_10k | 1 | 3 | 95.663mb +0.00% | 474.964ms +1.26% | ±0.13% -53.92% | | CSVLoaderBench | bench_load_10k | 1 | 3 | 54.145mb +0.00% | 70.435ms -2.61% | ±0.12% -82.40% | | JsonLoaderBench | bench_load_10k | 1 | 3 | 106.570mb +0.00% | 52.884ms -0.04% | ±1.29% +9.92% | | ParquetLoaderBench | bench_load_10k | 1 | 3 | 224.394mb +0.00% | 1.422s +0.38% | ±0.31% -25.50% | | TextLoaderBench | bench_load_10k | 1 | 3 | 17.960mb +0.00% | 40.221ms +0.08% | ±0.50% -33.85% | +--------------------+----------------+------+-----+------------------+------------------+----------------+ ```
Building Blocks ```shell +-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+ | RowsBench | bench_chunk_10_on_10k | 2 | 3 | 76.686mb +0.00% | 3.389ms -2.08% | ±1.36% -52.19% | | RowsBench | bench_diff_left_1k_on_10k | 2 | 3 | 96.413mb +0.00% | 184.117ms +2.09% | ±0.29% -64.82% | | RowsBench | bench_diff_right_1k_on_10k | 2 | 3 | 74.938mb +0.00% | 18.408ms +1.01% | ±2.14% +298.01% | | RowsBench | bench_drop_1k_on_10k | 2 | 3 | 77.926mb +0.00% | 1.776ms +6.13% | ±2.71% +178.19% | | RowsBench | bench_drop_right_1k_on_10k | 2 | 3 | 77.926mb +0.00% | 1.729ms +3.58% | ±1.06% -19.91% | | RowsBench | bench_entries_on_10k | 2 | 3 | 75.038mb +0.00% | 2.604ms +1.15% | ±2.55% +139.26% | | RowsBench | bench_filter_on_10k | 2 | 3 | 75.567mb +0.00% | 16.987ms +17.09% | ±2.39% +17.09% | | RowsBench | bench_find_on_10k | 2 | 3 | 75.567mb +0.00% | 17.820ms +21.71% | ±0.68% -70.59% | | RowsBench | bench_find_one_on_10k | 10 | 3 | 73.471mb +0.00% | 1.694μs -0.71% | ±2.83% +4.00% | | RowsBench | bench_first_on_10k | 10 | 3 | 73.471mb +0.00% | 0.300μs -25.00% | ±0.00% -100.00% | | RowsBench | bench_flat_map_on_1k | 2 | 3 | 87.025mb +0.00% | 13.096ms +2.17% | ±0.31% -83.31% | | RowsBench | bench_map_on_10k | 2 | 3 | 116.386mb +0.00% | 65.649ms +0.05% | ±1.17% +496.98% | | RowsBench | bench_merge_1k_on_10k | 2 | 3 | 76.087mb +0.00% | 1.343ms +14.45% | ±3.42% +33.49% | | RowsBench | bench_partition_by_on_10k | 2 | 3 | 79.433mb +0.00% | 60.599ms +4.29% | ±3.66% +297.07% | | RowsBench | bench_remove_on_10k | 2 | 3 | 78.188mb +0.00% | 3.940ms +2.38% | ±1.17% -14.57% | | RowsBench | bench_sort_asc_on_1k | 2 | 3 | 73.549mb +0.00% | 41.831ms +3.48% | ±2.22% +221.12% | | RowsBench | bench_sort_by_on_1k | 2 | 3 | 73.550mb +0.00% | 42.312ms +4.40% | ±0.86% +872.54% | | RowsBench | bench_sort_desc_on_1k | 2 | 3 | 73.549mb +0.00% | 42.453ms +2.43% | ±0.33% -60.00% | | RowsBench | bench_sort_entries_on_1k | 2 | 3 | 75.912mb +0.00% | 7.406ms +0.26% | ±1.76% +84.64% | | RowsBench | bench_sort_on_1k | 2 | 3 | 73.471mb +0.00% | 28.844ms +0.61% | ±0.78% -32.82% | | RowsBench | bench_take_1k_on_10k | 10 | 3 | 73.471mb +0.00% | 13.521μs +1.16% | ±0.93% -39.45% | | RowsBench | bench_take_right_1k_on_10k | 10 | 3 | 73.471mb +0.00% | 16.021μs +0.76% | ±0.78% +0.00% | | RowsBench | bench_unique_on_1k | 2 | 3 | 96.479mb +0.00% | 188.897ms +0.88% | ±1.62% +193.80% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 116.718mb +0.00% | 495.506ms -0.52% | ±1.14% +227.24% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 60.196mb +0.00% | 247.166ms +0.38% | ±0.99% +171.59% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 15.130mb +0.00% | 52.406ms +0.09% | ±0.39% +452.18% | | TypeDetectorBench | bench_type_detector | 1 | 3 | 59.961mb +0.00% | 433.461ms +0.49% | ±0.41% +61.27% | | TypeDetectorBench | bench_type_detector | 1 | 3 | 14.500mb +0.00% | 84.371ms -0.50% | ±0.29% +159.87% | +-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+ ```