flow-php / flow

Flow PHP - data processing framework
https://flow-php.com
MIT License
404 stars 23 forks source link

Respect additional/missing schema fields when creating rows #988

Closed norberttech closed 4 months ago

norberttech commented 4 months ago

Change Log

Added

  • Respect additional/missing schema fields when creating rows

Fixed

Changed

Removed

Deprecated

Security


Description

github-actions[bot] commented 4 months ago

Flow PHP - Benchmarks

Results of the benchmarks from this PR are compared with the results from 1.x branch.

Extractors ```shell +-----------------------+-------------------+------+-----+------------------+------------------+-----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-----------------------+-------------------+------+-----+------------------+------------------+-----------------+ | AvroExtractorBench | bench_extract_10k | 1 | 3 | 35.264mb +0.01% | 825.903ms +0.26% | ±0.68% -24.78% | | CSVExtractorBench | bench_extract_10k | 1 | 3 | 4.984mb +0.10% | 342.567ms +0.42% | ±0.49% +1.98% | | JsonExtractorBench | bench_extract_10k | 1 | 3 | 5.134mb +0.10% | 1.049s -0.18% | ±0.82% +90.78% | | ParquetExtractorBench | bench_extract_10k | 1 | 3 | 135.789mb +0.00% | 906.073ms +1.62% | ±1.08% +572.08% | | TextExtractorBench | bench_extract_10k | 1 | 3 | 4.895mb +0.10% | 35.074ms -0.24% | ±0.56% -30.14% | | XmlExtractorBench | bench_extract_10k | 1 | 3 | 4.897mb +0.10% | 437.087ms +1.05% | ±2.87% +26.97% | +-----------------------+-------------------+------+-----+------------------+------------------+-----------------+ ```
Transformers ```shell +-----------------------------+--------------------------+------+-----+------------------+-----------------+----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-----------------------------+--------------------------+------+-----+------------------+-----------------+----------------+ | RenameEntryTransformerBench | bench_transform_10k_rows | 1 | 3 | 110.604mb +0.00% | 65.317ms -0.65% | ±0.52% -29.66% | +-----------------------------+--------------------------+------+-----+------------------+-----------------+----------------+ ```
Loaders ```shell +--------------------+----------------+------+-----+------------------+------------------+----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +--------------------+----------------+------+-----+------------------+------------------+----------------+ | AvroLoaderBench | bench_load_10k | 1 | 3 | 95.645mb +0.01% | 455.903ms -1.10% | ±1.04% +0.15% | | CSVLoaderBench | bench_load_10k | 1 | 3 | 54.120mb +0.01% | 73.181ms -0.18% | ±0.30% -79.50% | | JsonLoaderBench | bench_load_10k | 1 | 3 | 106.549mb +0.00% | 54.161ms +1.02% | ±0.79% -76.78% | | ParquetLoaderBench | bench_load_10k | 1 | 3 | 224.357mb +0.00% | 1.433s -0.89% | ±0.59% -12.36% | | TextLoaderBench | bench_load_10k | 1 | 3 | 17.943mb +0.03% | 40.962ms -0.40% | ±0.45% -2.49% | +--------------------+----------------+------+-----+------------------+------------------+----------------+ ```
Building Blocks ```shell +-------------------------+----------------------------+------+-----+------------------+------------------+-------------------------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-------------------------+----------------------------+------+-----+------------------+------------------+-------------------------------+ | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 116.710mb +0.00% | 491.728ms -2.06% | ±0.59% +70.74% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 60.188mb +0.01% | 248.846ms -0.58% | ±0.14% -90.05% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 15.123mb +0.03% | 54.067ms +0.75% | ±1.45% +52.06% | | TypeDetectorBench | bench_type_detector | 1 | 3 | 59.954mb +0.01% | 439.363ms +0.78% | ±0.05% -18.02% | | TypeDetectorBench | bench_type_detector | 1 | 3 | 14.493mb +0.03% | 89.911ms +2.98% | ±1.70% +122.47% | | RowsBench | bench_chunk_10_on_10k | 2 | 3 | 76.678mb +0.01% | 3.825ms -0.70% | ±0.99% -69.04% | | RowsBench | bench_diff_left_1k_on_10k | 2 | 3 | 96.405mb +0.01% | 182.652ms +2.30% | ±1.13% +102.99% | | RowsBench | bench_diff_right_1k_on_10k | 2 | 3 | 74.931mb +0.01% | 18.725ms +3.33% | ±1.26% +136.05% | | RowsBench | bench_drop_1k_on_10k | 2 | 3 | 77.918mb +0.01% | 1.823ms -3.75% | ±2.06% +330.48% | | RowsBench | bench_drop_right_1k_on_10k | 2 | 3 | 77.918mb +0.01% | 1.958ms +0.36% | ±3.34% +164.79% | | RowsBench | bench_entries_on_10k | 2 | 3 | 75.030mb +0.01% | 2.882ms +0.85% | ±3.20% +132.38% | | RowsBench | bench_filter_on_10k | 2 | 3 | 75.559mb +0.01% | 14.794ms +0.04% | ±1.78% +102.18% | | RowsBench | bench_find_on_10k | 2 | 3 | 75.559mb +0.01% | 14.941ms -0.53% | ±3.61% +214.23% | | RowsBench | bench_find_one_on_10k | 10 | 3 | 73.464mb +0.01% | 1.906μs +0.64% | ±2.44% -3.45% | | RowsBench | bench_first_on_10k | 10 | 3 | 73.464mb +0.01% | 0.300μs -25.00% | ±0.00% -100.00% | | RowsBench | bench_flat_map_on_1k | 2 | 3 | 87.018mb +0.01% | 13.912ms +1.63% | ±2.42% -13.26% | | RowsBench | bench_map_on_10k | 2 | 3 | 116.379mb +0.00% | 64.284ms -5.77% | ±0.21% -70.95% | | RowsBench | bench_merge_1k_on_10k | 2 | 3 | 76.079mb +0.01% | 1.573ms +5.52% | ±1.87% -34.07% | | RowsBench | bench_partition_by_on_10k | 2 | 3 | 79.426mb +0.01% | 59.091ms +1.95% | ±0.75% +140.14% | | RowsBench | bench_remove_on_10k | 2 | 3 | 78.181mb +0.01% | 4.122ms -1.29% | ±2.26% -27.34% | | RowsBench | bench_sort_asc_on_1k | 2 | 3 | 73.542mb +0.01% | 40.495ms -0.15% | ±0.46% -80.82% | | RowsBench | bench_sort_by_on_1k | 2 | 3 | 73.542mb +0.01% | 41.305ms +1.50% | ±0.65% -26.10% | | RowsBench | bench_sort_desc_on_1k | 2 | 3 | 73.542mb +0.01% | 40.554ms +1.41% | ±0.76% -61.38% | | RowsBench | bench_sort_entries_on_1k | 2 | 3 | 75.905mb +0.01% | 7.684ms +4.00% | ±3.40% +1.63% | | RowsBench | bench_sort_on_1k | 2 | 3 | 73.464mb +0.01% | 29.211ms -1.17% | ±0.90% +104.14% | | RowsBench | bench_take_1k_on_10k | 10 | 3 | 73.464mb +0.01% | 13.930μs +1.68% | ±1.68% +12922986954029000.00% | | RowsBench | bench_take_right_1k_on_10k | 10 | 3 | 73.464mb +0.01% | 16.317μs -0.51% | ±1.33% +0.00% | | RowsBench | bench_unique_on_1k | 2 | 3 | 96.472mb +0.01% | 188.180ms +0.97% | ±0.28% -54.87% | +-------------------------+----------------------------+------+-----+------------------+------------------+-------------------------------+ ```