flow-php / flow

Flow PHP - data processing framework
https://flow-php.com
MIT License
470 stars 26 forks source link

Fixed merging schemas with different entries #1023

Closed norberttech closed 6 months ago

norberttech commented 6 months ago

Change Log

Added

Fixed

  • merging schemas with different entries

Changed

Removed

Deprecated

Security


Description

Before, when we merged schema with new columns into schema with all non nullable columns, output schema was still not nullable on the left side but nullable on the right side. Now when we are merging schema and element is not exactly the same on the other side it will become nullable even if it wasn't nullable before.

github-actions[bot] commented 6 months ago

Flow PHP - Benchmarks

Results of the benchmarks from this PR are compared with the results from 1.x branch.

Extractors ```shell +-----------------------+-------------------+------+-----+------------------+------------------+-----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-----------------------+-------------------+------+-----+------------------+------------------+-----------------+ | AvroExtractorBench | bench_extract_10k | 1 | 3 | 35.279mb +0.00% | 816.588ms +0.60% | ±1.04% +175.76% | | CSVExtractorBench | bench_extract_10k | 1 | 3 | 5.003mb +0.01% | 340.739ms +0.08% | ±0.30% -28.40% | | JsonExtractorBench | bench_extract_10k | 1 | 3 | 5.153mb +0.01% | 1.053s -1.56% | ±2.35% +246.87% | | ParquetExtractorBench | bench_extract_10k | 1 | 3 | 135.824mb +0.00% | 902.476ms +0.23% | ±0.46% -25.55% | | TextExtractorBench | bench_extract_10k | 1 | 3 | 4.910mb +0.01% | 35.222ms +0.56% | ±0.33% -59.62% | | XmlExtractorBench | bench_extract_10k | 1 | 3 | 4.916mb +0.01% | 433.496ms -1.02% | ±0.67% -74.84% | +-----------------------+-------------------+------+-----+------------------+------------------+-----------------+ ```
Transformers ```shell +-----------------------------+--------------------------+------+-----+------------------+-----------------+-----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-----------------------------+--------------------------+------+-----+------------------+-----------------+-----------------+ | RenameEntryTransformerBench | bench_transform_10k_rows | 1 | 3 | 110.616mb +0.00% | 64.086ms -1.11% | ±3.48% +379.55% | +-----------------------------+--------------------------+------+-----+------------------+-----------------+-----------------+ ```
Loaders ```shell +--------------------+----------------+------+-----+------------------+------------------+-----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +--------------------+----------------+------+-----+------------------+------------------+-----------------+ | AvroLoaderBench | bench_load_10k | 1 | 3 | 95.660mb +0.00% | 475.233ms +1.25% | ±0.75% -33.25% | | CSVLoaderBench | bench_load_10k | 1 | 3 | 54.142mb +0.00% | 71.729ms +1.83% | ±0.12% -94.75% | | JsonLoaderBench | bench_load_10k | 1 | 3 | 106.567mb +0.00% | 52.878ms +2.47% | ±1.47% +160.69% | | ParquetLoaderBench | bench_load_10k | 1 | 3 | 224.387mb +0.00% | 1.436s +1.59% | ±0.74% +227.47% | | TextLoaderBench | bench_load_10k | 1 | 3 | 17.957mb +0.00% | 39.803ms -0.74% | ±0.26% -77.16% | +--------------------+----------------+------+-----+------------------+------------------+-----------------+ ```
Building Blocks ```shell +-------------------------+----------------------------+------+-----+------------------+------------------+------------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-------------------------+----------------------------+------+-----+------------------+------------------+------------------+ | RowsBench | bench_chunk_10_on_10k | 2 | 3 | 76.683mb +0.00% | 3.483ms +2.84% | ±1.48% -16.74% | | RowsBench | bench_diff_left_1k_on_10k | 2 | 3 | 96.410mb +0.00% | 180.971ms +0.91% | ±0.06% -52.37% | | RowsBench | bench_diff_right_1k_on_10k | 2 | 3 | 74.936mb +0.00% | 18.507ms +2.44% | ±1.31% +291.34% | | RowsBench | bench_drop_1k_on_10k | 2 | 3 | 77.923mb +0.00% | 1.953ms +11.45% | ±3.58% +62.62% | | RowsBench | bench_drop_right_1k_on_10k | 2 | 3 | 77.923mb +0.00% | 1.959ms +0.56% | ±0.71% -79.54% | | RowsBench | bench_entries_on_10k | 2 | 3 | 75.035mb +0.00% | 2.773ms +6.68% | ±1.88% +46.87% | | RowsBench | bench_filter_on_10k | 2 | 3 | 75.564mb +0.00% | 15.082ms +4.49% | ±1.82% +78.46% | | RowsBench | bench_find_on_10k | 2 | 3 | 75.564mb +0.00% | 15.458ms +2.19% | ±1.88% +8.05% | | RowsBench | bench_find_one_on_10k | 10 | 3 | 73.468mb +0.00% | 1.906μs +6.25% | ±2.44% -8.62% | | RowsBench | bench_first_on_10k | 10 | 3 | 73.468mb +0.00% | 0.400μs 0.00% | ±0.00% 0.00% | | RowsBench | bench_flat_map_on_1k | 2 | 3 | 87.023mb +0.00% | 13.208ms +3.74% | ±0.59% -76.17% | | RowsBench | bench_map_on_10k | 2 | 3 | 116.383mb +0.00% | 67.157ms -0.80% | ±0.38% -72.30% | | RowsBench | bench_merge_1k_on_10k | 2 | 3 | 76.084mb +0.00% | 1.557ms +26.87% | ±3.35% +49.15% | | RowsBench | bench_partition_by_on_10k | 2 | 3 | 79.431mb +0.00% | 58.275ms +1.25% | ±1.46% +81.50% | | RowsBench | bench_remove_on_10k | 2 | 3 | 78.185mb +0.00% | 4.193ms +7.42% | ±2.09% +6809.31% | | RowsBench | bench_sort_asc_on_1k | 2 | 3 | 73.546mb +0.00% | 39.719ms -0.53% | ±0.88% +84.82% | | RowsBench | bench_sort_by_on_1k | 2 | 3 | 73.547mb +0.00% | 40.887ms +1.17% | ±1.16% +4.49% | | RowsBench | bench_sort_desc_on_1k | 2 | 3 | 73.546mb +0.00% | 40.285ms -1.80% | ±1.53% -33.12% | | RowsBench | bench_sort_entries_on_1k | 2 | 3 | 75.909mb +0.00% | 7.356ms -0.07% | ±0.57% +53.99% | | RowsBench | bench_sort_on_1k | 2 | 3 | 73.468mb +0.00% | 29.158ms -0.43% | ±0.94% +5.18% | | RowsBench | bench_take_1k_on_10k | 10 | 3 | 73.468mb +0.00% | 13.842μs +3.04% | ±2.15% +39.78% | | RowsBench | bench_take_right_1k_on_10k | 10 | 3 | 73.468mb +0.00% | 16.359μs +2.75% | ±1.51% +92.71% | | RowsBench | bench_unique_on_1k | 2 | 3 | 96.477mb +0.00% | 186.330ms +0.12% | ±3.44% +112.65% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 116.715mb +0.00% | 496.785ms -0.32% | ±0.84% -56.89% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 60.193mb +0.00% | 248.684ms +1.91% | ±0.92% +180.22% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 15.127mb +0.00% | 53.445ms +2.50% | ±0.89% +10.55% | | TypeDetectorBench | bench_type_detector | 1 | 3 | 59.958mb +0.00% | 436.298ms +0.47% | ±0.18% -36.76% | | TypeDetectorBench | bench_type_detector | 1 | 3 | 14.498mb +0.00% | 87.232ms +3.27% | ±0.70% -32.86% | +-------------------------+----------------------------+------+-----+------------------+------------------+------------------+ ```