search

flow-php / flow

Flow PHP - data processing framework
https://flow-php.com
MIT License
404 stars 23 forks source link

Added type comparable assertion to scalar comparison functions to avoid unpredictable comparisons between different types #1081

Closed norberttech closed 1 month ago

norberttech commented 1 month ago

Change Log

Added

Fixed

  • Scalar comparison functions are now also detecting types of compared values to prevent comparing not compatible values

Changed

  • String based Scalar functions are now using caster to cast values to string
  • Moved caster to dataframe configuration

Removed

Deprecated

Security

Description

github-actions[bot] commented 1 month ago

Flow PHP - Benchmarks

Results of the benchmarks from this PR are compared with the results from 1.x branch.

Extractors ```shell +-----------------------+-------------------+------+-----+------------------+------------------+----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-----------------------+-------------------+------+-----+------------------+------------------+----------------+ | AvroExtractorBench | bench_extract_10k | 1 | 3 | 35.437mb +0.03% | 826.871ms +0.72% | ±0.46% -48.76% | | CSVExtractorBench | bench_extract_10k | 1 | 3 | 5.157mb +0.15% | 336.703ms +0.50% | ±0.13% -43.55% | | JsonExtractorBench | bench_extract_10k | 1 | 3 | 5.188mb +0.18% | 1.076s +0.32% | ±0.37% -47.46% | | ParquetExtractorBench | bench_extract_10k | 1 | 3 | 135.860mb +0.01% | 902.966ms -0.49% | ±0.07% -82.96% | | TextExtractorBench | bench_extract_10k | 1 | 3 | 4.948mb +0.16% | 35.709ms -1.08% | ±1.06% -13.34% | | XmlExtractorBench | bench_extract_10k | 1 | 3 | 4.954mb +0.16% | 430.814ms -1.63% | ±0.56% +36.80% | +-----------------------+-------------------+------+-----+------------------+------------------+----------------+ ```
Transformers ```shell +-----------------------------+--------------------------+------+-----+------------------+-----------------+----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-----------------------------+--------------------------+------+-----+------------------+-----------------+----------------+ | RenameEntryTransformerBench | bench_transform_10k_rows | 1 | 3 | 116.254mb +0.01% | 58.939ms -0.45% | ±1.21% +50.37% | +-----------------------------+--------------------------+------+-----+------------------+-----------------+----------------+ ```
Loaders ```shell +--------------------+----------------+------+-----+------------------+------------------+----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +--------------------+----------------+------+-----+------------------+------------------+----------------+ | AvroLoaderBench | bench_load_10k | 1 | 3 | 96.822mb +0.01% | 449.871ms -1.02% | ±0.35% -32.57% | | CSVLoaderBench | bench_load_10k | 1 | 3 | 55.236mb +0.01% | 68.943ms -0.05% | ±0.77% +21.82% | | JsonLoaderBench | bench_load_10k | 1 | 3 | 107.608mb +0.01% | 51.655ms -1.59% | ±0.90% -10.62% | | ParquetLoaderBench | bench_load_10k | 1 | 3 | 227.028mb +0.00% | 1.396s -0.78% | ±0.18% -56.12% | | TextLoaderBench | bench_load_10k | 1 | 3 | 17.991mb +0.04% | 38.605ms -0.30% | ±0.12% -74.55% | +--------------------+----------------+------+-----+------------------+------------------+----------------+ ```
Building Blocks ```shell +-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+ | RowsBench | bench_chunk_10_on_10k | 2 | 3 | 87.075mb +0.01% | 3.315ms -3.17% | ±0.89% -62.18% | | RowsBench | bench_diff_left_1k_on_10k | 2 | 3 | 102.673mb +0.01% | 186.883ms -2.57% | ±0.52% -63.91% | | RowsBench | bench_diff_right_1k_on_10k | 2 | 3 | 85.393mb +0.01% | 18.583ms -2.38% | ±0.45% +11.63% | | RowsBench | bench_drop_1k_on_10k | 2 | 3 | 88.315mb +0.01% | 1.712ms +2.08% | ±1.17% +190.33% | | RowsBench | bench_drop_right_1k_on_10k | 2 | 3 | 88.315mb +0.01% | 1.727ms +2.21% | ±0.62% -53.68% | | RowsBench | bench_entries_on_10k | 2 | 3 | 85.427mb +0.01% | 2.774ms +5.48% | ±0.79% -45.51% | | RowsBench | bench_filter_on_10k | 2 | 3 | 85.956mb +0.01% | 18.437ms +24.93% | ±0.92% +65.20% | | RowsBench | bench_find_on_10k | 2 | 3 | 85.956mb +0.01% | 18.280ms +22.11% | ±2.60% +162.91% | | RowsBench | bench_find_one_on_10k | 10 | 3 | 83.860mb +0.01% | 1.606μs 0.00% | ±2.89% 0.00% | | RowsBench | bench_first_on_10k | 10 | 3 | 83.860mb +0.01% | 0.300μs 0.00% | ±0.00% 0.00% | | RowsBench | bench_flat_map_on_1k | 2 | 3 | 93.210mb +0.01% | 12.408ms +1.44% | ±1.31% +331.40% | | RowsBench | bench_map_on_10k | 2 | 3 | 122.580mb +0.00% | 60.240ms -2.97% | ±0.33% -50.77% | | RowsBench | bench_merge_1k_on_10k | 2 | 3 | 86.475mb +0.01% | 1.239ms +2.06% | ±1.23% -12.72% | | RowsBench | bench_partition_by_on_10k | 2 | 3 | 89.822mb +0.01% | 64.382ms +3.52% | ±0.94% +3.13% | | RowsBench | bench_remove_on_10k | 2 | 3 | 88.577mb +0.01% | 4.084ms +5.57% | ±0.78% -56.38% | | RowsBench | bench_sort_asc_on_1k | 2 | 3 | 84.003mb +0.01% | 39.524ms +0.94% | ±1.44% +53.08% | | RowsBench | bench_sort_by_on_1k | 2 | 3 | 84.004mb +0.01% | 39.245ms -1.04% | ±0.43% -42.49% | | RowsBench | bench_sort_desc_on_1k | 2 | 3 | 84.003mb +0.01% | 40.407ms +3.82% | ±1.08% +997.97% | | RowsBench | bench_sort_entries_on_1k | 2 | 3 | 86.301mb +0.01% | 7.333ms +0.28% | ±0.71% -18.22% | | RowsBench | bench_sort_on_1k | 2 | 3 | 83.860mb +0.01% | 28.709ms +0.85% | ±0.31% -41.71% | | RowsBench | bench_take_1k_on_10k | 10 | 3 | 83.860mb +0.01% | 13.294μs -3.66% | ±0.36% -79.98% | | RowsBench | bench_take_right_1k_on_10k | 10 | 3 | 83.860mb +0.01% | 15.940μs -2.92% | ±3.15% +267.23% | | RowsBench | bench_unique_on_1k | 2 | 3 | 102.674mb +0.01% | 190.592ms -4.62% | ±0.90% +225.96% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 116.804mb +0.01% | 495.973ms +0.07% | ±0.88% +262.11% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 60.282mb +0.01% | 250.537ms +0.51% | ±0.97% -59.76% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 15.216mb +0.04% | 52.802ms -2.41% | ±1.07% -24.61% | | TypeDetectorBench | bench_type_detector | 1 | 3 | 59.990mb +0.01% | 425.484ms -2.12% | ±0.30% -90.54% | | TypeDetectorBench | bench_type_detector | 1 | 3 | 14.529mb +0.03% | 85.085ms -1.15% | ±1.14% +102.93% | +-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+ ```