flow-php / flow

Flow PHP - data processing framework
https://flow-php.com
MIT License
404 stars 23 forks source link

Drop partition columns #987

Closed norberttech closed 4 months ago

norberttech commented 4 months ago

Change Log

Added

  • DataFrame::dropPartitions can now additionally drop also partition columns

Fixed

Changed

Removed

Deprecated

Security


Description

github-actions[bot] commented 4 months ago

Flow PHP - Benchmarks

Results of the benchmarks from this PR are compared with the results from 1.x branch.

Extractors ```shell +-----------------------+-------------------+------+-----+------------------+------------------+-----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-----------------------+-------------------+------+-----+------------------+------------------+-----------------+ | AvroExtractorBench | bench_extract_10k | 1 | 3 | 35.262mb +0.01% | 818.482ms +0.34% | ±0.68% -64.46% | | CSVExtractorBench | bench_extract_10k | 1 | 3 | 4.981mb +0.06% | 341.785ms +0.23% | ±0.89% +457.98% | | JsonExtractorBench | bench_extract_10k | 1 | 3 | 5.131mb +0.05% | 1.050s -0.26% | ±1.27% -42.33% | | ParquetExtractorBench | bench_extract_10k | 1 | 3 | 135.787mb +0.00% | 888.791ms +0.47% | ±0.55% -38.04% | | TextExtractorBench | bench_extract_10k | 1 | 3 | 4.895mb +0.01% | 35.583ms +0.54% | ±0.36% -33.76% | | XmlExtractorBench | bench_extract_10k | 1 | 3 | 4.897mb +0.01% | 433.034ms -1.86% | ±0.69% -69.97% | +-----------------------+-------------------+------+-----+------------------+------------------+-----------------+ ```
Transformers ```shell +-----------------------------+--------------------------+------+-----+------------------+-----------------+---------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-----------------------------+--------------------------+------+-----+------------------+-----------------+---------------+ | RenameEntryTransformerBench | bench_transform_10k_rows | 1 | 3 | 110.601mb +0.00% | 65.204ms +0.29% | ±0.70% -2.09% | +-----------------------------+--------------------------+------+-----+------------------+-----------------+---------------+ ```
Loaders ```shell +--------------------+----------------+------+-----+------------------+------------------+-----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +--------------------+----------------+------+-----+------------------+------------------+-----------------+ | AvroLoaderBench | bench_load_10k | 1 | 3 | 95.642mb +0.00% | 462.004ms +0.16% | ±0.61% +15.29% | | CSVLoaderBench | bench_load_10k | 1 | 3 | 54.117mb +0.01% | 72.508ms -0.14% | ±0.37% -16.19% | | JsonLoaderBench | bench_load_10k | 1 | 3 | 106.547mb +0.00% | 52.819ms +0.66% | ±0.57% +107.14% | | ParquetLoaderBench | bench_load_10k | 1 | 3 | 224.354mb +0.00% | 1.423s -0.33% | ±0.29% -12.04% | | TextLoaderBench | bench_load_10k | 1 | 3 | 17.940mb +0.02% | 40.544ms -0.41% | ±1.09% +383.10% | +--------------------+----------------+------+-----+------------------+------------------+-----------------+ ```
Building Blocks ```shell +-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+ | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 116.708mb +0.00% | 498.156ms -0.53% | ±1.25% +134.29% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 60.186mb +0.00% | 251.725ms +0.58% | ±1.70% +16.18% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 15.120mb +0.02% | 52.667ms -1.08% | ±1.63% -18.22% | | TypeDetectorBench | bench_type_detector | 1 | 3 | 59.954mb +0.00% | 432.793ms -1.41% | ±0.46% -54.75% | | TypeDetectorBench | bench_type_detector | 1 | 3 | 14.493mb +0.00% | 86.149ms -0.60% | ±1.98% +54.47% | | RowsBench | bench_chunk_10_on_10k | 2 | 3 | 76.676mb +0.00% | 3.452ms -7.69% | ±1.62% -53.47% | | RowsBench | bench_diff_left_1k_on_10k | 2 | 3 | 96.403mb +0.00% | 180.469ms +0.64% | ±0.72% +110.59% | | RowsBench | bench_diff_right_1k_on_10k | 2 | 3 | 74.928mb +0.00% | 18.020ms -0.98% | ±0.25% -58.97% | | RowsBench | bench_drop_1k_on_10k | 2 | 3 | 77.916mb +0.00% | 1.639ms -19.41% | ±0.71% -0.57% | | RowsBench | bench_drop_right_1k_on_10k | 2 | 3 | 77.916mb +0.00% | 1.725ms -14.27% | ±1.70% +306.03% | | RowsBench | bench_entries_on_10k | 2 | 3 | 75.028mb +0.00% | 2.542ms -5.04% | ±3.06% +21.57% | | RowsBench | bench_filter_on_10k | 2 | 3 | 75.557mb +0.00% | 14.795ms -2.19% | ±0.70% -57.07% | | RowsBench | bench_find_on_10k | 2 | 3 | 75.557mb +0.00% | 15.207ms +2.56% | ±0.35% -57.14% | | RowsBench | bench_find_one_on_10k | 10 | 3 | 73.461mb +0.00% | 1.800μs -9.73% | ±0.00% -100.00% | | RowsBench | bench_first_on_10k | 10 | 3 | 73.461mb +0.00% | 0.300μs -25.00% | ±0.00% -100.00% | | RowsBench | bench_flat_map_on_1k | 2 | 3 | 87.016mb +0.00% | 12.796ms -12.53% | ±0.72% -18.20% | | RowsBench | bench_map_on_10k | 2 | 3 | 116.376mb +0.00% | 67.047ms -0.68% | ±2.06% +15.01% | | RowsBench | bench_merge_1k_on_10k | 2 | 3 | 76.077mb +0.00% | 1.161ms -20.26% | ±2.22% -8.68% | | RowsBench | bench_partition_by_on_10k | 2 | 3 | 79.423mb +0.00% | 57.571ms -4.14% | ±0.90% +4.42% | | RowsBench | bench_remove_on_10k | 2 | 3 | 78.178mb +0.00% | 3.819ms -3.28% | ±3.00% +154.98% | | RowsBench | bench_sort_asc_on_1k | 2 | 3 | 73.539mb +0.00% | 40.985ms -1.89% | ±2.57% +173.39% | | RowsBench | bench_sort_by_on_1k | 2 | 3 | 73.540mb +0.00% | 41.279ms +0.15% | ±2.47% +225.01% | | RowsBench | bench_sort_desc_on_1k | 2 | 3 | 73.539mb +0.00% | 40.433ms -4.32% | ±0.72% +175.79% | | RowsBench | bench_sort_entries_on_1k | 2 | 3 | 75.902mb +0.00% | 7.406ms -2.18% | ±0.27% -82.75% | | RowsBench | bench_sort_on_1k | 2 | 3 | 73.461mb +0.00% | 29.627ms +0.26% | ±1.01% +4.52% | | RowsBench | bench_take_1k_on_10k | 10 | 3 | 73.461mb +0.00% | 13.258μs -4.74% | ±2.20% +40.34% | | RowsBench | bench_take_right_1k_on_10k | 10 | 3 | 73.461mb +0.00% | 15.966μs -2.61% | ±1.28% +344.95% | | RowsBench | bench_unique_on_1k | 2 | 3 | 96.469mb +0.00% | 184.916ms -1.02% | ±0.27% -82.94% | +-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+ ```