flow-php / flow

Flow PHP - data processing framework
https://flow-php.com
MIT License
404 stars 23 forks source link

Overwrite mode on partitions #1052

Closed norberttech closed 2 months ago

norberttech commented 2 months ago

Change Log

Added

  • Test that covers writing to a partitioned dataset with overwrite mode

Fixed

Changed

Removed

Deprecated

Security


Description

github-actions[bot] commented 2 months ago

Flow PHP - Benchmarks

Results of the benchmarks from this PR are compared with the results from 1.x branch.

Extractors ```shell +-----------------------+-------------------+------+-----+------------------+------------------+----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-----------------------+-------------------+------+-----+------------------+------------------+----------------+ | AvroExtractorBench | bench_extract_10k | 1 | 3 | 35.299mb +0.00% | 842.329ms -2.33% | ±0.25% -42.25% | | CSVExtractorBench | bench_extract_10k | 1 | 3 | 5.019mb +0.01% | 348.683ms +1.01% | ±0.37% -69.15% | | JsonExtractorBench | bench_extract_10k | 1 | 3 | 5.174mb +0.01% | 1.077s +0.05% | ±0.34% -3.28% | | ParquetExtractorBench | bench_extract_10k | 1 | 3 | 135.846mb +0.00% | 925.602ms +0.07% | ±0.43% -63.61% | | TextExtractorBench | bench_extract_10k | 1 | 3 | 4.929mb +0.01% | 35.304ms -0.96% | ±0.12% -65.69% | | XmlExtractorBench | bench_extract_10k | 1 | 3 | 4.934mb +0.01% | 430.460ms -0.90% | ±0.70% -37.98% | +-----------------------+-------------------+------+-----+------------------+------------------+----------------+ ```
Transformers ```shell +-----------------------------+--------------------------+------+-----+------------------+-----------------+-----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-----------------------------+--------------------------+------+-----+------------------+-----------------+-----------------+ | RenameEntryTransformerBench | bench_transform_10k_rows | 1 | 3 | 116.240mb +0.00% | 61.212ms +1.92% | ±1.27% +198.23% | +-----------------------------+--------------------------+------+-----+------------------+-----------------+-----------------+ ```
Loaders ```shell +--------------------+----------------+------+-----+------------------+------------------+------------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +--------------------+----------------+------+-----+------------------+------------------+------------------+ | AvroLoaderBench | bench_load_10k | 1 | 3 | 96.685mb +0.00% | 459.643ms -1.47% | ±0.11% -90.33% | | CSVLoaderBench | bench_load_10k | 1 | 3 | 55.164mb +0.00% | 70.339ms +1.69% | ±1.23% +1036.22% | | JsonLoaderBench | bench_load_10k | 1 | 3 | 107.594mb +0.00% | 51.393ms +0.13% | ±1.14% +183.57% | | ParquetLoaderBench | bench_load_10k | 1 | 3 | 227.014mb +0.00% | 1.438s +1.51% | ±0.93% +26.80% | | TextLoaderBench | bench_load_10k | 1 | 3 | 17.976mb +0.00% | 38.801ms +0.12% | ±0.41% +131.95% | +--------------------+----------------+------+-----+------------------+------------------+------------------+ ```
Building Blocks ```shell +-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+ | RowsBench | bench_chunk_10_on_10k | 2 | 3 | 87.060mb +0.00% | 3.516ms -3.84% | ±2.29% -13.43% | | RowsBench | bench_diff_left_1k_on_10k | 2 | 3 | 102.659mb +0.00% | 194.889ms +2.49% | ±0.33% -59.83% | | RowsBench | bench_diff_right_1k_on_10k | 2 | 3 | 85.379mb +0.00% | 19.317ms +3.20% | ±0.23% -72.52% | | RowsBench | bench_drop_1k_on_10k | 2 | 3 | 88.300mb +0.00% | 1.751ms -1.35% | ±1.45% -7.73% | | RowsBench | bench_drop_right_1k_on_10k | 2 | 3 | 88.300mb +0.00% | 1.668ms -0.73% | ±2.51% +91.27% | | RowsBench | bench_entries_on_10k | 2 | 3 | 85.413mb +0.00% | 2.537ms -3.22% | ±0.05% -96.43% | | RowsBench | bench_filter_on_10k | 2 | 3 | 85.941mb +0.00% | 16.390ms -19.80% | ±1.64% +44.34% | | RowsBench | bench_find_on_10k | 2 | 3 | 85.941mb +0.00% | 16.131ms -21.56% | ±1.89% -26.42% | | RowsBench | bench_find_one_on_10k | 10 | 3 | 83.846mb +0.00% | 1.706μs +0.72% | ±2.72% -3.85% | | RowsBench | bench_first_on_10k | 10 | 3 | 83.846mb +0.00% | 0.400μs 0.00% | ±0.00% 0.00% | | RowsBench | bench_flat_map_on_1k | 2 | 3 | 93.196mb +0.00% | 12.596ms +0.35% | ±1.86% +128.38% | | RowsBench | bench_map_on_10k | 2 | 3 | 122.566mb +0.00% | 62.147ms -0.28% | ±0.93% +37.73% | | RowsBench | bench_merge_1k_on_10k | 2 | 3 | 86.461mb +0.00% | 1.317ms -5.22% | ±3.27% +263.13% | | RowsBench | bench_partition_by_on_10k | 2 | 3 | 89.808mb +0.00% | 64.833ms -1.69% | ±0.69% -42.03% | | RowsBench | bench_remove_on_10k | 2 | 3 | 88.563mb +0.00% | 3.852ms -0.75% | ±0.83% +32.52% | | RowsBench | bench_sort_asc_on_1k | 2 | 3 | 83.989mb +0.00% | 39.454ms -0.30% | ±1.84% +969.93% | | RowsBench | bench_sort_by_on_1k | 2 | 3 | 83.990mb +0.00% | 40.549ms +0.30% | ±1.83% +65.75% | | RowsBench | bench_sort_desc_on_1k | 2 | 3 | 83.989mb +0.00% | 40.578ms +0.00% | ±0.56% -51.60% | | RowsBench | bench_sort_entries_on_1k | 2 | 3 | 86.287mb +0.00% | 7.409ms -0.36% | ±0.19% -69.08% | | RowsBench | bench_sort_on_1k | 2 | 3 | 83.846mb +0.00% | 28.748ms -0.20% | ±1.12% +202.94% | | RowsBench | bench_take_1k_on_10k | 10 | 3 | 83.846mb +0.00% | 13.294μs -0.64% | ±0.36% -71.85% | | RowsBench | bench_take_right_1k_on_10k | 10 | 3 | 83.846mb +0.00% | 15.880μs -0.60% | ±1.06% -10.43% | | RowsBench | bench_unique_on_1k | 2 | 3 | 102.660mb +0.00% | 193.395ms +0.95% | ±0.92% +33.11% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 116.738mb +0.00% | 519.527ms -0.70% | ±0.61% -59.47% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 60.216mb +0.00% | 259.472ms +1.27% | ±0.80% -14.91% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 15.151mb +0.00% | 54.461ms -2.81% | ±0.84% -36.44% | | TypeDetectorBench | bench_type_detector | 1 | 3 | 59.970mb +0.00% | 432.785ms -0.55% | ±0.93% +78.26% | | TypeDetectorBench | bench_type_detector | 1 | 3 | 14.509mb +0.00% | 85.517ms -1.66% | ±0.27% -92.57% | +-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+ ```