flow-php / flow

Flow PHP - data processing framework
https://flow-php.com
MIT License
404 stars 23 forks source link

Fixed writing parquet to remote locations #989

Closed norberttech closed 4 months ago

norberttech commented 4 months ago

Change Log

Added

Fixed

  • Fixed writing parquet to remote locations

Changed

Removed

Deprecated

Security


Description

github-actions[bot] commented 4 months ago

Flow PHP - Benchmarks

Results of the benchmarks from this PR are compared with the results from 1.x branch.

Extractors ```shell +-----------------------+-------------------+------+-----+------------------+------------------+-----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-----------------------+-------------------+------+-----+------------------+------------------+-----------------+ | AvroExtractorBench | bench_extract_10k | 1 | 3 | 35.269mb +0.00% | 827.415ms +1.82% | ±1.97% +168.31% | | CSVExtractorBench | bench_extract_10k | 1 | 3 | 4.989mb +0.01% | 341.055ms +0.17% | ±0.33% -28.03% | | JsonExtractorBench | bench_extract_10k | 1 | 3 | 5.138mb +0.01% | 1.044s -4.15% | ±0.28% -85.52% | | ParquetExtractorBench | bench_extract_10k | 1 | 3 | 135.794mb +0.00% | 901.091ms +1.34% | ±0.27% -70.46% | | TextExtractorBench | bench_extract_10k | 1 | 3 | 4.900mb +0.01% | 34.995ms -0.82% | ±1.50% +101.02% | | XmlExtractorBench | bench_extract_10k | 1 | 3 | 4.902mb +0.01% | 434.830ms +0.90% | ±1.17% -18.25% | +-----------------------+-------------------+------+-----+------------------+------------------+-----------------+ ```
Transformers ```shell +-----------------------------+--------------------------+------+-----+------------------+-----------------+-----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-----------------------------+--------------------------+------+-----+------------------+-----------------+-----------------+ | RenameEntryTransformerBench | bench_transform_10k_rows | 1 | 3 | 110.608mb +0.00% | 67.093ms +5.19% | ±2.19% +340.16% | +-----------------------------+--------------------------+------+-----+------------------+-----------------+-----------------+ ```
Loaders ```shell +--------------------+----------------+------+-----+------------------+------------------+-----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +--------------------+----------------+------+-----+------------------+------------------+-----------------+ | AvroLoaderBench | bench_load_10k | 1 | 3 | 95.649mb +0.00% | 465.450ms +0.97% | ±1.08% -33.83% | | CSVLoaderBench | bench_load_10k | 1 | 3 | 54.125mb +0.00% | 73.046ms +1.44% | ±0.22% -76.39% | | JsonLoaderBench | bench_load_10k | 1 | 3 | 106.554mb +0.00% | 53.226ms +1.24% | ±1.32% -21.90% | | ParquetLoaderBench | bench_load_10k | 1 | 3 | 224.361mb +0.00% | 1.453s +2.10% | ±1.36% +179.26% | | TextLoaderBench | bench_load_10k | 1 | 3 | 17.948mb +0.00% | 41.263ms +1.06% | ±0.21% -58.71% | +--------------------+----------------+------+-----+------------------+------------------+-----------------+ ```
Building Blocks ```shell +-------------------------+----------------------------+------+-----+------------------+------------------+------------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-------------------------+----------------------------+------+-----+------------------+------------------+------------------+ | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 116.715mb +0.00% | 494.678ms +1.45% | ±0.50% -55.92% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 60.193mb +0.00% | 248.565ms +1.89% | ±1.77% +76.33% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 15.127mb +0.00% | 52.538ms +1.37% | ±0.99% +26.39% | | TypeDetectorBench | bench_type_detector | 1 | 3 | 59.959mb +0.00% | 437.718ms +1.91% | ±0.45% +67.12% | | TypeDetectorBench | bench_type_detector | 1 | 3 | 14.498mb +0.00% | 86.708ms +2.12% | ±0.56% +53.43% | | RowsBench | bench_chunk_10_on_10k | 2 | 3 | 76.683mb +0.00% | 4.072ms +24.91% | ±1.68% +59.31% | | RowsBench | bench_diff_left_1k_on_10k | 2 | 3 | 96.410mb +0.00% | 181.248ms +0.29% | ±0.29% -48.49% | | RowsBench | bench_diff_right_1k_on_10k | 2 | 3 | 74.936mb +0.00% | 18.135ms -0.38% | ±0.96% +79.87% | | RowsBench | bench_drop_1k_on_10k | 2 | 3 | 77.923mb +0.00% | 2.066ms +22.49% | ±3.33% +163.70% | | RowsBench | bench_drop_right_1k_on_10k | 2 | 3 | 77.923mb +0.00% | 2.042ms +22.56% | ±0.96% -40.50% | | RowsBench | bench_entries_on_10k | 2 | 3 | 75.035mb +0.00% | 2.936ms +19.65% | ±1.97% +79.43% | | RowsBench | bench_filter_on_10k | 2 | 3 | 75.564mb +0.00% | 14.940ms +5.65% | ±1.65% -41.02% | | RowsBench | bench_find_on_10k | 2 | 3 | 75.564mb +0.00% | 14.959ms +3.54% | ±1.78% -5.89% | | RowsBench | bench_find_one_on_10k | 10 | 3 | 73.468mb +0.00% | 1.906μs +6.25% | ±2.44% -8.62% | | RowsBench | bench_first_on_10k | 10 | 3 | 73.468mb +0.00% | 0.400μs 0.00% | ±0.00% 0.00% | | RowsBench | bench_flat_map_on_1k | 2 | 3 | 87.023mb +0.00% | 14.229ms +8.66% | ±0.57% -69.48% | | RowsBench | bench_map_on_10k | 2 | 3 | 116.383mb +0.00% | 67.896ms +3.17% | ±1.70% +471.90% | | RowsBench | bench_merge_1k_on_10k | 2 | 3 | 76.084mb +0.00% | 1.792ms +48.28% | ±2.17% -28.18% | | RowsBench | bench_partition_by_on_10k | 2 | 3 | 79.431mb +0.00% | 59.747ms +3.10% | ±0.79% +37.66% | | RowsBench | bench_remove_on_10k | 2 | 3 | 78.185mb +0.00% | 4.546ms +11.47% | ±2.06% -35.24% | | RowsBench | bench_sort_asc_on_1k | 2 | 3 | 73.546mb +0.00% | 41.924ms +3.58% | ±2.14% +148.06% | | RowsBench | bench_sort_by_on_1k | 2 | 3 | 73.547mb +0.00% | 40.717ms +1.29% | ±2.79% +115.67% | | RowsBench | bench_sort_desc_on_1k | 2 | 3 | 73.546mb +0.00% | 41.510ms +2.14% | ±1.85% +1090.65% | | RowsBench | bench_sort_entries_on_1k | 2 | 3 | 75.910mb +0.00% | 7.483ms +2.37% | ±0.22% +128.42% | | RowsBench | bench_sort_on_1k | 2 | 3 | 73.468mb +0.00% | 29.680ms +4.01% | ±0.28% -86.83% | | RowsBench | bench_take_1k_on_10k | 10 | 3 | 73.468mb +0.00% | 14.021μs -0.42% | ±0.89% -25.57% | | RowsBench | bench_take_right_1k_on_10k | 10 | 3 | 73.468mb +0.00% | 16.400μs +2.50% | ±0.50% -2.44% | | RowsBench | bench_unique_on_1k | 2 | 3 | 96.477mb +0.00% | 185.170ms -1.12% | ±0.72% +12.93% | +-------------------------+----------------------------+------+-----+------------------+------------------+------------------+ ```