search

flow-php / flow

Flow PHP - data processing framework
https://flow-php.com
MIT License
491 stars 28 forks source link

Added more output styles for to_output() loader #1238

Closed norberttech closed 2 months ago

norberttech commented 2 months ago

Change Log

Added

  • Added more output styles for to_output() loader

Fixed

Changed

Removed

Deprecated

Security

Description

Example: 

df()
    ->read(from_parquet(__DIR__ . '/orders_with_products.parquet'))
    ->batchSize(1000)
    ->write(to_output(output: Output::rows_and_column_count))
    ->run();

Output: 

php .scratchpad/flow/read.php
Rows: 1000, Columns: 9
Rows: 1000, Columns: 9
Rows: 1000, Columns: 9
Rows: 1000, Columns: 9
Rows: 1000, Columns: 9
Rows: 1000, Columns: 9
Rows: 1000, Columns: 9
Rows: 1000, Columns: 9
Rows: 1000, Columns: 9
Rows: 1000, Columns: 9
Rows: 1000, Columns: 9
github-actions[bot] commented 2 months ago

Flow PHP - Benchmarks

Results of the benchmarks from this PR are compared with the results from 1.x branch.

Extractors ```shell +-----------------------+-------------------+------+-----+-----------------+------------------+-----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-----------------------+-------------------+------+-----+-----------------+------------------+-----------------+ | CSVExtractorBench | bench_extract_10k | 1 | 3 | 4.631mb +0.01% | 521.060ms +0.79% | ±0.36% +267.85% | | JsonExtractorBench | bench_extract_10k | 1 | 3 | 4.719mb +0.01% | 1.093s +1.02% | ±0.17% -42.81% | | ParquetExtractorBench | bench_extract_10k | 1 | 3 | 29.175mb +0.00% | 436.013ms +0.16% | ±0.70% -49.44% | | TextExtractorBench | bench_extract_10k | 1 | 3 | 4.360mb +0.01% | 33.330ms -2.30% | ±0.86% -43.12% | | XmlExtractorBench | bench_extract_10k | 1 | 3 | 4.341mb +0.01% | 652.655ms -6.04% | ±0.72% -64.54% | +-----------------------+-------------------+------+-----+-----------------+------------------+-----------------+ ```
Transformers ```shell +-----------------------------+--------------------------+------+-----+------------------+-----------------+---------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-----------------------------+--------------------------+------+-----+------------------+-----------------+---------------+ | RenameEntryTransformerBench | bench_transform_10k_rows | 1 | 3 | 116.701mb +0.00% | 60.429ms +0.76% | ±0.28% -5.43% | +-----------------------------+--------------------------+------+-----+------------------+-----------------+---------------+ ```
Loaders ```shell +--------------------+----------------+------+-----+------------------+------------------+------------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +--------------------+----------------+------+-----+------------------+------------------+------------------+ | CSVLoaderBench | bench_load_10k | 1 | 3 | 54.829mb +0.00% | 119.763ms -2.67% | ±1.50% +1016.33% | | JsonLoaderBench | bench_load_10k | 1 | 3 | 90.412mb +0.00% | 99.163ms -2.81% | ±0.40% -70.77% | | ParquetLoaderBench | bench_load_10k | 1 | 3 | 124.532mb +0.00% | 1.231s -0.76% | ±0.30% -73.34% | | TextLoaderBench | bench_load_10k | 1 | 3 | 17.551mb +0.00% | 27.716ms -6.12% | ±0.56% -56.77% | +--------------------+----------------+------+-----+------------------+------------------+------------------+ ```
Building Blocks ```shell +-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+ | TypeDetectorBench | bench_type_detector | 1 | 3 | 53.271mb +0.00% | 400.413ms +1.37% | ±0.73% -34.88% | | TypeDetectorBench | bench_type_detector | 1 | 3 | 13.537mb +0.00% | 80.791ms +0.87% | ±0.30% -65.28% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 107.468mb +0.00% | 493.502ms +3.24% | ±0.79% +0.71% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 55.826mb +0.00% | 243.685ms +0.75% | ±0.61% -29.71% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 14.664mb +0.00% | 52.249ms -0.08% | ±2.29% +311.38% | | RowsBench | bench_chunk_10_on_10k | 2 | 3 | 87.445mb +0.00% | 3.300ms -2.10% | ±3.05% +108.35% | | RowsBench | bench_diff_left_1k_on_10k | 2 | 3 | 102.983mb +0.00% | 190.942ms +1.62% | ±0.45% -8.18% | | RowsBench | bench_diff_right_1k_on_10k | 2 | 3 | 85.703mb +0.00% | 19.133ms +1.61% | ±0.54% -44.56% | | RowsBench | bench_drop_1k_on_10k | 2 | 3 | 88.685mb +0.00% | 1.859ms -1.84% | ±1.17% -49.38% | | RowsBench | bench_drop_right_1k_on_10k | 2 | 3 | 88.685mb +0.00% | 1.823ms -3.56% | ±0.67% -78.77% | | RowsBench | bench_entries_on_10k | 2 | 3 | 85.797mb +0.00% | 2.746ms -2.85% | ±1.60% +10.11% | | RowsBench | bench_filter_on_10k | 2 | 3 | 86.326mb +0.00% | 17.879ms +7.93% | ±0.94% -13.94% | | RowsBench | bench_find_on_10k | 2 | 3 | 86.326mb +0.00% | 18.118ms +6.89% | ±2.38% +16.35% | | RowsBench | bench_find_one_on_10k | 10 | 3 | 84.230mb +0.00% | 1.694μs +5.47% | ±2.83% -2.00% | | RowsBench | bench_first_on_10k | 10 | 3 | 84.230mb +0.00% | 0.300μs 0.00% | ±0.00% 0.00% | | RowsBench | bench_flat_map_on_1k | 2 | 3 | 93.580mb +0.00% | 12.576ms +0.14% | ±0.70% -26.07% | | RowsBench | bench_map_on_10k | 2 | 3 | 122.951mb +0.00% | 62.517ms +2.97% | ±1.07% -20.69% | | RowsBench | bench_merge_1k_on_10k | 2 | 3 | 86.846mb +0.00% | 1.391ms -9.56% | ±0.90% -72.69% | | RowsBench | bench_partition_by_on_10k | 2 | 3 | 90.136mb +0.00% | 60.822ms +1.12% | ±0.38% -83.40% | | RowsBench | bench_remove_on_10k | 2 | 3 | 88.947mb +0.00% | 4.141ms -9.53% | ±0.19% -93.44% | | RowsBench | bench_sort_asc_on_1k | 2 | 3 | 84.314mb +0.00% | 40.582ms +2.26% | ±0.95% -21.57% | | RowsBench | bench_sort_by_on_1k | 2 | 3 | 84.314mb +0.00% | 40.479ms +3.34% | ±0.88% +45.05% | | RowsBench | bench_sort_desc_on_1k | 2 | 3 | 84.314mb +0.00% | 40.812ms +3.16% | ±1.18% -16.47% | | RowsBench | bench_sort_entries_on_1k | 2 | 3 | 86.671mb +0.00% | 7.474ms -1.10% | ±0.10% -92.20% | | RowsBench | bench_sort_on_1k | 2 | 3 | 84.230mb +0.00% | 29.161ms +1.05% | ±1.13% -12.38% | | RowsBench | bench_take_1k_on_10k | 10 | 3 | 84.230mb +0.00% | 13.112μs -8.69% | ±0.72% -80.20% | | RowsBench | bench_take_right_1k_on_10k | 10 | 3 | 84.230mb +0.00% | 15.961μs -1.68% | ±2.90% +128.47% | | RowsBench | bench_unique_on_1k | 2 | 3 | 102.988mb +0.00% | 193.753ms +0.25% | ±0.36% -61.35% | +-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+ ```