flow-php / flow

Flow PHP - data processing framework
https://flow-php.com
MIT License
470 stars 26 forks source link

Make sure that filesystem scans are returning files in proper order #1051

Closed norberttech closed 5 months ago

norberttech commented 5 months ago

Change Log

Added

Fixed

  • Make sure that filesystem scans are returning files in proper order

Changed

Removed

Deprecated

Security


Description

github-actions[bot] commented 5 months ago

Flow PHP - Benchmarks

Results of the benchmarks from this PR are compared with the results from 1.x branch.

Extractors ```shell +-----------------------+-------------------+------+-----+------------------+------------------+----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-----------------------+-------------------+------+-----+------------------+------------------+----------------+ | AvroExtractorBench | bench_extract_10k | 1 | 3 | 35.299mb +0.00% | 863.744ms +1.53% | ±1.15% +48.76% | | CSVExtractorBench | bench_extract_10k | 1 | 3 | 5.019mb +0.00% | 345.612ms +0.25% | ±0.32% -26.54% | | JsonExtractorBench | bench_extract_10k | 1 | 3 | 5.174mb +0.00% | 1.086s +0.38% | ±0.35% -56.32% | | ParquetExtractorBench | bench_extract_10k | 1 | 3 | 135.846mb +0.00% | 911.576ms -1.10% | ±0.04% -85.56% | | TextExtractorBench | bench_extract_10k | 1 | 3 | 4.928mb +0.01% | 35.371ms -0.47% | ±1.51% +26.68% | | XmlExtractorBench | bench_extract_10k | 1 | 3 | 4.934mb +0.01% | 454.823ms +6.00% | ±2.84% +12.66% | +-----------------------+-------------------+------+-----+------------------+------------------+----------------+ ```
Transformers ```shell +-----------------------------+--------------------------+------+-----+------------------+-----------------+----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-----------------------------+--------------------------+------+-----+------------------+-----------------+----------------+ | RenameEntryTransformerBench | bench_transform_10k_rows | 1 | 3 | 116.240mb +0.00% | 60.063ms -1.13% | ±0.06% -68.33% | +-----------------------------+--------------------------+------+-----+------------------+-----------------+----------------+ ```
Loaders ```shell +--------------------+----------------+------+-----+------------------+------------------+-----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +--------------------+----------------+------+-----+------------------+------------------+-----------------+ | AvroLoaderBench | bench_load_10k | 1 | 3 | 96.685mb +0.00% | 450.679ms -4.00% | ±0.37% -16.58% | | CSVLoaderBench | bench_load_10k | 1 | 3 | 55.164mb +0.00% | 67.276ms -3.16% | ±0.20% -47.05% | | JsonLoaderBench | bench_load_10k | 1 | 3 | 107.594mb +0.00% | 50.063ms -1.90% | ±0.25% -79.03% | | ParquetLoaderBench | bench_load_10k | 1 | 3 | 227.013mb +0.00% | 1.408s -1.53% | ±0.92% +140.02% | | TextLoaderBench | bench_load_10k | 1 | 3 | 17.976mb +0.00% | 38.409ms -1.51% | ±0.55% +192.93% | +--------------------+----------------+------+-----+------------------+------------------+-----------------+ ```
Building Blocks ```shell +-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+ | RowsBench | bench_chunk_10_on_10k | 2 | 3 | 87.060mb +0.00% | 3.306ms -16.46% | ±3.55% +29.11% | | RowsBench | bench_diff_left_1k_on_10k | 2 | 3 | 102.658mb +0.00% | 187.879ms -1.61% | ±0.53% -49.10% | | RowsBench | bench_diff_right_1k_on_10k | 2 | 3 | 85.378mb +0.00% | 18.917ms -3.25% | ±0.02% -98.96% | | RowsBench | bench_drop_1k_on_10k | 2 | 3 | 88.300mb +0.00% | 1.746ms -7.44% | ±1.35% +169.12% | | RowsBench | bench_drop_right_1k_on_10k | 2 | 3 | 88.300mb +0.00% | 1.725ms -9.69% | ±3.22% +13.20% | | RowsBench | bench_entries_on_10k | 2 | 3 | 85.412mb +0.00% | 2.475ms -14.20% | ±2.40% +67.36% | | RowsBench | bench_filter_on_10k | 2 | 3 | 85.941mb +0.00% | 16.011ms -5.23% | ±2.12% +849.91% | | RowsBench | bench_find_on_10k | 2 | 3 | 85.941mb +0.00% | 15.648ms -6.12% | ±0.54% -72.86% | | RowsBench | bench_find_one_on_10k | 10 | 3 | 83.846mb +0.00% | 1.794μs -10.30% | ±2.67% +0.00% | | RowsBench | bench_first_on_10k | 10 | 3 | 83.846mb +0.00% | 0.400μs 0.00% | ±0.00% 0.00% | | RowsBench | bench_flat_map_on_1k | 2 | 3 | 93.195mb +0.00% | 12.445ms -8.82% | ±0.43% -72.84% | | RowsBench | bench_map_on_10k | 2 | 3 | 122.566mb +0.00% | 60.195ms -4.97% | ±0.19% -63.73% | | RowsBench | bench_merge_1k_on_10k | 2 | 3 | 86.461mb +0.00% | 1.506ms -3.03% | ±0.62% -62.71% | | RowsBench | bench_partition_by_on_10k | 2 | 3 | 89.808mb +0.00% | 61.096ms -7.81% | ±0.83% -40.41% | | RowsBench | bench_remove_on_10k | 2 | 3 | 88.562mb +0.00% | 3.833ms -8.99% | ±2.84% +175.82% | | RowsBench | bench_sort_asc_on_1k | 2 | 3 | 83.989mb +0.00% | 39.480ms -1.59% | ±1.21% -57.80% | | RowsBench | bench_sort_by_on_1k | 2 | 3 | 83.990mb +0.00% | 40.177ms -0.50% | ±1.65% +71.05% | | RowsBench | bench_sort_desc_on_1k | 2 | 3 | 83.989mb +0.00% | 40.020ms +1.32% | ±1.70% +5.71% | | RowsBench | bench_sort_entries_on_1k | 2 | 3 | 86.287mb +0.00% | 7.401ms +0.85% | ±0.53% +269.48% | | RowsBench | bench_sort_on_1k | 2 | 3 | 83.846mb +0.00% | 29.154ms +2.52% | ±1.42% +43.82% | | RowsBench | bench_take_1k_on_10k | 10 | 3 | 83.846mb +0.00% | 13.721μs +0.64% | ±0.91% -39.75% | | RowsBench | bench_take_right_1k_on_10k | 10 | 3 | 83.846mb +0.00% | 15.880μs -0.49% | ±1.06% -31.43% | | RowsBench | bench_unique_on_1k | 2 | 3 | 102.660mb +0.00% | 190.693ms -1.64% | ±0.63% -3.46% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 116.738mb +0.00% | 523.532ms +1.78% | ±0.97% +80.73% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 60.216mb +0.00% | 258.044ms -0.08% | ±0.55% -62.38% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 15.150mb +0.00% | 55.627ms +1.56% | ±0.15% -77.39% | | TypeDetectorBench | bench_type_detector | 1 | 3 | 59.970mb +0.00% | 434.383ms -0.81% | ±0.24% -70.67% | | TypeDetectorBench | bench_type_detector | 1 | 3 | 14.509mb +0.00% | 86.445ms +0.10% | ±0.40% -64.91% | +-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+ ```