flow-php / flow

Flow PHP - data processing framework
https://flow-php.com
MIT License
404 stars 23 forks source link

Fixed writing nullable parquet values that are marked in the schema as optional #1025

Closed norberttech closed 3 months ago

norberttech commented 3 months ago

Change Log

Added

Fixed

  • writing nullable parquet values that are marked in the schema as optional

Changed

  • updated dependencies

Removed

Deprecated

Security


Description

github-actions[bot] commented 3 months ago

Flow PHP - Benchmarks

Results of the benchmarks from this PR are compared with the results from 1.x branch.

Extractors ```shell +-----------------------+-------------------+------+-----+------------------+------------------+----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-----------------------+-------------------+------+-----+------------------+------------------+----------------+ | AvroExtractorBench | bench_extract_10k | 1 | 3 | 35.280mb +0.00% | 807.948ms -2.12% | ±0.44% +6.29% | | CSVExtractorBench | bench_extract_10k | 1 | 3 | 5.003mb +0.01% | 345.004ms +0.81% | ±0.64% +58.26% | | JsonExtractorBench | bench_extract_10k | 1 | 3 | 5.153mb +0.00% | 1.048s +0.32% | ±0.13% -79.72% | | ParquetExtractorBench | bench_extract_10k | 1 | 3 | 135.825mb +0.00% | 903.630ms -0.69% | ±0.26% -83.53% | | TextExtractorBench | bench_extract_10k | 1 | 3 | 4.910mb +0.01% | 39.058ms +10.47% | ±2.26% +71.56% | | XmlExtractorBench | bench_extract_10k | 1 | 3 | 4.916mb +0.01% | 431.263ms -1.01% | ±0.42% -37.99% | +-----------------------+-------------------+------+-----+------------------+------------------+----------------+ ```
Transformers ```shell +-----------------------------+--------------------------+------+-----+------------------+-----------------+----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-----------------------------+--------------------------+------+-----+------------------+-----------------+----------------+ | RenameEntryTransformerBench | bench_transform_10k_rows | 1 | 3 | 110.616mb +0.00% | 64.192ms -0.28% | ±0.66% -15.72% | +-----------------------------+--------------------------+------+-----+------------------+-----------------+----------------+ ```
Loaders ```shell +--------------------+----------------+------+-----+------------------+------------------+----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +--------------------+----------------+------+-----+------------------+------------------+----------------+ | AvroLoaderBench | bench_load_10k | 1 | 3 | 95.660mb +0.00% | 470.798ms +1.15% | ±0.05% -93.39% | | CSVLoaderBench | bench_load_10k | 1 | 3 | 54.142mb +0.00% | 71.440ms -3.47% | ±0.43% -40.66% | | JsonLoaderBench | bench_load_10k | 1 | 3 | 106.567mb +0.00% | 51.669ms -0.74% | ±0.69% -9.68% | | ParquetLoaderBench | bench_load_10k | 1 | 3 | 224.388mb +0.00% | 1.440s +1.69% | ±0.74% -65.58% | | TextLoaderBench | bench_load_10k | 1 | 3 | 17.957mb +0.00% | 40.183ms +0.97% | ±0.30% -54.94% | +--------------------+----------------+------+-----+------------------+------------------+----------------+ ```
Building Blocks ```shell +-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+ | benchmark | subject | revs | its | mem_peak | mode | rstdev | +-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+ | RowsBench | bench_chunk_10_on_10k | 2 | 3 | 76.683mb +0.00% | 3.400ms +0.71% | ±2.55% +67.99% | | RowsBench | bench_diff_left_1k_on_10k | 2 | 3 | 96.410mb +0.00% | 182.939ms +1.09% | ±1.17% +89.91% | | RowsBench | bench_diff_right_1k_on_10k | 2 | 3 | 74.936mb +0.00% | 18.458ms +2.47% | ±0.34% -72.32% | | RowsBench | bench_drop_1k_on_10k | 2 | 3 | 77.923mb +0.00% | 1.697ms -1.49% | ±1.18% -59.21% | | RowsBench | bench_drop_right_1k_on_10k | 2 | 3 | 77.923mb +0.00% | 1.887ms +11.57% | ±2.32% +96.40% | | RowsBench | bench_entries_on_10k | 2 | 3 | 75.035mb +0.00% | 2.603ms +1.69% | ±0.94% -41.35% | | RowsBench | bench_filter_on_10k | 2 | 3 | 75.564mb +0.00% | 15.174ms +2.73% | ±1.88% +79.00% | | RowsBench | bench_find_on_10k | 2 | 3 | 75.564mb +0.00% | 15.207ms +5.03% | ±1.06% -52.55% | | RowsBench | bench_find_one_on_10k | 10 | 3 | 73.468mb +0.00% | 1.794μs +5.15% | ±2.67% -1.89% | | RowsBench | bench_first_on_10k | 10 | 3 | 73.468mb +0.00% | 0.400μs +33.33% | ±0.00% +0.00% | | RowsBench | bench_flat_map_on_1k | 2 | 3 | 87.023mb +0.00% | 13.273ms +3.02% | ±2.41% -31.72% | | RowsBench | bench_map_on_10k | 2 | 3 | 116.383mb +0.00% | 66.412ms +3.96% | ±0.49% -54.38% | | RowsBench | bench_merge_1k_on_10k | 2 | 3 | 76.084mb +0.00% | 1.382ms +11.84% | ±3.22% +2.54% | | RowsBench | bench_partition_by_on_10k | 2 | 3 | 79.431mb +0.00% | 59.683ms +2.09% | ±2.84% +755.43% | | RowsBench | bench_remove_on_10k | 2 | 3 | 78.185mb +0.00% | 3.860ms -9.19% | ±2.45% +18.01% | | RowsBench | bench_sort_asc_on_1k | 2 | 3 | 73.546mb +0.00% | 40.968ms +1.00% | ±0.38% -74.69% | | RowsBench | bench_sort_by_on_1k | 2 | 3 | 73.547mb +0.00% | 42.729ms +4.92% | ±2.24% +108.97% | | RowsBench | bench_sort_desc_on_1k | 2 | 3 | 73.546mb +0.00% | 41.048ms +1.11% | ±2.18% +229.16% | | RowsBench | bench_sort_entries_on_1k | 2 | 3 | 75.909mb +0.00% | 7.447ms +1.89% | ±1.26% -35.98% | | RowsBench | bench_sort_on_1k | 2 | 3 | 73.468mb +0.00% | 29.404ms -1.94% | ±0.30% -85.52% | | RowsBench | bench_take_1k_on_10k | 10 | 3 | 73.468mb +0.00% | 13.312μs -2.88% | ±0.71% -69.12% | | RowsBench | bench_take_right_1k_on_10k | 10 | 3 | 73.468mb +0.00% | 15.800μs +0.04% | ±0.52% -73.31% | | RowsBench | bench_unique_on_1k | 2 | 3 | 96.477mb +0.00% | 186.186ms -0.42% | ±0.88% +11.87% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 116.715mb +0.00% | 499.216ms +0.66% | ±1.65% +24.03% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 60.193mb +0.00% | 246.117ms -0.85% | ±0.67% -30.50% | | NativeEntryFactoryBench | bench_entry_factory | 1 | 3 | 15.128mb +0.00% | 52.898ms +1.45% | ±1.84% -17.91% | | TypeDetectorBench | bench_type_detector | 1 | 3 | 59.959mb +0.00% | 434.770ms +0.27% | ±0.35% -42.72% | | TypeDetectorBench | bench_type_detector | 1 | 3 | 14.498mb +0.00% | 85.823ms +0.30% | ±0.59% -63.64% | +-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+ ```