datafuselabs / databend

๐——๐—ฎ๐˜๐—ฎ, ๐—”๐—ป๐—ฎ๐—น๐˜†๐˜๐—ถ๐—ฐ๐˜€ & ๐—”๐—œ. Modern alternative to Snowflake. Cost-effective and simple for massive-scale analytics. https://databend.com
https://docs.databend.com
Other
7.29k stars 701 forks source link

refactor: remove stage use stream to avoid OOM. #15378

Closed youngsofun closed 2 weeks ago

youngsofun commented 2 weeks ago

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

remove stage use stream instead of load all paths into one Vec to avoid OOM.

00_0004_copy_with_max_files.sh is changed too, no longer rely on the order of list of backends . cc @Xuanwo

Tests

Type of change


This change isโ€‚Reviewable