Closed BohuTANG closed 2 weeks ago
docs need to be updated
COPY INTO ensures idempotence by automatically tracking and preventing the reloading of files for a default period of 7 days. This can be customized using the load_file_metadata_expire_hours setting to control the expiration time for file metadata.
This parameter defaults to False meaning COPY INTO will skip duplicate files when copying data. If True, duplicate files will not be skipped.
docs need to be updated
COPY INTO ensures idempotence by automatically tracking and preventing the reloading of files for a default period of 7 days. This can be customized using the load_file_metadata_expire_hours setting to control the expiration time for file metadata. This parameter defaults to False meaning COPY INTO will skip duplicate files when copying data. If True, duplicate files will not be skipped.
~Will update after this PR merged.~ PR: https://github.com/datafuselabs/databend-docs/pull/788
("load_file_metadata_expire_hours", DefaultSettingValue {
value: UserSettingValue::UInt64(24),
desc: "Sets the hours that the metadata of files you load data from with COPY INTO will expire in.",
mode: SettingMode::Both,
range: Some(SettingRange::Numeric(0..=u64::MAX)),
}),
The value means hours. So the range's max use u64::MAX that may not be very reasonable
I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/
Summary
load_file_metadata_expire_hours
now is 12 hoursFor optimal COPY operations, it is recommended to use the following syntax:
COPY INTO <table> FROM FILES = ('file1', 'file2')
, no need s3 list. https://docs.databend.com/sql/sql-commands/dml/dml-copy-into-table#filesFixes #15500
Tests
Type of change
This change isโ