apache / doris

Apache Doris is an easy-to-use, high performance and unified analytics database.
https://doris.apache.org
Apache License 2.0
12.61k stars 3.25k forks source link

export a big table to S3, but the job cannot finished until query timeout[Bug] #23153

Open xufenxu opened 1 year ago

xufenxu commented 1 year ago

Search before asking

Version

2.0.0rc4

What's Wrong?

export a big size table into s3, the export job status finally show ‘cancelled’, but in fact the file is total exported into the s3 bucket.

What You Expected?

The table lineorder_felat is generated by the ssb-tools, about 60G size. and I see the data is all exported to the s3 bucket object, generated 198 files, one file size is about 1G, total size about 190G, but the job state is always 'PENDING' util the query timeout. The finally status of the job is 'CANCELLED'.

How to Reproduce?

1.set query_timeout=30000; 2.export table lineorder_flat to "s3://{bucket_name}/export_path/export" with s3( "AWS_ENDPOINT"="", "AWS_ACCESS_KEY"="", "AWS_SECRET_KEY"="", "AWS_REGION"="", "use_path_style"="true" );

Anything Else?

No response

Are you willing to submit PR?

Code of Conduct

ixzc commented 1 year ago

You'd better according to the data partition export and set up the bigger parallelism.

morningman commented 1 year ago

The entire export feature is being refactoring now, and will be release in 2.0.2 or 2.1