search

thanos-io / thanos

Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.
https://thanos.io
Apache License 2.0
13.15k stars 2.1k forks source link

s3/aliyunOSS: Support multipart upload without pre known object size. #678

Open bwplotka opened 5 years ago

bwplotka commented 5 years ago

Discussion for Reference https://github.com/improbable-eng/thanos/pull/617

Acceptance Critiria:

bwplotka commented 5 years ago

Or better: https://github.com/thanos-io/thanos/issues/443

bwplotka commented 5 years ago

The same story is for new provider AliYun. We should either implement multipart on our own or extend objstore interface: https://github.com/thanos-io/thanos/pull/1573/files#r336389276

bwplotka commented 5 years ago

We have a solid use case for this issue to be fixed: https://github.com/observatorium/thanos-replicate/issues/22

Help wanted!

daixiang0 commented 4 years ago

@bwplotka could move thanos-replicate under thanos-io to increase its exposure and make it look more important for thaons?

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

yeya24 commented 4 years ago

This issue is still valid, right?

bwplotka commented 4 years ago

yup :+1:

jojohappy commented 4 years ago

Hi, @bwplotka @adrien-f I see the minio-go has been fixed the bug to upload large file without the size of object. I try to upload a 5.7GB file to my S3 with minio-go client v6.0.49, it works. Maybe we can remove the changes by #617.

For aliyun oss, we can use the resumable upload without known object size. cc @wujinhu

What do you think?

stale[bot] commented 4 years ago

This issue/PR has been automatically marked as stale because it has not had recent activity. Please comment on status otherwise the issue will be closed in a week. Thank you for your contributions.

stale[bot] commented 4 years ago

This issue/PR has been automatically marked as stale because it has not had recent activity. Please comment on status otherwise the issue will be closed in a week. Thank you for your contributions.

stale[bot] commented 4 years ago

Closing for now as promised, let us know if you need this to be reopened! 🤗

stale[bot] commented 4 years ago

Hello 👋 Looks like there was no activity on this issue for last 30 days. Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity for next week, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

stale[bot] commented 4 years ago

Closing for now as promised, let us know if you need this to be reopened! 🤗

xiaozongyang commented 4 years ago

Hi @wujinhu, it seems that this issue hasn't been fixed yet. Do we have any plans for that? Is there some work that I can do for solving it?

We use aliyun OSS in our production environment as Thanos bucket store. But we failed to do downsampling jobs because we always get annoying errors unsupported type of io.Reader.

And I find there is a todo comment referring to this issue in line 72 in the source file oss.go.

stale[bot] commented 2 years ago

Hello 👋 Looks like there was no activity on this issue for the last two months. Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

bdharsan04 commented 5 months ago

@yeya24 i would like to look into this issue if it's still open