[AMORO-3301] Support OSS for iceberg in InternalCatalog

shouwangyw commented 3 weeks ago

Why are the changes needed?

Close #3301.

Brief change log

Add OSS storage type, support OSS for iceberg in InternalCatalog

How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
[ ] Add screenshots for manual tests if appropriate
[ ] Run test locally before making a pull request

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

czy006 commented 3 weeks ago

Thank you for your contribution. As far as I know, the current S3 protocol can also use Ali Cloud OSS object storage. Is iceberg storage support added here? By the way, we should make OSS optional at compile time

shouwangyw commented 3 weeks ago

Thank you for your contribution. As far as I know, the current S3 protocol can also use Ali Cloud OSS object storage. Is iceberg storage support added here? By the way, we should make OSS optional at compile time

I don't understand how to use Ali Cloud OSS object storage with the S3 protocol？ This PR has added Ali Cloud OSS related dependencies

shouwangyw commented 3 weeks ago

OSS support can be correctly added through the front end

I have completed the test in my local environment, able to obtain relevant data correctly

czy006 commented 2 weeks ago

you can use s3 protocol to read it

cc @shouwangyw

czy006 commented 2 weeks ago

CI is Failing

shouwangyw commented 2 weeks ago

you can use s3 protocol to read it

cc @czy006

But this might not be intuitive

shouwangyw commented 2 weeks ago

CI is Failing

I have fixed the ci

czy006 commented 2 weeks ago

you can use s3 protocol to read it cc @czy006

But this might not be intuitive

Of course this is a compromise. I would like to ask if we can have a compilation switch to compile the OSS part when we needed? As object storage support increases, compiling all by default will increase Amoro's binary installation package

shouwangyw commented 2 weeks ago

you can use s3 protocol to read it cc @czy006

But this might not be intuitive

Of course this is a compromise. I would like to ask if we can have a compilation switch to compile the OSS part when we needed? As object storage support increases, compiling all by default will increase Amoro's binary installation package

yes, I can choose whether to package aliyun sdk by scope

codecov-commenter commented 2 weeks ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 22.54%. Comparing base (f13f3f3) to head (99e0eba). Report is 7 commits behind head on master.

:exclamation: There is a different number of reports uploaded between BASE (f13f3f3) and HEAD (99e0eba). Click for more details.

HEAD has 1 upload less than BASE
| Flag | BASE (f13f3f3) | HEAD (99e0eba) | |------|------|------| |core|1|0|

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #3306 +/- ## ============================================ - Coverage 30.17% 22.54% -7.63% + Complexity 3840 2312 -1528 ============================================ Files 580 397 -183 Lines 48020 38201 -9819 Branches 6207 5437 -770 ============================================ - Hits 14488 8611 -5877 + Misses 32542 28863 -3679 + Partials 990 727 -263 ``` | [Flag](https://app.codecov.io/gh/apache/amoro/pull/3306/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | |---|---|---| | [core](https://app.codecov.io/gh/apache/amoro/pull/3306/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | | [trino](https://app.codecov.io/gh/apache/amoro/pull/3306/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `22.54% <ø> (?)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

apache / amoro