[DSIP-45] Polish the Storage SPI

ruanwenjun commented 3 months ago

Purpose of the pull request

close https://github.com/apache/dolphinscheduler/issues/16140 close https://github.com/apache/dolphinscheduler/issues/16072

Brief change log

Refactor the storage api
Refactor the storage service
Add IT for LocalStorage/HdfsStorage/S3Storage
Remove some mock UT, this is meaningless.
Fix fetch file content and update file content will cut 3000 lines.

Verify this pull request

Verify by e2e(s3 case) / IT(local hdfs/ s3 case) and manual test.

Resource CRUD

UDF CRUD

Using in task

codecov-commenter commented 3 months ago

Codecov Report

Attention: Patch coverage is 41.84477% with 517 lines in your changes missing coverage. Please review.

Project coverage is 41.07%. Comparing base (61a6bd0) to head (12b3efd).

:exclamation: Current head 12b3efd differs from pull request most recent head 82a6f2d

Please upload reports for the commit 82a6f2d to get more accurate results.

Files	Patch %	Lines
...heduler/api/service/impl/ResourcesServiceImpl.java	0.00%	147 Missing :warning:
...nscheduler/api/controller/ResourcesController.java	50.57%	43 Missing :warning:
...scheduler/plugin/storage/s3/S3StorageOperator.java	77.61%	25 Missing and 5 partials :warning:
...pache/dolphinscheduler/common/utils/FileUtils.java	7.40%	25 Missing :warning:
...or/resource/CreateDirectoryRequestTransformer.java	4.16%	23 Missing :warning:
...resource/PagingResourceItemRequestTransformer.java	0.00%	23 Missing :warning:
...duler/plugin/storage/hdfs/HdfsStorageOperator.java	79.79%	18 Missing and 2 partials :warning:
...or/resource/FileFromContentRequestTransformer.java	9.52%	19 Missing :warning:
...api/validator/resource/FileRequestTransformer.java	9.52%	19 Missing :warning:
...apache/dolphinscheduler/api/vo/ResourceItemVO.java	0.00%	14 Missing :warning:
... and 28 more

Additional details and impacted files

```diff @@ Coverage Diff @@ ## dev #16141 +/- ## ============================================ + Coverage 40.70% 41.07% +0.36% + Complexity 5245 5155 -90 ============================================ Files 1385 1399 +14 Lines 46109 44856 -1253 Branches 4923 4756 -167 ============================================ - Hits 18768 18423 -345 + Misses 25414 24628 -786 + Partials 1927 1805 -122 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

ruanwenjun commented 3 months ago