search

eclipse-tractusx / sig-release

https://eclipse-tractusx.github.io/sig-release
Apache License 2.0
8 stars 8 forks source link

EDC - Transfer of data between S3 buckets or between Azure Blob stores #755

Open lgblaumeiser opened 1 month ago

lgblaumeiser commented 1 month ago

Description

Currently, data transfer, even for big data goes through the connector data plane. For transmissions between two S3 Buckets or two Azure Blob Stores, this is not efficient and especially not cost efficient, especially if the transfer is about TB or even PB.

Possible solution: Find a way of copying the Data as near as possible to the source/destinations (e.g. within the AWS infrastructure)

Requestor: @florianrusch-zf

Impact

No impact observable from outside except for a smaller cloud invoice, behavior of data exchange does not change.

Additional Information

Contribution planned by: Cofinity-X

stephanbcbauer commented 1 month ago

Presented in the DRAFT Feature Freeze -> Committer is available

rafaelmag110 commented 1 month ago

This is a good idea. There is a similar discussion open in one of the upstream repositories: https://github.com/eclipse-edc/Technology-Aws/discussions/270

We wanted to contribute with this improvement but couldn't crack a particular problem of permission management. I'll take some time to add more details to that discussion to kick start it again.