EDC - Transfer of data between S3 buckets or between Azure Blob stores

[lgblaumeiser]

Overview

Explain the topic in 2 sentences

Currently, data transfer, even for big data goes through the connector data plane. For transmissions between two S3 Buckets, this is not efficient and especially not cost efficient, especially if the transfer is about TB or even PB. The issue is about optimizing that by using means of S3 to directly transfer the data.

What's the benefit?

Less cost for data transfer between S3 buckets

What are the Risks/Dependencies ?

No risks

Depends on features within AWS to transfer data within the AWS cloud

Detailed explanation

Current implementation

All data is processed through the data plane, which means in case of a Azure based deployment for AWS S3 data transfer, the data is transfered from AWS to Azure back to AWS which is extremely expensive.

Proposed improvements

The targeted solution provides direct usage of mechanisms within AWS S3 that allow to transfer data. The dataplane implementation will in case of a suited transfer request make use of the mechanism so that the data does not flow through the dataplane but is exchanged directly within S3.

Feature Team

Contributor

• Cofinity-X

Committer

• Andrea Bertagnolli

User Stories

• tbd

Acceptance Criteria

• [ ] In case of a suited data transfer, e.g., from a S3 bucket to another S3 bucket, the data is transfered using a S3 mechanism
• [ ] Proper test is provided

Test Cases

Test Case 1

Run a S§ data transfer within the same region between two different companies which provide the S3 bucket that on the provider side stores the data to be transfered and on the consumer side provides the place to put the data.

Steps

1. Create an asset that offers a data item in a S3 bucket on the provider side
2. Negotiate a contract initiated from the consumer to push the data item to the consumer bucket
3. Initiate the data transfer after the contract is established

Expected Result

1. The data is transfered from the provider S3 bucket to the consumer S3 bucket
2. The data transfer was done within the AWS S3 system without data transfer to the outside

Architectural Relevance

The following items are ensured (answer: yes) after this issue is implemented:

• [X] This feature aligns with our current architectural guidelines
  • Data Sovereignty: All data sharing activities across company boundaries follow the Catena-X Regulatory Framework, in particular the Data Exchange Governance, and the Dataspace Protocol via a compliant Connector (like the tractusx-edc or similar, see Connector KIT)
  • [X] CX-0010 Business Partner Number
  • [X] CX-0013 Identity of Member Companies
  • [X] CX-0018 Data Space Connectivity (EDC)
  • [X] CX-0049 DID Document Schema
  • [X] CX-0050 Framework Agreement Credential
  • [X] CX-0149 Verified Company Identity
  • Interoperability: Digital Twins are used (compliant to the Digital Twin KIT and the Industry Core KIT)
  • [X] CX-0001 EDC Discovery API
  • [X] CX-0002 Digital Twins in Catena-X
  • [X] CX-0018 Data Space Connectivity (EDC)
  • Data Format:
  • [X] The data model is based on a published Semantic Model
• [X] The impact on the overall system architecture has been assessed. The Feature does not require changes to the architecture or any existing standard? Please have a look here on the overarching architecture
• [X] Potential risks or conflicts with existing architecture has been assessed

Justification: (Fill this out, if at least one of the checkboxes above cannot be ticked. Contact the Architecture Management Committee to get an approval for the justification) n/a

Additional information

• [X] I am aware that my request may not be developed if no developer can be found for it. I'll try to contribute a developer (bring your own developer)

[stephanbcbauer]

Presented in the DRAFT Feature Freeze -> Committer is available

[rafaelmag110]

This is a good idea. There is a similar discussion open in one of the upstream repositories: https://github.com/eclipse-edc/Technology-Aws/discussions/270

We wanted to contribute with this improvement but couldn't crack a particular problem of permission management. I'll take some time to add more details to that discussion to kick start it again.

[lgblaumeiser]

PoC work ongoing, a final implementation will not make it into this release

[lgblaumeiser]

Spill over from 24/12 release planning