search

eclipse-edc / Technology-Aws

Apache License 2.0
5 stars 15 forks source link

File transfer from MinIO to MinIO: Provider always tries to push the data into its own storage #10

Closed Julian-Marco closed 1 year ago

Julian-Marco commented 1 year ago

Bug Report

Describe the Bug

When using the S3-core and the S3-Dataplane-extensions in combination with the endpoint.override functionality in order to work with two different MinIO Storages, it happens that the provider edc always tries to push the file into its own MinIO Storage instead of the MinIO Storage of the consumer, no matter what destination is given in the filetransfer.json. For example, when the provider edc uses a config properties with

edc.aws.endpoint.override=http://localhost:9000

and the consumer edc uses a config properties with

edc.aws.endpoint.override=http://localhost:10000,

and the datadestination in the filetransfer.json looks like this,

  "dataDestination": {
    "@type": "DataAddress",
        "properties": {
            "type": "AmazonS3",
            "region": "us-east-1",
            "bucketName": "bucketname",
            "keyName": "fileName",
            "endpoint": "http://localhost:10000"
        }
    },

then the provider edc still tries to transfer the file to its own MinIO Storage that gets hosted on localhost:9000.

Expected Behavior

When using two different MinIO Storages, the provider EDC should be able to distinguish between its own storage and the storage of the provider.

Context Information

Is there any way to make this work as expected without modifying the base s3 code? I know that this problem has been known for older edc version too. https://github.com/eclipse-edc/Technology-Aws/issues/24 If I understand this correctly, it might work if we'd use pull instead of push? But the only example for actually using the pull mechanism doesn't look like it's useful for working with S3, does it? https://github.com/eclipse-edc/Samples/tree/main/transfer/transfer-06-consumer-pull-http

Any help would be appreciated!

github-actions[bot] commented 1 year ago

Thanks for your contribution :fire: We will take a look asap :rocket:

github-actions[bot] commented 1 year ago

Thanks for your contribution :fire: We will take a look asap :rocket:

github-actions[bot] commented 1 year ago

This issue is stale because it has been open for 28 days with no activity.

github-actions[bot] commented 1 year ago

This issue was closed because it has been inactive for 14 days since being marked as stale.

paullatzelsperger commented 1 year ago

somewhat related PR: https://github.com/eclipse-edc/Connector/pull/1904. Revisiting it and porting it here may be useful.

maciejkizlich-zf commented 1 year ago

@paullatzelsperger @reisman234 @Julian-Marco @ndr-brt

https://github.com/eclipse-edc/Technology-Aws/pull/68 I have recreated the PR (draft) in Technology-Aws, in case original author would no longer be interested in fixing this. Please take a look, and let me know if/how can we proceed.

ndr-brt commented 1 year ago

This should have been fixed by #68