search

EMCECS / ecs-sync

ecs-sync is a bulk copy utility that can move data between various systems in parallel
Apache License 2.0
60 stars 22 forks source link

com.emc.object.s3.S3Exception: Forbidden - Migrating ECS DD Cloudtier Bucket #71

Closed leyita closed 3 years ago

leyita commented 3 years ago

All Bucket are constructed with folder structure starting with folder "/". If try to run a migration will fail with 2020-08-03 13:24:38 WARN [sync-pool-5-t-1] SyncTask: O--! object /Folder1/file1 failed com.emc.object.s3.S3Exception: Forbidden at com.emc.object.s3.jersey.ErrorFilter.handle(ErrorFilter.java:80) at com.emc.object.s3.jersey.RetryFilter.handle(RetryFilter.java:65) at com.emc.object.s3.jersey.ChecksumFilter.handle(ChecksumFilter.java:73) at com.emc.object.s3.jersey.AuthorizationFilter.handle(AuthorizationFilter.java:72) at com.emc.object.s3.jersey.BucketFilter.handle(BucketFilter.java:80) at com.emc.object.s3.jersey.NamespaceFilter.handle(NamespaceFilter.java:78) at com.sun.jersey.api.client.Client.handle(Client.java:652) at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682) at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) at com.sun.jersey.api.client.WebResource$Builder.method(WebResource.java:623) at com.emc.object.AbstractJerseyClient.executeRequest(AbstractJerseyClient.java:107) at com.emc.object.AbstractJerseyClient.executeAndClose(AbstractJerseyClient.java:54) at com.emc.object.s3.jersey.S3JerseyClient.getObjectMetadata(S3JerseyClient.java:646) at com.emc.ecs.sync.storage.s3.EcsS3Storage.lambda$getS3Metadata$11(EcsS3Storage.java:514) at com.emc.ecs.sync.util.TimingUtil.time(TimingUtil.java:55) at com.emc.ecs.sync.AbstractPlugin.time(AbstractPlugin.java:72) at com.emc.ecs.sync.storage.s3.EcsS3Storage.getS3Metadata(EcsS3Storage.java:514) at com.emc.ecs.sync.storage.s3.EcsS3Storage.loadObject(EcsS3Storage.java:240) at com.emc.ecs.sync.storage.s3.AbstractS3Storage.loadObject(AbstractS3Storage.java:63) at com.emc.ecs.sync.storage.s3.EcsS3Storage.loadObject(EcsS3Storage.java:231) at com.emc.ecs.sync.SyncTask.run(SyncTask.java:72) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) when browsing the bucket using s3browser, the first folder is "/". any ideas how to make the migration works?

jrosenth commented 3 years ago

Is the directory name actually "/" ? Can you provide examples of full paths to files (Objects).

There is a recently discovered issue with migrating paths that contain consecutive/multiple "/"'s, like "foo/bar//file.ext".

As a workaround you can try disabling (turn off) the smart client setting in both the source and target. Be aware that this also disables ECS-Sync's built in load balancer. All ECS-Sync requests may go to single ECS node. I suggest only using the workaround for retrying failed objects not for an entire large migration.

leyita commented 3 years ago

i will try the workaround but it is not a small migration (300M objects 1.2PB) and the problem is from the beginning because the first folder is the "/" for instance http://ip1:9020/bucket//folder1/object1. the extra "/" is between the bucket and the first folder. if you connect with s3browser and select the bucket, the first level path is "/". i will post examples.

leyita commented 3 years ago

i tried with the smart client disable and it worked. can you please explain why??. From the debug log i cannot detect the difference, they go thru the same steps but when is run with the smartclient i received the same com.emc.object.s3.S3Exception: Forbidden.

2020-08-04 19:02:07 DEBUG [sync-pool-9-t-1] ObjectConfig: raw path & query: //EncryptionKeys/encrypted_deks?null 2020-08-04 19:02:07 DEBUG [sync-pool-9-t-1] ObjectConfig: resolved URI: http://10.101.41.11:9020/%2FEncryptionKeys/encrypted_deks

jrosenth commented 3 years ago

I don't have a complete answer for "why" as the issue is still under investigation. The high level answer is that the path is being improperly escapified/encoded before being sent to ECS. ECS attempts to verify signature of the HTTP headers and fails because the original signature used the correct path however ECS uses the wrong path which results in the "Forbidden" error back to the client. If you look in the ECS dataheadsvc logs you will see an error containing the text "Signature mismatch CalcSignature".

The code path of not going through smart client appears to avoid the problem. As I said the issue is still under investigation.

You are in a bit of bind if all your paths start off like that.

Gabby-ScaleIO commented 3 years ago

Hi Please remove me from this DL Thanks

On Wed, Aug 5, 2020, 18:15 Jay Rosenthal notifications@github.com wrote:

I don't have a complete answer for "why" as the issue is still under investigation. The high level answer is that the path is being improperly escapified/encoded before being sent to ECS. ECS attempts to verify signature of the HTTP headers and fails because the original signature used the correct path however ECS uses the wrong path which results in the "Forbidden" error back to the client. If you look in the ECS dataheadsvc logs you will see an error containing the text "Signature mismatch CalcSignature".

The code path of not going through smart client appears to avoid the problem. As I said the issue is still under investigation.

You are in a bit of bind if all your paths start off like that.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/EMCECS/ecs-sync/issues/71#issuecomment-669252967, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4X2XEDPEM7WRBKZRNKXCDR7FZQXANCNFSM4PTXQU7A .

jasoncwik commented 3 years ago

You have to login to your GitHub account and unstar the repository.

Sent from my iPhone

On Aug 12, 2020, at 4:48 AM, Gabby.Koren_ScaleIO notifications@github.com wrote:



Hi Please remove me from this DL Thanks

On Wed, Aug 5, 2020, 18:15 Jay Rosenthal notifications@github.com wrote:

I don't have a complete answer for "why" as the issue is still under investigation. The high level answer is that the path is being improperly escapified/encoded before being sent to ECS. ECS attempts to verify signature of the HTTP headers and fails because the original signature used the correct path however ECS uses the wrong path which results in the "Forbidden" error back to the client. If you look in the ECS dataheadsvc logs you will see an error containing the text "Signature mismatch CalcSignature".

The code path of not going through smart client appears to avoid the problem. As I said the issue is still under investigation.

You are in a bit of bind if all your paths start off like that.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/EMCECS/ecs-sync/issues/71#issuecomment-669252967, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4X2XEDPEM7WRBKZRNKXCDR7FZQXANCNFSM4PTXQU7A .

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/EMCECS/ecs-sync/issues/71#issuecomment-672772253, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAG2ZGGU5PVSKLOIMNTOQCLSAJQOXANCNFSM4PTXQU7A.

twincitiesguy commented 3 years ago

closing as resolved