search

gaul / s3proxy

Access other storage backends via the S3 API
Apache License 2.0
1.75k stars 228 forks source link

Issues connecting to Azure Blob Storage #262

Closed RyanMorash closed 2 weeks ago

RyanMorash commented 6 years ago

I'm having trouble connecting to Azure Blob Storage as a backend. To try and troubleshoot I set the s3proxy auth to 'none' and ran 'curl http://s3proxy/container' and got a HTTP error 400. These were returned in the debug logs:

2018-02-19T07:12:49.238722860Z app[web.1]: [s3proxy] D 02-19 07:12:49.238 S3Proxy-Jetty-13 o.gaul.s3proxy.S3ProxyHandler:280 |::] request: Request(GET /container)@188a433d
2018-02-19T07:12:49.238912464Z app[web.1]: [s3proxy] D 02-19 07:12:49.238 S3Proxy-Jetty-13 o.gaul.s3proxy.S3ProxyHandler:302 |::] header: Host: s3proxy
2018-02-19T07:12:49.239123267Z app[web.1]: [s3proxy] D 02-19 07:12:49.238 S3Proxy-Jetty-13 o.gaul.s3proxy.S3ProxyHandler:302 |::] header: X-Forwarded-Proto: http
2018-02-19T07:12:49.239333571Z app[web.1]: [s3proxy] D 02-19 07:12:49.239 S3Proxy-Jetty-13 o.gaul.s3proxy.S3ProxyHandler:302 |::] header: X-Forwarded-For: 8.8.8.8
2018-02-19T07:12:49.239541374Z app[web.1]: [s3proxy] D 02-19 07:12:49.239 S3Proxy-Jetty-13 o.gaul.s3proxy.S3ProxyHandler:302 |::] header: User-Agent: curl/7.54.0
2018-02-19T07:12:49.239720977Z app[web.1]: [s3proxy] D 02-19 07:12:49.239 S3Proxy-Jetty-13 o.gaul.s3proxy.S3ProxyHandler:302 |::] header: X-Request-Start: 1519024369.236
2018-02-19T07:12:49.239920081Z app[web.1]: [s3proxy] D 02-19 07:12:49.239 S3Proxy-Jetty-13 o.gaul.s3proxy.S3ProxyHandler:302 |::] header: Accept: */*
2018-02-19T07:12:49.240116884Z app[web.1]: [s3proxy] D 02-19 07:12:49.239 S3Proxy-Jetty-13 o.gaul.s3proxy.S3ProxyHandler:302 |::] header: Connection: upgrade
2018-02-19T07:12:49.240325887Z app[web.1]: [s3proxy] D 02-19 07:12:49.240 S3Proxy-Jetty-13 o.gaul.s3proxy.S3ProxyHandler:302 |::] header: X-Forwarded-Port: 80
2018-02-19T07:12:49.242509124Z app[web.1]: [s3proxy] T 02-19 07:12:49.242 S3Proxy-Jetty-13 o.j.r.i.InvokeHttpMethod:47 |::] >> converting ListBlobs
2018-02-19T07:12:49.244756062Z app[web.1]: [s3proxy] T 02-19 07:12:49.244 S3Proxy-Jetty-13 o.j.r.i.RestAnnotationProcessor:47 |::] no annotations on class or invocation.getInvoked(): org.jclouds.azureblob.AzureBlobClient.public abstract org.jclouds.azureblob.domain.ListBlobsResponse org.jclouds.azureblob.AzureBlobClient.listBlobs(java.lang.String,org.jclouds.azureblob.options.ListBlobsOptions[])
2018-02-19T07:12:49.245069868Z app[web.1]: [s3proxy] T 02-19 07:12:49.244 S3Proxy-Jetty-13 o.j.r.i.RestAnnotationProcessor:47 |::] no annotations on class or invocation.getInvoked(): org.jclouds.azureblob.AzureBlobClient.public abstract org.jclouds.azureblob.domain.ListBlobsResponse org.jclouds.azureblob.AzureBlobClient.listBlobs(java.lang.String,org.jclouds.azureblob.options.ListBlobsOptions[])
2018-02-19T07:12:49.245406373Z app[web.1]: [s3proxy] T 02-19 07:12:49.245 S3Proxy-Jetty-13 o.j.r.i.RestAnnotationProcessor:47 |::] looking up default endpoint for org.jclouds.azureblob.AzureBlobClient.public abstract org.jclouds.azureblob.domain.ListBlobsResponse org.jclouds.azureblob.AzureBlobClient.listBlobs(java.lang.String,org.jclouds.azureblob.options.ListBlobsOptions[])[dropshare, [Lorg.jclouds.azureblob.options.ListBlobsOptions;@599bb5d7]
2018-02-19T07:12:49.245810180Z app[web.1]: [s3proxy] T 02-19 07:12:49.245 S3Proxy-Jetty-13 o.j.r.i.RestAnnotationProcessor:47 |::] using default endpoint Optional.of() for org.jclouds.azureblob.AzureBlobClient.public abstract org.jclouds.azureblob.domain.ListBlobsResponse org.jclouds.azureblob.AzureBlobClient.listBlobs(java.lang.String,org.jclouds.azureblob.options.ListBlobsOptions[])[dropshare, [Lorg.jclouds.azureblob.options.ListBlobsOptions;@599bb5d7]
2018-02-19T07:12:49.246188287Z app[web.1]: [s3proxy] T 02-19 07:12:49.245 S3Proxy-Jetty-13 o.j.r.i.RestAnnotationProcessor:47 |::] adding filter org.jclouds.azure.storage.filters.SharedKeyLiteAuthentication@74d7313b from annotation on org.jclouds.azureblob.AzureBlobClient

Any clues?

gaul commented 6 years ago

I cannot reproduce these symptoms. An invalid identity yields 500 error due to a hidden UnknownHostException and an invalid credential yields the expected AccessDenied. The former is a bit weird but the aws cli returns InvalidAccessKeyId in this situation which is also weird.

Your logs end before S3Proxy issues the HTTP request. I see the following:

[s3proxy] D 02-19 10:30:40.214 S3Proxy-Jetty-15 o.j.r.i.InvokeHttpMethod:56 |::] >> invoking ListBlobs
[s3proxy] D 02-19 10:30:40.217 S3Proxy-Jetty-15 jclouds.signature:56 |::] >> GET https://gaultest2.blob.core.windows.net/foo?restype=container&comp=list&maxresults=1000&include=metadata HTTP/1.1

Can you give any more details? Do you only receive a 400 error or do you have the S3 error code like:

<?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchBucket</Code><Message>The specified bucket does not exist</Message><RequestId>4442587FB7D0A2F9</RequestId></Error>
RyanMorash commented 6 years ago

That’s all I’m seeing from docker logs. The error page is just a standard Jetty HTTP error page giving the reason Bad Request. The only thing I can think of that might be messing things up is that I’m using an NGINX proxy, I’ll double check the config there.

RyanMorash commented 6 years ago

Doesn’t seem like a NGINX issue, I get the same thing if I run curl on the host pointed to the container’s IP.

gaul commented 6 years ago

I am sorry without more logs or some exact steps to reproduce your issue I cannot debug it further. Perhaps if you run S3Proxy outside Docker you will get some different result? You can also add some debug statements to help track this down, likely in S3ProxyHandlerJetty. Even if you lack a Java background, building is easy only requiring JDK and mvn.

KYamunaSri commented 6 years ago

I am unable to connect to the Azure blob storage, Get an Illegal Key size. The key is valid and I connect using the same key to the blob from my Azure Storage explorer.

org.jclouds.http.HttpResponseException: java.security.InvalidKeyException: Illegal key size connecting to GET https://dcmpoc01.blob.core.windows.net/localhost/dcmpoc01 HTTP/1.1
17:17:31.505 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:120)
17:17:31.505 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
17:17:31.505 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
17:17:31.505 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
17:17:31.505 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
17:17:31.505 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
17:17:31.505 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at com.sun.proxy.$Proxy46.getBlob(Unknown Source)
17:17:31.505 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at org.jclouds.azureblob.blobstore.AzureBlobStore.getBlob(AzureBlobStore.java:206)
17:17:31.506 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
17:17:31.506 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
17:17:31.506 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
17:17:31.506 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at java.lang.reflect.Method.invoke(Method.java:498)
17:17:31.506 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
17:17:31.506 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at com.sun.proxy.$Proxy47.getBlob(Unknown Source)
17:17:31.506 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at org.gaul.s3proxy.S3ProxyHandler.handleGetBlob(S3ProxyHandler.java:1569)
17:17:31.506 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at org.gaul.s3proxy.S3ProxyHandler.doHandle(S3ProxyHandler.java:676)
17:17:31.506 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at org.gaul.s3proxy.S3ProxyHandlerJetty.handle(S3ProxyHandlerJetty.java:70)
17:17:31.506 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
17:17:31.506 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at org.eclipse.jetty.server.Server.handle(Server.java:499)
17:17:31.506 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
17:17:31.506 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258)
17:17:31.506 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
17:17:31.506 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
17:17:31.506 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
17:17:31.506 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at java.lang.Thread.run(Thread.java:748)
17:17:31.507 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main - Caused by: com.rsa.sslj.x.aK: java.security.InvalidKeyException: Illegal key size
17:17:31.507 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at com.rsa.sslj.x.aj.a(Unknown Source)
17:17:31.507 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at com.rsa.sslj.x.aH.z(Unknown Source)
17:17:31.507 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at com.rsa.sslj.x.aH.a(Unknown Source)
17:17:31.507 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at com.rsa.sslj.x.aH.b(Unknown Source)
17:17:31.507 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at com.rsa.sslj.x.aH.a(Unknown Source)
17:17:31.507 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at com.rsa.sslj.x.aH.b(Unknown Source)
17:17:31.507 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at com.rsa.sslj.x.ap.d(Unknown Source)
17:17:31.507 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at com.rsa.sslj.x.ap.a(Unknown Source)
17:17:31.507 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at com.rsa.sslj.x.ap.j(Unknown Source)
17:17:31.507 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at com.rsa.sslj.x.ap.i(Unknown Source)
17:17:31.507 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at com.rsa.sslj.x.ap.h(Unknown Source)
17:17:31.507 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at com.rsa.sslj.x.aS.startHandshake(Unknown Source)
17:17:31.509 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
17:17:31.509 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
17:17:31.509 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1564)
17:17:31.509 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
17:17:31.509 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)
17:17:31.509 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.invoke(JavaUrlHttpCommandExecutorService.java:96)
17:17:31.509 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.invoke(JavaUrlHttpCommandExecutorService.java:65)
17:17:31.509 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
17:17:31.509 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   ... 24 more
17:17:31.510 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main - Caused by: java.security.InvalidKeyException: Illegal key size
17:17:31.510 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at javax.crypto.Cipher.checkCryptoPerm(Cipher.java:1039)
17:17:31.510 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at javax.crypto.Cipher.init(Cipher.java:1393)
17:17:31.510 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at javax.crypto.Cipher.init(Cipher.java:1327)
17:17:31.510 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at com.rsa.sslj.x.ah.a(Unknown Source)
17:17:31.510 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   at com.rsa.sslj.x.aj.b(Unknown Source)
17:17:31.510 [S3Proxy-Jetty-17 - /dcmpoc01] ERROR org.gaul.s3proxy.Main -   ... 44 more
KYamunaSri commented 6 years ago

Can I understand why is the key size called as Illegal here? Any pointers/suggestions would be very helpful.

gaul commented 6 years ago

Any chance your Java has the weakened security strength?

https://stackoverflow.com/questions/6481627/java-security-illegal-key-size-or-default-parameters

ohthehugemanatee commented 6 years ago

I also have this problem:

With the docker version of s3proxy configured like this:

        env:
        - name: S3PROXY_AUTHORIZATION
          value: none
        - name: JCLOUDS_PROVIDER
          value: azureblob
        - name: JCLOUDS_IDENTIY
          value: storageaccountid
        - name: JCLOUDS_CREDENTIAL
          value: <my azure storage key>
        - name: JCLOUD_REGIONS
          value: us-east-1

I get a 400 error .

$ curl http://azure-s3proxy/   
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 400 </title>
</head>
<body>
<h2>HTTP ERROR: 400</h2>
<p>Problem accessing /. Reason:
<pre>    Bad Request</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
</body>
</html>

I have nothing at all in the container logs, and I'm not sure how to request a higher loglevel.

gaul commented 5 years ago

I could also not reproduce this symptom. You can try setting LOG_LEVEL to trace for more context.

eden881 commented 5 years ago

I can confirm that I have this problem too.

I have the following docker parameters:

LOG_LEVEL=trace
S3PROXY_AUTHORIZATION=aws-v2
S3PROXY_IDENTITY=local-identity
S3PROXY_CREDENTIAL=<my_cred>
JCLOUDS_PROVIDER=azureblob
JCLOUDS_IDENTITY=<my_storage_account_name>
JCLOUDS_CREDENTIAL=<my_storage_account_key>

This is the log output when I try to list my buckets:

[s3proxy] I 05-20 22:58:55.019 main o.g.s.CrossOriginResourceSharing:82 |::] CORS allowed origins: []
[s3proxy] I 05-20 22:58:55.023 main o.g.s.CrossOriginResourceSharing:83 |::] CORS allowed methods: []
[s3proxy] I 05-20 22:58:55.023 main o.g.s.CrossOriginResourceSharing:84 |::] CORS allowed headers: []
[s3proxy] I 05-20 22:58:55.037 main o.g.s.o.eclipse.jetty.util.log:186 |::] Logging initialized @1751ms
[s3proxy] I 05-20 22:58:55.105 main o.g.s.o.e.jetty.server.Server:327 |::] jetty-9.2.z-SNAPSHOT
[s3proxy] I 05-20 22:58:55.136 main o.g.s.o.e.j.s.ServerConnector:266 |::] Started ServerConnector@38ba35f8{HTTP/1.1}{0.0.0.0:80}
[s3proxy] I 05-20 22:58:55.136 main o.g.s.o.e.jetty.server.Server:379 |::] Started @1851ms
[s3proxy] D 05-20 22:59:08.457 S3Proxy-Jetty-17 o.gaul.s3proxy.S3ProxyHandler:286 |::] request: Request(GET /)@61cc5ef
[s3proxy] D 05-20 22:59:08.509 S3Proxy-Jetty-17 o.gaul.s3proxy.S3ProxyHandler:311 |::] header: User-Agent: Cyberduck/6.9.4.30164 (Windows 10/10.0) (x86)
[s3proxy] D 05-20 22:59:08.509 S3Proxy-Jetty-17 o.gaul.s3proxy.S3ProxyHandler:311 |::] header: Authorization: AWS local-identity:<secret>
[s3proxy] D 05-20 22:59:08.509 S3Proxy-Jetty-17 o.gaul.s3proxy.S3ProxyHandler:311 |::] header: Connection: Keep-Alive
[s3proxy] D 05-20 22:59:08.509 S3Proxy-Jetty-17 o.gaul.s3proxy.S3ProxyHandler:311 |::] header: Host: <my_url>:80
[s3proxy] D 05-20 22:59:08.510 S3Proxy-Jetty-17 o.gaul.s3proxy.S3ProxyHandler:311 |::] header: Date: Mon, 20 May 2019 22:59:02 GMT
[s3proxy] D 05-20 22:59:08.513 S3Proxy-Jetty-17 o.gaul.s3proxy.S3ProxyHandler:446 |::] dateheader 1558393142000
[s3proxy] D 05-20 22:59:08.513 S3Proxy-Jetty-17 o.gaul.s3proxy.S3ProxyHandler:451 |::] dateheader 1558393142
[s3proxy] D 05-20 22:59:08.513 S3Proxy-Jetty-17 o.gaul.s3proxy.S3ProxyHandler:456 |::] dateSkew 1558393142
[s3proxy] T 05-20 22:59:08.526 S3Proxy-Jetty-17 o.gaul.s3proxy.S3ProxyHandler:166 |::] stringToSign: GET

Mon, 20 May 2019 22:59:02 GMT
/
[s3proxy] T 05-20 22:59:08.568 S3Proxy-Jetty-17 o.j.r.i.InvokeHttpMethod:47 |::] >> converting ListContainers
[s3proxy] T 05-20 22:59:08.570 S3Proxy-Jetty-17 o.j.r.i.RestAnnotationProcessor:47 |::] no annotations on class or invocation.getInvoked(): org.jclouds.azureblob.AzureBlobClient.public abstract org.jclouds.azure.storage.domain.BoundedSet org.jclouds.azureblob.AzureBlobClient.listContainers(org.jclouds.azure.storage.options.ListOptions[])
[s3proxy] T 05-20 22:59:08.571 S3Proxy-Jetty-17 o.j.r.i.RestAnnotationProcessor:47 |::] no annotations on class or invocation.getInvoked(): org.jclouds.azureblob.AzureBlobClient.public abstract org.jclouds.azure.storage.domain.BoundedSet org.jclouds.azureblob.AzureBlobClient.listContainers(org.jclouds.azure.storage.options.ListOptions[])
[s3proxy] T 05-20 22:59:08.571 S3Proxy-Jetty-17 o.j.r.i.RestAnnotationProcessor:47 |::] looking up default endpoint for org.jclouds.azureblob.AzureBlobClient.public abstract org.jclouds.azure.storage.domain.BoundedSet org.jclouds.azureblob.AzureBlobClient.listContainers(org.jclouds.azure.storage.options.ListOptions[])[[Lorg.jclouds.azure.storage.options.ListOptions;@7523d043]
[s3proxy] T 05-20 22:59:08.572 S3Proxy-Jetty-17 o.j.r.i.RestAnnotationProcessor:47 |::] using default endpoint Optional.of() for org.jclouds.azureblob.AzureBlobClient.public abstract org.jclouds.azure.storage.domain.BoundedSet org.jclouds.azureblob.AzureBlobClient.listContainers(org.jclouds.azure.storage.options.ListOptions[])[[Lorg.jclouds.azure.storage.options.ListOptions;@7523d043]
[s3proxy] T 05-20 22:59:08.578 S3Proxy-Jetty-17 o.j.r.i.RestAnnotationProcessor:47 |::] adding filter org.jclouds.azure.storage.filters.SharedKeyLiteAuthentication@e72b85b from annotation on org.jclouds.azureblob.AzureBlobClient
gaul commented 5 years ago

I added some more logging; could you repeat your test and share the results?

eden881 commented 5 years ago

Thanks for your answer, unfortunately the logs looks pretty much the same:

[s3proxy] I 05-21 18:33:17.929 main o.g.s.CrossOriginResourceSharing:82 |::] CORS allowed origins: []
[s3proxy] I 05-21 18:33:17.933 main o.g.s.CrossOriginResourceSharing:83 |::] CORS allowed methods: []
[s3proxy] I 05-21 18:33:17.934 main o.g.s.CrossOriginResourceSharing:84 |::] CORS allowed headers: []
[s3proxy] I 05-21 18:33:17.948 main o.g.s.o.eclipse.jetty.util.log:186 |::] Logging initialized @2043ms
[s3proxy] I 05-21 18:33:18.019 main o.g.s.o.e.jetty.server.Server:327 |::] jetty-9.2.z-SNAPSHOT
[s3proxy] I 05-21 18:33:18.055 main o.g.s.o.e.j.s.ServerConnector:266 |::] Started ServerConnector@3c4d36a8{HTTP/1.1}{0.0.0.0:80}
[s3proxy] I 05-21 18:33:18.056 main o.g.s.o.e.jetty.server.Server:379 |::] Started @2151ms
[s3proxy] D 05-21 18:34:29.719 S3Proxy-Jetty-17 o.gaul.s3proxy.S3ProxyHandler:286 |::] request: Request(GET /)@5c04a70d
[s3proxy] D 05-21 18:34:29.720 S3Proxy-Jetty-17 o.gaul.s3proxy.S3ProxyHandler:311 |::] header: User-Agent: Cyberduck/6.9.4.30164 (Mac OS X/10.14.4) (x86_64)
[s3proxy] D 05-21 18:34:29.720 S3Proxy-Jetty-17 o.gaul.s3proxy.S3ProxyHandler:311 |::] header: Authorization: AWS local-identity:<secret>
[s3proxy] D 05-21 18:34:29.721 S3Proxy-Jetty-17 o.gaul.s3proxy.S3ProxyHandler:311 |::] header: Connection: Keep-Alive
[s3proxy] D 05-21 18:34:29.721 S3Proxy-Jetty-17 o.gaul.s3proxy.S3ProxyHandler:311 |::] header: Host: <my_url>:80
[s3proxy] D 05-21 18:34:29.721 S3Proxy-Jetty-17 o.gaul.s3proxy.S3ProxyHandler:311 |::] header: Date: Tue, 21 May 2019 18:34:29 GMT
[s3proxy] D 05-21 18:34:29.729 S3Proxy-Jetty-17 o.gaul.s3proxy.S3ProxyHandler:446 |::] dateheader 1558463669000
[s3proxy] D 05-21 18:34:29.732 S3Proxy-Jetty-17 o.gaul.s3proxy.S3ProxyHandler:451 |::] dateheader 1558463669
[s3proxy] D 05-21 18:34:29.732 S3Proxy-Jetty-17 o.gaul.s3proxy.S3ProxyHandler:456 |::] dateSkew 1558463669
[s3proxy] T 05-21 18:34:29.737 S3Proxy-Jetty-17 o.gaul.s3proxy.S3ProxyHandler:166 |::] stringToSign: GET

Tue, 21 May 2019 18:34:29 GMT
/
[s3proxy] T 05-21 18:34:29.778 S3Proxy-Jetty-17 o.j.r.i.InvokeHttpMethod:47 |::] >> converting ListContainers
[s3proxy] T 05-21 18:34:29.780 S3Proxy-Jetty-17 o.j.r.i.RestAnnotationProcessor:47 |::] no annotations on class or invocation.getInvoked(): org.jclouds.azureblob.AzureBlobClient.public abstract org.jclouds.azure.storage.domain.BoundedSet org.jclouds.azureblob.AzureBlobClient.listContainers(org.jclouds.azure.storage.options.ListOptions[])
[s3proxy] T 05-21 18:34:29.780 S3Proxy-Jetty-17 o.j.r.i.RestAnnotationProcessor:47 |::] no annotations on class or invocation.getInvoked(): org.jclouds.azureblob.AzureBlobClient.public abstract org.jclouds.azure.storage.domain.BoundedSet org.jclouds.azureblob.AzureBlobClient.listContainers(org.jclouds.azure.storage.options.ListOptions[])
[s3proxy] T 05-21 18:34:29.781 S3Proxy-Jetty-17 o.j.r.i.RestAnnotationProcessor:47 |::] looking up default endpoint for org.jclouds.azureblob.AzureBlobClient.public abstract org.jclouds.azure.storage.domain.BoundedSet org.jclouds.azureblob.AzureBlobClient.listContainers(org.jclouds.azure.storage.options.ListOptions[])[[Lorg.jclouds.azure.storage.options.ListOptions;@6cc46b6b]
[s3proxy] T 05-21 18:34:29.782 S3Proxy-Jetty-17 o.j.r.i.RestAnnotationProcessor:47 |::] using default endpoint Optional.of() for org.jclouds.azureblob.AzureBlobClient.public abstract org.jclouds.azure.storage.domain.BoundedSet org.jclouds.azureblob.AzureBlobClient.listContainers(org.jclouds.azure.storage.options.ListOptions[])[[Lorg.jclouds.azure.storage.options.ListOptions;@6cc46b6b]
[s3proxy] T 05-21 18:34:29.788 S3Proxy-Jetty-17 o.j.r.i.RestAnnotationProcessor:47 |::] adding filter org.jclouds.azure.storage.filters.SharedKeyLiteAuthentication@241d2e5 from annotation on org.jclouds.azureblob.AzureBlobClient
gaul commented 5 years ago

Could you add the following to S3ProxyHandlerJetty.sendS3Exception:

logger.trace("sendS3Exception", se);

This may reveal more context.

eden881 commented 5 years ago

I've managed to solve this problem. It seems that the default JCLOUDS_ENDPOINT for the azureblob provider is incorrect.

For a Storage Account named foo, I've passed to the container the following environment variables:

With those parameters, the S3 proxy works as expected.

gaul commented 3 years ago

Closing due to inactivity. Please reopen if symptoms persist.

kasbuunk commented 2 years ago

I have similar configuration as @eden881, but get 500 error with the following log in the s3proxy container:

[s3proxy] W 03-15 15:40:43.349 S3Proxy-Jetty-16 o.j.a.s.h.AzureStorageClientErrorRetryHandler:74 |::] Cannot retry after server error, command is not replayable: [method=org.jclouds.azureblob.AzureBlobClient.public abstract java.lang.String org.jclouds.azureblob.AzureBlobClient.putBlob(java.lang.String,org.jclouds.azureblob.domain.AzureBlob)[mybucket, [properties=[name=upload/3c549054-7173-45e5-aa0e-3d5b4cfa2829/thisfile, container=null, url=null, contentMetadata=[cacheControl=null, contentDisposition=null, contentEncoding=null, contentLanguage=null, contentLength=24, contentMD5=null, contentType=application/unknown, expires=null], eTag=null, lastModified=null, leaseStatus=Unlocked, metadata={}, type=BlockBlob]]], request=PUT https://xxx.blob.core.windows.net/mybucket/upload/3c549054-7173-45e5-aa0e-3d5b4cfa2829/thisfile HTTP/1.1]
rightpad commented 2 years ago

I have similar configuration as @eden881, but get 500 error with the following log in the s3proxy container:

[s3proxy] W 03-15 15:40:43.349 S3Proxy-Jetty-16 o.j.a.s.h.AzureStorageClientErrorRetryHandler:74 |::] Cannot retry after server error, command is not replayable: [method=org.jclouds.azureblob.AzureBlobClient.public abstract java.lang.String org.jclouds.azureblob.AzureBlobClient.putBlob(java.lang.String,org.jclouds.azureblob.domain.AzureBlob)[mybucket, [properties=[name=upload/3c549054-7173-45e5-aa0e-3d5b4cfa2829/thisfile, container=null, url=null, contentMetadata=[cacheControl=null, contentDisposition=null, contentEncoding=null, contentLanguage=null, contentLength=24, contentMD5=null, contentType=application/unknown, expires=null], eTag=null, lastModified=null, leaseStatus=Unlocked, metadata={}, type=BlockBlob]]], request=PUT https://xxx.blob.core.windows.net/mybucket/upload/3c549054-7173-45e5-aa0e-3d5b4cfa2829/thisfile HTTP/1.1]

I'm experiencing the exact same issue, running the latest docker image with the following config:

JCLOUDS_PROVIDER=azureblob
JCLOUDS_AZUREBLOB_AUTH=azureKey
JCLOUDS_ENDPOINT=https://xxxxxx.blob.core.windows.net
JCLOUDS_IDENTITY=xxxxxx"
JCLOUDS_CREDENTIAL=xxxxx"
S3PROXY_AUTHORIZATION=none"
LOGLEVEL=trace"
JCLOUDS_WIRE=debug"

error:

[s3proxy] W 08-04 21:13:34.358 S3Proxy-Jetty-14 o.j.a.s.h.AzureStorageClientErrorRetryHandler:74 |::] Cannot retry after server error, command is not replayable: [method=org.jclouds.azureblob.AzureBlobClient.public abstract java.lang.String org.jclouds.azureblob.AzureBlobClient.putBlob(java.lang.String,org.jclouds.azureblob.domain.AzureBlob)[xxx, [properties=[name=xxxxx, container=null, url=null, contentMetadata=[cacheControl=null, contentDisposition=null, contentEncoding=null, contentLanguage=null, contentLength=23, contentMD5=fa2e0e4e17dded0a609b18761dc075cb, contentType=text/plain, expires=null], eTag=null, lastModified=null, leaseStatus=Unlocked, metadata={xxxx=xxxxx}, type=BlockBlob]]], request=PUT https://xxxxxxx.blob.core.windows.net/xxxx/xxxxx HTTP/1.1]
nr-ey commented 1 year ago

@pdspx Did you figure this out?

rightpad commented 1 year ago

I didn't, sorry

-------- Original Message -------- On Nov 28, 2022, 4:53 PM, nr-ey wrote:

@.***(https://github.com/pdspx) Did you figure this out?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

timuralp commented 1 year ago

@nr-ey @pdspx can you share the request you're making? I can reproduce this issue when using metadata on the object that is not supported by Azure.

missingcharacter commented 1 year ago

@timuralp is this the way you can reproduce?

aws --no-verify-ssl --endpoint-url http://s3proxy s3 --debug cp test.txt s3://<bucketname>/ --metadata '{"cms-id":"34533452"}'

Seems like the problem is -

image

TArtiushenko commented 1 year ago

@missingcharacter is there any solution?

missingcharacter commented 1 year ago

@TArtiushenko seems like azure blob storage does not let you use - in metadata, the solution is to not use - in metadat

TArtiushenko commented 1 year ago

@missingcharacter how about change the s3proxy code to replace all - with _ in metadata when using azure?

missingcharacter commented 1 year ago

@TArtiushenko The limitation is not on s3proxy; in the screenshot I shared the notification The name: 'something-id' should be a valid C# identifier comes from Azure itself, maybe there should be a better error from s3proxy saying The name: '<metadata key>' should be a valid C# identifier

Also, you could try making a PR to s3proxy for it to replace all - with _, maybe you can convince the maintainers to support this.

gaul commented 1 year ago

This sounds like #466. I don't think it is reasonable to map all - to _ for Azure since this could confuse some users. But I would be happy if someone contributed a middleware that maps arbitrary buckets names, blob names, and user metadata. This could be similar to the tr command which maps from one set of characters to another.

gaul commented 2 weeks ago

Duplicate of https://github.com/gaul/s3proxy/issues/466. Please test with the proposed https://github.com/gaul/s3proxy/pull/702.