Open craftzneko opened 1 year ago
Hi, to be sure are you trying to use s3proxy with an Azure backend ?
@alifirat hi, yes with an azure backend
Please check https://github.com/gaul/s3proxy/wiki/Storage-backend-examples
Just set the authentication method to none and also the one about that ignores unknown headers.
@alifirat set as above, server still returns 501 not implemented,
If i navigate to endpoint with bucket in path https://itestproxy.azurewebsites.net/testBucketname i see this response
The logs from S3proxy state
Unknown method POST with URI /Testbucketname
Did you try to add a folder after testBucket ? Also can you share your s3proxy configuration ?
Cheers @alifirat Forgive my ignorance S3 is new to me. so if i misunderstand please correct me. I created a folder in the backend called Uploads as that is what the client is trying to save to (along with random guid string folders). I can see this at the endpoint now so i know it can see my azure side
https://testproxy.azurewebsites.net/testbucketname
Config is created by environment variables here JCLOUDS_AZUREBLOB_AUTH=azureKey JCLOUDS_CREDENTIAL=REDACTED But IS my Azure KEY JCLOUDS_ENDPOINT=https://MyStorageAccount.blob.core.windows.net/ JCLOUDS_IDENTITY=REDACTED but is storage account name JCLOUDS_PROVIDER=azureblob S3PROXY_AUTHORIZATION=none S3PROXY_CORS_ALLOW_HEADERS=* S3PROXY_CORS_ALLOW_METHODS=GET PUT POST HEAD S3PROXY_CORS_ALLOW_ORIGINS=https://MySite.azurewebsites.net S3PROXY_CREDENTIAL=TestCred S3PROXY_IDENTITY=TestPass S3PROXY_ENDPOINT=http://0.0.0.0:8080 S3PROXY_IGNORE_UNKNOWN_HEADERS=true
Looking at the code where the unsupported gets thrown, none of the headers in the request match the list but the form data does have policy in
In s3proxyhandler.java UNSUPPORTED_PARAMETERS = ImmutableSet.of( "accelerate", "analytics", "cors", "inventory", "lifecycle", "list-type", "logging", "metrics", "notification", "policy", "replication", "requestPayment", "restore", "tagging", "torrent", "versioning", "versions", "website"}
my request headers
POST /mytestbucket HTTP/1.1 Accept: / Accept-Encoding: gzip, deflate, br Accept-Language: en-GB,en;q=0.9,en-US;q=0.8 Connection: keep-alive Content-Length: 22860 Content-Type: multipart/form-data; boundary=----WebKitFormBoundary9LQ2gN3Nv9OqJiQA DNT: 1 Host: testproxy.azurewebsites.net Origin: https://testwebsite.azurewebsites.net Sec-Fetch-Dest: empty Sec-Fetch-Mode: cors Sec-Fetch-Site: cross-site User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36 Edg/115.0.1901.200 sec-ch-ua: "Not/A)Brand";v="99", "Microsoft Edge";v="115", "Chromium";v="115" sec-ch-ua-mobile: ?0 sec-ch-ua-platform: "Windows"
Formdata payload
Cache-Control: max-age=31557600 Content-Type: image/png Content-Disposition: attachment key: uploads/7bbca2fd-69eb-48e5-baee-60afd34c4505/e5876117-7741-4d25-9350-dab2d1f305f6/2bc62b04-fb9b-4c91-ab78-402c02a3a3ef.png acl: private bucket: mytestbucket X-Amz-Algorithm: AWS4-HMAC-SHA256 X-Amz-Credential: testcred/20230810/us-east-1/s3/aws4_request X-Amz-Date: 20230810T082529Z Policy: eyJleHBpcmF0aW9uIjoiMjAyMy0wOC0xMFQwOToyNToyOVoiLCJjb25kaXRpb25zIjpbWyJjb250ZW50LWxlbmd0aC1yYW5nZSIsMCwxMDAwMDAwMDBdLFsic3RhcnRzLXdpdGgiLCIkQ24tgs9udGVudC1UeXBlIiwiaW1hZ2UvcG5nIl0sWyJzdGFydHMtd2l0aCIsIiRDYWNoZS1Db250cm9sIiwiIl0seyJDb250ZW50LURpc3Bvc2l0aW9uIjoiYXR0YWNobWVudCJ9LHsia2V5IjoidXBsb2Fkcy83YmJjYTJzYyYjA0LWZiOWItNGM5MS1hYjc4LTQwMmMwMmEzYTNlZi5wbmcifSx7ImFjbCI6InByaXZhdGUifSx7ImJ1Y2tldCI6InRlY2hub3Rlc21lZGlhIn0seyJYLUFtei1BbGdvcsdfsd0aG0iOiJBV1M0LUhNQUcdMtU0hBMjU2In0seyJYLUFtei1DcmVkZW50aWFsIjoiMTIzNDU2NzgvMjAyMzA4MTAvdXMtZWFzdC0xL3MzL2F3czRfcmVxdWVzdCJ9LHsiWC1BbXotRsdsdcsdcsdcsdcsdvcMwODEwVDA4MjUyOVoifV19 X-Amz-Signature: 43373e09647af6399d896754c214bccbc9f014a56726491c9fd15622df1d711e file: (binary)
I believe the client is using presigned URLS but not sure what policy is, is there any way of working around this @gaul
I am really desperate now to get this resovled. Its been 4 weeks of hell trying to get outline to work with s3 storage and i am no closer to a solution. As far as i can see above outline makes a valid S3 request but 501 is returned and i cant see how. If it is never gonna work with S3proxy then i just need to move on, anybody able to see what the issue is?
I'm having a look tomorrow
Hi
I have been trying to implement this project in Azure app Service docs.getoutline.com. It requires s3 storage. I have setup s3proxy as another container in Azure i am trying to set it up to function. It appears outline requires the following variables to be filled in or it will not attempt to send files to s3
AWS_REGION=us-east-2 AWS_S3_FORCE_PATH_STYLE=false AWS_S3_UPLOAD_BUCKET_NAME=my-bucket-name AWS_S3_UPLOAD_BUCKET_URL=https://my-bucket-name.s3.us-east-2.amazonaws.com AWS_S3_UPLOAD_MAX_SIZE=26214400 AWS_ACCESS_KEY_ID=AK66L9HZpTtfrpFFgtVxcxOUTn AWS_SECRET_ACCESS_KEY=97L9HfssZpTtfrxOUTVxcnpgtSa
I note that the bucket is part of the url, but in order to use s3proxy i am unsure what i need to change above for it to work. I have created an app registration in azure and have the secret as well as the storage account key but what is the bucket URL i need, also what would i put in AWS_S3_UPLOAD_BUCKET_NAME?
Any help would be very much appreciated, spent 3 weeks so far getting this setup and this is my last challenge now, hoping it wont all have been for waste
I have got much further now, i did not realise that it was javascript running in the browser that is sending the requests. I configured CORS and some other bits and i have got further now but i am getting a 501 response back
A header you provided implies functionality that is not implemented
@gaul i see there is another issue with the same error https://github.com/gaul/s3proxy/issues/73
I can provide the post headers and payload if it can help. It could be i have missed some settings as i see this in log
2023-08-02T21:56:35.165188289Z [s3proxy] E 08-02 21:56:35.155 S3Proxy-Jetty-14 o.gaul.s3proxy.S3ProxyHandler:769 |::] Unknown method POST with URI /mybucket