bucket name has to be at the beginning of url when put-object

[WildBench]

It works here only when the url is set as "bucket-name.s3.amazonaws.com".

[xach]

I don't understand this, sorry - can you give a more complete example?

[WildBench]

Sorry that I didn't make it clear. Here is the link which helped me figure out the issue. It seems when uploading with "put-object", bucket-name may be required to be in the url. Now in ZS3, the url in "put-object" request is like http[s]://endpoint/file-name. I changed the url to http[s]://bucket-name.endpoint/file-name to work it out.

https://stackoverflow.com/questions/31838990/amazon-s3-bucket-malformedxml-error-when-uploading

[xach]

put-object takes a bucket name and a key - you don't normally specify an URL. How does the URL come into play?

[WildBench]

Given a bucket name and a key, put-object ran into "Condition of type MALFORMED-XML", returned "MalformedXML: The XML you provided was not well-formed or did not validate against our published schema". I had to look into it and found that it works when changing the url in request.

[xach]

What region is the bucket in?

[WildBench]

Far away, cn-north-1

[xach]

Thanks, I will look into it.

[ghard]

I'm encountering the same problem trying to use this library with DigitalOcean which should only require the endpoint to be changed, but, trying to create a bucket on DigitalOcean the URL method called upon generation the request never adds the bucket in the url passed on to drakma:http-request. Probably because of the logic baked into request-response

(setf (endpoint request) (redirected-endpoint (endpoint request) (bucket request)))

that expects a bunch of redirects to happen prior to these operations.

I tried to get by this by changing the redirected-endpoint to always return "bucket.endpoint" but that seems to interfere with how requests are signed.

[kat-co]

I can get this to work by changing the endpoint within the lexical scope of the put operations, e.g.:

(let ((zs3:*s3-endpoint* "my-bucket.sfo2.digitaloceanspaces.com"))
  (zs3:put-string "hello world" "my-bucket" "hello-world.txt"))

$ aws --endpoint='sfo2.digitaloceanspaces.com' s3 ls my-bucket
2020-01-24 09:57:13         11 hello-world.txt

So there are a few problems here:

• The endpoint must be prepended with the bucket name for the put functions to work
• The bucket passed into the put functions ends up doing nothing
• Calling a put function without the bucket name prepended to the endpoint places the object in the root of the space, but succeeds
• All of this leads to the very confusing behavior of a put succeeding, but a subsequent get failing.

[ghard]

It actually wasn't too big a deal to get stuff working on Digitalocean. A minimal change in the endpoint, and host for the auth signature.

There's another thing with handling errors, as they seem to not always be what the specialize-response and its XML parser expects and I haven't touched that at all.

I haven't run anything even close to a proper range of tests to find out if there are other parts of the API that are borked there - I've seen create-bucket put-file, copy-object, delete-object, get-acl and put-acl work so far and that is already quite close to all I need myself.

The branch digitalocean-endpoint on my fork contains the changes I've made so far if you want to take a look. I haven't made a pull request as I'm not certain I haven't broken something else there.

Xach might want to have a peek. I bypassed some redirect handling that I believe only happens on Amazon but I might've cargo-culted it :)