vsoch
commented
4 years ago okay here is an update after working today. I was able to at least get a signature (and the associated headers) into the signed url by adding it like this:
from botocore.client import Config
...
s3_external = session.client(
"s3",
use_ssl=MINIO_SSL,
region_name=MINIO_REGION,
endpoint_url=MINIO_HTTP_PREFIX + MINIO_EXTERNAL_SERVER,
verify=False,
config=Config(signature_version='s3v4'),
)and from this comment I think the (still) signature mismatch is because s3v4 uses the host to validate (which is different inside the container, minio:9000 from outside the container 127.0.0.1:9000).
- this page shows usage of the generate_presigned_url function but I don't think this is the issue, unless we can somehow customize the Host header that is used?
- this shows inputs for generate_signed_url, but I don't see anything about setting the host. Interestingly, the returned metadata does include a "HostID" which is an empty string.
- these are the s3 authentication types and per Minio documentation I think it just supports s3v4 and s3v2.
- this shows the signer that generates the URL, I'm wondering if we need to interact with that directly with some customization of the input params to get around not being able to set the correct hostname?
- this gist has a good example and I posted asking for help (no response yet).
So my thinking is that one of the following needs to be done:
- the request to create the signed URL modified to use the external host header - I had thought by creating the "external_s3" client this would work, but I think the issue is that the request needs to be made with the regular s3 client, to ping the minio server from inside the container.
- the signature version needs to be modified to not use the host as a variable (and this would also require modifying the minio server to use a different version other than s3v4
- some change to how minio is deployed so it's accessed from inside and outside the container via the same hostname (if anyone has ideas this would probably be best to try first).
harshavardhana
Aneoshun
This is an extension of #297 that includes multipart based on the example here I've successfully been able to:
However there is some error triggered by the scs-library-client in this function so that it calls the _multipart_abort endpoint and the request does not finish. I have a feeling it has something to do with a parameter missing for the request, for example here are the differences between a working PUT request (non multipart) and a not working single Part:
This URL works (a single PUT request for minio, generated with presigned_put_url
And note that the only way I am able to do this is by instantiating two minio clients, one with an internal url (inside the container the server is minio:9000), and one with an external url (from the client the server is 127.0.0.1:9000). That was tricky but seemed to work (the work in #297).
This next URL is generated based on the example here and I again needed to create an s3 client to handle an internal request to start the multipart upload:
And then a separate client (s3_external) to generate URLs with the upload id generated from above:
The above generates a url that looks like this and does not work as the Singularity client aborts and calls _multiupload_abort:
I was just able to enable debug output for Singularity, and then use the mc client for minio to determine the next issue to address:
Likely I need to look into how to make sure the correct signature version / method is being used, I'm not sure what the default is but I don't think it's the same that minio expects.