Support authenticated S3 repositories

[rombert]

S3 is a convenient option for hosting yum repositories, and using a public S3 bucket works very well.

When trying to enabled authentication hewever, things get hairy. AWS requires either a specific Authorization header or passing a query string with the parameters. The signature that is passed depends on the day when it is generated, so I can't 'presign' the zypper URL with it.

http://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html

It would be great if zypper would have native support for S3 authentication.

[dmacvicar]

Ok, if the Authorization header that is required is not static, you need a plugin similar to what we developed for Spacewalk, using the ZYpp plugin API. It should take you like 20 lines of python, and I am happy to assist you with it.

The url resolver type of plugin can "create" a url and inject headers. Look into bin/spacewalk-resolver.py. You would install it in /usr/lib/zypp/plugins/urlresolver/s3.

When you add a repo, lets say s3://bucket (and may be some query string), the plugin can based on the url, perform all the required authentication (may be using the s3 client libraries), resolve a normal https url out of it. The plugin would return the resolved url PLUS the http headers libzypp needs to inject.

We provide that ZyppPlugin python class in zypp-plugin-python package for convenience. In theory you can implement them in any language if you implement the text based protocol.

It would be great that you could publish then your git repository as zypp-plugin-s3, may be in http://github.com/openSUSE ?.

[rombert]

Thanks for the information @dmacvicar - I'll try and see if I can code it myself, though my python is quite rusty.

I can't promise anything about making it public though, I would need to first clear it with my employer.

[dmacvicar]

I'd prefer to do it myself in my private time than to see this going closed source. Can you point me to a repository I can test?

[rombert]

I can set up a repository for you to test and also S3 authentication credentials. Can I send them to the email address linked to your OBS account?

[dmacvicar]

you can send it to my username at suse . com

[rombert]

Sent just now, thanks!

[dmacvicar]

Do you have some guidelines on how to generate the signatures using the boto api? like which input would you need? (eg. the availability zone?) in addition to the access credentials.

[rombert]

I have not found the right boto API to use yet - it seems to be too high-level for that. I did find some python examples in the Amazon docs.

http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html

It looks like it needs the date, region name and service name.

[dmacvicar]

@mlandres @rombert

So I did a prototype and I did found a roadblock. https://s3.amazonaws.com/doc/s3-developer-guide/RESTAuthentication.html http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html

https://github.com/openSUSE/libzypp/blob/a7a038aeda1ad6d9e441e7d3755612aa83320dce/zypp/media/MediaAccess.cc#L113 allows us to inject headers for a url.

So I can take a repo foo

[foo]
enabled=1
autorefresh=0
baseurl=plugin:s3?aws_access_key_id=ACCESS_KEY&aws_secret_access_key=SECRET&name=NAME
type=yum

And resolve it to http://NAME.s3.amazonaws.com plus the auth headers. Problem is that amazon expects every request to be signed, and the signature includes the path to the resource, while the resolver plugin only resolves the baseurl, and then the path is appended (this means the plugins never gets the path to be used in the signature).

This is of course solvable but it would require some changes and to update libzypp everywhere to change the plugin API.

Unless there is a way to authenticate without passing the resource path? I only found https://www.whitneyindustries.com/aws/2014/11/16/boto-plus-s3-plus-sts-tokens.html but I am not familiar with it, and it requires Ensure your IAM user has STS access.

[rombert]

@dmacvicar I don't think there's another way of getting S3 to authenticate without passing in the resource path. Reading the documentation on temporary security credentials

http://docs.aws.amazon.com/AmazonS3/latest/dev/MakingRequests.html#requestsUsingTempCred

it looks to me like a signature would be required in addition to the x-amz-security-token ( emphasis mine )

You can use these temporary security credentials in making requests to Amazon S3. The API libraries compute the necessary signature value using those credentials to authenticate your request. If you send requests using expired credentials, Amazon S3 denies the request.

But since I have no experience with STS I can't tell for sure.whether that is the case or not.

[frezbo]

@rombert if you set a s3 vpc endpoint and setup bucket policies allowing access to the VPC, you can just add s3 as a normal repo, but this is the only issue: https://github.com/openSUSE/zypper/issues/143

[rombert]

@frezbo - that's a good pointer, thanks. It does not completely solve my issue as for me bootstrapping repositories is an action I do when provisioning a machine, which is way before I can add VPN credentials.

[frezbo]

@rombert Why is that not possible. host the repo in s3, create a VPC endpoint, attach a bucket policy either allowing the vpc or the vpc endpoint or even the particular instance, so the downloads work over plain HTTPS without any authentication.

[rombert]

@frezbo - it's possible in general, but it's not in my use case :-) As I mentioned, the way I provision servers means that defining repositories should not require any dependencies - in this case I would depend on a VPN connection which I am trying to avoid.

[sclausson]

I’m currently using a yum plugin which supports using IAM access keys (or an ec2 role) to authenticate to an S3 bucket. Now I find myself having to support SuSE clients running zypper.The “heavy lifting” of S3 Auth is already taken care of in this python script To rewrite this for zypper is it just a matter of getting familiar with the libzypp library?

[mlandres]

(I was almost certain I left a comment here.)

ZYPP supports using a plugin to prepare the repositories base URL. But for this use case we'll need a plugin which is able to finalize the download URL (base URL / filepath). Such a plugin would then be able to add the signature and properly encode (#143) the URL.

(@sclausson: looks like I added the comment to the wrong issue: #143 Jan 5; I'm sorry)

[Gregory-Berkman-Imprivata]

I’m currently using a yum plugin which supports using IAM access keys (or an ec2 role) to authenticate to an S3 bucket. Now I find myself having to support SuSE clients running zypper.The “heavy lifting” of S3 Auth is already taken care of in this python script To rewrite this for zypper is it just a matter of getting familiar with the libzypp library?

@sclausson Were you able to get this to work?