ERROR 400 when getting package from S3

[electrosenpai]

Hello, got some trouble with the new versions

Dockerfile version : 1 Storage = s3

config.ini :

use = egg:pypicloud

pyramid.reload_templates = false
pyramid.debug_authorization = false
pyramid.debug_notfound = false
pyramid.debug_routematch = false
pyramid.default_locale_name = en

pypi.storage = s3
storage.bucket = s3-repository
storage.region = eu-central-1
pypi.default_read = everyone
pypi.auth = sql

##########
# user   #
#########

db.url = sqlite:////var/lib/pypicloud/db.sqlite
auth.db.url = sqlite:////var/lib/pypicloud/db.sqlite

# For beaker
session.encrypt_key = replaceme
session.validate_key = replaceme
session.secure = false

###
# wsgi server configuration
###

[uwsgi]
paste = config:%p
paste-logger = %p
master = true
processes = 20
reload-mercy = 15
worker-reload-mercy = 15
max-requests = 1000
virtualenv = /env
http = 0.0.0.0:8080

###
# logging configuration
# http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/logging.html
###

[loggers]
keys = root, boto

[handlers]
keys = console

[formatters]
keys = generic

[logger_root]
level = INFO
handlers = console

[logger_boto]
level = WARN
qualname = boto
handlers =

[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic

[formatter_generic]
format = %(levelname)s %(asctime)s [%(name)s] %(message)s

I'm getting error 400 when trying to get package.

This error comes from the url generate on package should be something like this :

https://somes3repository/?X-Amz-Algorithm=AWS4-HMAC-SHA256

but look like this :

https://s3-pip-test.s3.amazonaws.com/aba5/boto3/boto3-1.4.7-py2.py3-none-any.whl?AWSAccessKeyId=ZAEGTBHZTHHYZRT

How ever I just rollback to the 0.5 version and it work correctly.

Do not hesitate to ask me more informations, I will be happy to help you.

Edit : should I put this issue on the main repository ?

[electrosenpai]

How to reproduce my use case :

• Create a service in aws cluster with alb
• uploading version 1.0.0 from the dockerhub https://hub.docker.com/r/stevearc/pypicloud/tags/
• adding s3 in storage.

You will be able to add new library but you just can't get them even if you use the GUI or/and using pip install -i http://myurlform-pypicloud.com

[stevearc]

Thanks for the report! I am traveling and don't have my computer right now, but I'll look into this as soon as I get the chance

[electrosenpai]

No problem :)

[stevearc]

While doing the steps for the repro, I noticed that your config is missing credentials and that storage.region should be storage.region_name. The url you pasted is also seems to be missing a couple parameters (mine also have &Expires=1510626276&Signature=eS06ZxxO7M0zTj6cB7QrhTj0ujM%3D). I'm going to assume that these are all copy/paste errors (or intentional), but if any of those sound suspicious let me know and I'll dig in more.

My best guess is that the eu-central-1 region has different signature requirements. Can you try changing the signature version and see if that fixes your issue? I think s3v4 would be a good starting bet.

Pypicloud docs: http://pypicloud.readthedocs.io/en/latest/topics/storage.html#s3 Botocore docs: https://botocore.readthedocs.io/en/stable/reference/config.html#botocore.config.Config List of potential values: https://github.com/boto/botocore/blob/8483d71daf66234644e760050ab2371eae95eac9/botocore/auth.py#L831-L844

[electrosenpai]

I noticed that your config is missing credentials

I'm not using credentials but IAM role on the machine that's why there are not set here.

Thank's for your anwser I will try as soon as possible !

[electrosenpai]

Hello, sorry for taking that time to respond I had a lot of things to do :(

I follow your advises and now i can download correctly a python package ! But ... There is no more package in the repository detect by pypi-cloud but the files are in my bucket s3

To make my service operationnel again I use version "0.5.6".

[stevearc]

When you say that pypicloud doesn't detect the packages in your bucket, do you mean the cache doesn't get populated? Did you try refreshing the cache? Were any files visible or was the cache totally empty?

[electrosenpai]

do you mean the cache doesn't get populated

Cache was empty, but package was present on bucket s3. What do you mean by refreshing the cache ?

[stevearc]

If you go to the admin page there's a button that lets you refresh the cache. You can also trigger it with a simple curl http://pypicloud.readthedocs.io/en/latest/topics/api.html#get-admin-rebuild

If the cache was empty, how did you manage to test downloading a package?

[electrosenpai]

Happy new year ! And one more time sorry for the very long anwser i'm truly sorry and I won't let this happen again

Reload doesn't change anything, still no package when reload.

My uses case to test :

• Create empty bucket
• Create a tmp ec2 with correct port open and docker pre-install on
• Running container
• Fetching or uploading a new package (for example boto3)

At this moment everythings seems good, the package is correctly display and we can access to it (thank's to the s3v4 :) ) and the package is present in the new bucket.

Now I want to test if these data are persistent, so i just shutdown the docker and re-run it.

And now there is no more package in the GUI but it's present in my s3 bucket.

So i'm using the /rebuild command and now all my package are back :+1:

There is a way to rebuild everytime the docker starting for the first time ?

Edit :

I found this issue in main repository https://github.com/stevearc/pypicloud/issues/5

curl user:pass@yourserver.com/admin/rebuild

And it's answer my question many thank's for your patience and your time.