Internal Mirroring

[ewdurbin]

Our VCL should fallback to an internal mirror for ^/simple and ^/pypi/.*/json$

This will allow us to serve the bulk of traffic during maintenance and downtime.

[dstufft]

Thinking about this, one thing that will make this harder is that Warehouse no longer uses relative URLs for the /simple/ but instead uses files.pythonhosted.org. We'll need add a feature to bandersnatch that allows us to point the files to a different domain name.

[brainwane]

@ewdurbin Go ahead and put this in a milestone, please.

[ewdurbin]

Feature PRs in bandersnatch:

• https://bitbucket.org/pypa/bandersnatch/pull-requests/39
• https://bitbucket.org/pypa/bandersnatch/pull-requests/40

[ewdurbin]

Setting up a new internal mirror running from my fork via an hg+https:// install to test.

[ewdurbin]

new mirror is loading up at https://mirror.dub1.pypi.io/

[ewdurbin]

Internal mirror is up, but talking to pypi.org it is failing to stay up to date due to purging issues.

[di]

@ewdurbin Is this resolved now that #3396 is merged?

[ewdurbin]

great question! we need to validate that bandersnatch continues to be able to mirror from pypi.org over the course of the next few days, and then update our VCL use this as an internal failover for /simple, /pypi/.*/json$, and under extreme circumstances /packages.

[ewdurbin]

our new internal mirror has been holding strong! next step is VCL to do fallback at the edge for /simple, /packages, and /pypi/*/json$

[brainwane]

For those reading this who, like me, didn't know, VCL means "Varnish Configuration Language".

[brainwane]

Ernest is planning on finishing this on Monday.

[dstufft]

I went ahead and got the VCL sorted out to do this in https://github.com/python/pypi-infra/pull/12 and https://github.com/python/pypi-infra/pull/14.