alan-turing-institute / data-safe-haven

https://data-safe-haven.readthedocs.io

BSD 3-Clause "New" or "Revised" License

57 stars 14 forks source link

PyPI mirror keeps failing #278

Closed martintoreilly closed 5 years ago

martintoreilly commented 5 years ago

On two occasions connecting from the DSG9 compute VM to the PyPI mirror in the test SHM has failed with a ReadTimeoutError when trying to install packages via pip.

The first time, this was solved by deleting the internal mirror's package directory and restarting the rsync job to push the external mirror's packages over to the internal mirror.

jemrobinson commented 5 years ago

Possible cause

Unknown configuration/caching issues

Possible solution

Restart. The pypiserver service should already be configured to auto-run on restart

Test

Can the InternalMirror see its own packages using /pypiserver/bin/pip install a2svm --index http://localhost:3128
If it can then can a compute VM see packages on the mirror

Result

Failed at (1) with the following output:

atiadmin@MirrorVMInternalDSG1PyPI:~$ pypiserver/bin/pip install a2svm --index http://localhost:3128
Collecting a2svm
  Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='localhost', port=3128): Read timed out. (read timeout=15)",)': /simple/a2svm/
  Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='localhost', port=3128): Read timed out. (read timeout=15)",)': /simple/a2svm/
  Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='localhost', port=3128): Read timed out. (read timeout=15)",)': /simple/a2svm/
  Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='localhost', port=3128): Read timed out. (read timeout=15)",)': /simple/a2svm/
  Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='localhost', port=3128): Read timed out. (read timeout=15)",)': /simple/a2svaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=3128): Max retries exceeded with url: /simple/a2svm/ (Caused by ReadTimeoutError("HTTPConnectionPool(host='localhost', port=3128): Read timed out. (read timeout=15)",))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/pypiserver/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/pypiserver/lib/python3.6/site-packages/pip/commands/install.py", line 353, in run
    wb.build(autobuilding=True)
  File "/pypiserver/lib/python3.6/site-packages/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/pypiserver/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/pypiserver/lib/python3.6/site-packages/pip/req/req_set.py", line 554, in _prepare_file
    require_hashes
  File "/pypiserver/lib/python3.6/site-packages/pip/req/req_install.py", line 278, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/pypiserver/lib/python3.6/site-packages/pip/index.py", line 465, in find_requirement
    all_candidates = self.find_all_candidates(req.name)
  File "/pypiserver/lib/python3.6/site-packages/pip/index.py", line 423, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/pypiserver/lib/python3.6/site-packages/pip/index.py", line 568, in _get_pages
    page = self._get_page(location)
  File "/pypiserver/lib/python3.6/site-packages/pip/index.py", line 683, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/pypiserver/lib/python3.6/site-packages/pip/index.py", line 792, in get_page
    "Cache-Control": "max-age=600",
  File "/pypiserver/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/pypiserver/lib/python3.6/site-packages/pip/download.py", line 386, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/pypiserver/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/pypiserver/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/sessions.py", line 640, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/pypiserver/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/sessions.py", line 640, in <listcomp>
    history = [resp for resp in gen] if allow_redirects else []
  File "/pypiserver/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/sessions.py", line 218, in resolve_redirects
    **adapter_kwargs
  File "/pypiserver/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/pypiserver/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/adapters.py", line 508, in send
    raise ConnectionError(e, request=request)
pip._vendor.requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=3128): Max retries exceeded with url: /simple/a2svm/ (Caused by ReadTimeoutError("HTTPConnectionPool(host='localhost', port=3128): Readtimed out. (read timeout=15)",))

jemrobinson commented 5 years ago

Possible cause

From the pypiserver issue tracker, it seems that memory issues could cause this issue.

By default, pypiserver scans the entire packages directory each time an incoming HTTP request occurs. This isn't a problem for a small number of packages, but causes noticeable slow-downs when serving thousands of packages.

Source: https://github.com/pypiserver/pypiserver#serving-thousands-of-packages

Possible solution

More memory. Increase VM size: currently we use F4s_v2 which has 4 vCores and 8GB RAM. Test with F72s_v2 which has 72 vCores and 144 GB RAM.

Test

As above

Result

Timeout error still occurs.

atiadmin@MirrorVMInternalDSG1PyPI:~$ pypiserver/bin/pip install a2svm --index http://localhost:3128
Collecting a2svm
  Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='localhost', port=3128): Read timed out. (read timeout=15)",)': /simple/a2svm/

jemrobinson commented 5 years ago

Possible cause

Package search is taking too long

Possible solution

Use a package that will be found quickly. Try again with a package from the 00/00 directory. We will use sismic-0.22.7-py3-none-any.whl for this test

Test

/pypiserver/bin/pip install sismic --index http://localhost:3128

Result

Timeout error still occurs.

> http://localhost:3128rnalDSG1PyPI:~$ /pypiserver/bin/pip install sismic --index h

Collecting sismic
  Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='localhost', port=3128): Read timed out. (read timeout=15)",)': /sismic/

jemrobinson commented 5 years ago

Possible cause

From documentation

If you run into this problem, significant speedups can be gained by enabling pypiserver's directory caching functionality. The only requirement is to install the watchdog package, or it can be installed during pypiserver installation, by specifying the cache extras option: pip install pypiserver[cache]

Possible solution

Uninstall pypiserver and reinstall with caching enabled

[Add a temporary NSG rule allowing outbound access to pypi. Then, as root]
source /pypiserver/bin/activate
pip uninstall pypiserver
pip install pypiserver[cache]
systemctl restart pypiserver.service
systemctl status pypiserver.service

Test

/pypiserver/bin/pip install sismic --index http://localhost:3128

Result

HTTP error:

> atiadmin@MirrorVMInternalDSG1PyPI:~$ pypiserver/bin/pip install a2svm --index http://localhost:3128
The directory '/home/atiadmin/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, youmay want sudo's -H flag.
The directory '/home/atiadmin/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting sismic
Exception:
Traceback (most recent call last):
  File "/pypiserver/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/pypiserver/lib/python3.6/site-packages/pip/commands/install.py", line 342, in run
    requirement_set.prepare_files(finder)
  File "/pypiserver/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/pypiserver/lib/python3.6/site-packages/pip/req/req_set.py", line 554, in _prepare_file
    require_hashes
  File "/pypiserver/lib/python3.6/site-packages/pip/req/req_install.py", line 278, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/pypiserver/lib/python3.6/site-packages/pip/index.py", line 465, in find_requirement
    all_candidates = self.find_all_candidates(req.name)
  File "/pypiserver/lib/python3.6/site-packages/pip/index.py", line 423, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/pypiserver/lib/python3.6/site-packages/pip/index.py", line 568, in _get_pages
    page = self._get_page(location)
  File "/pypiserver/lib/python3.6/site-packages/pip/index.py", line 683, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/pypiserver/lib/python3.6/site-packages/pip/index.py", line 795, in get_page
    resp.raise_for_status()
  File "/pypiserver/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
pip._vendor.requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://localhost:3128/simple/sismic/

jemrobinson commented 5 years ago

Possible cause

Webserver isn't configured correctly

Possible solution

Try to curl a page that should be on the server

Test

curl http://localhost:3128/simple/sismic/ --verbose

Result

Page hangs (http://pypi.org returns the a page that looks like the static index.html page that is in the simple/sismic directory, but 404s when we ask for the page directly)

*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 3128 (#0)
> GET /simple/sismic/ HTTP/1.1
> Host: localhost:3128
> User-Agent: curl/7.58.0
> Accept: */*
>

jemrobinson commented 5 years ago

Possible cause

Built-in webserver can't cope.

Possible solution

Switch to Apache. From the Pypiserver docs, which references the bottle docs.

To use your Apache2 with pypiserver, prefer to utilize mod_wsgi as explained in bottle's documentation.

Test

Install Apache with: sudo apt-get install apache2
Ensure a dedicated pypisrv userpypisrv` and group for the PyPI server to run under

Configure Apache (adapted from Apache section of pypiserver docs with umask changed from 0007 to 0022)

Edit /etc/apache2/sites-enabled/000-default.conf to be as follows:

<VirtualHost *:3128>
    DocumentRoot /datadrive/mirrordaemon/pypi/web
    ErrorLog /datadrive/mirrordaemon/apache2_error.log=
    CustomLog /datadrive/mirrordaemon/apache2_access.log combined

    WSGIScriptAlias    /    /wsgi/pypiserver-wsgi.py
    WSGIDaemonProcess    pypisrv user=pypisrv group=pypisrv umask=0027 \
                                             processes=1 threads=5 maximum-requests=500 \
                                             display-name=wsgi-pypisrv inactivity-timeout=300
    WSGIProcessGroup        pypisrv
    WSGIPassAuthorization On    ## (Optional) Use also apache's authentication.

    <Directory /wsgi >
        Require all granted
    </Directory>
</VirtualHost>

Create the /datadrive/mirrordaemon/pypi/web/wsgi/pypiserver.wsgi file with contents as below:

import pypiserver

conf = pypiserver.default_config(
    root = "/packages",
    port=3128,
    password_file = "/htpasswd", )
application = pypiserver.app(**conf)

Make sure that the user and group of the WSGIDaemonProcess directive (pypisrv:pypisrv in the example) have read permission on it (updated to read and execute): sudo chown root:pypisrv /datadrive/mirrordaemon/pypi/web/wsgi/pypiserver.wsgi sudo chmod ug=rx,o= /datadrive/mirrordaemon/pypi/web/wsgi/pypiserver.wsgi

Result

Test not yet run

jemrobinson commented 5 years ago

Possible cause

Misconfiguration of pypiserver.service. In 42039465d ExecStart and WorkingDirectory became out of sync. File reads as follows (cat /etc/systemd/system/pypiserver.service)

Description=A minimal PyPI server for use with pip/easy_install.
After=network.target

[Service]
Type=simple
PIDFile=/var/run/pypiserver.pid
User=mirrordaemon
Group=mirrordaemon

ExecStart=/pypiserver/bin/pypi-server -p 3128 -P . -a . --log-file /datadrive/mirrordaemon/pypiserver.log /datadrive/mirrordaemon/pypi/web/packages
ExecStop=/bin/kill -TERM $MAINPID
ExecReload=/bin/kill -HUP $MAINPID
Restart=always

WorkingDirectory=/datadrive/mirrordaemon/pypi

TimeoutStartSec=3
RestartSec=5

[Install]
WantedBy=multi-user.target

Possible solution

Put them back in sync, testing each of /datadrive/mirrordaemon/pypi/web/packages, /datadrive/mirrordaemon/pypi/web/ and /datadrive/mirrordaemon/pypi/

Test A:

Use /datadrive/mirrordaemon/pypi/web/packages.

sudo systemctl daemon-reload
sudo systemctl restart pypiserver.service
sudo systemctl status pypiserver.service
/pypiserver/bin/pip install sismic --index http://localhost:3128

Result A

Failure

atiadmin@MirrorVMInternalDSG1PyPI:~$ /pypiserver/bin/pip install sismic --index http://localhost:3128
[...]
pip._vendor.requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://localhost:3128/simple/sismic/

Test B:

Use /datadrive/mirrordaemon/pypi/web.

sudo systemctl daemon-reload
sudo systemctl restart pypiserver.service
sudo systemctl status pypiserver.service
/pypiserver/bin/pip install sismic --index http://localhost:3128

Result B

Failure

atiadmin@MirrorVMInternalDSG1PyPI:~$ /pypiserver/bin/pip install sismic --index http://localhost:3128
[...]
pip._vendor.requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://localhost:3128/simple/sismic/

Test C:

Use /datadrive/mirrordaemon/pypi.

sudo systemctl daemon-reload
sudo systemctl restart pypiserver.service
sudo systemctl status pypiserver.service
/pypiserver/bin/pip install sismic --index http://localhost:3128

Result C

Failure

atiadmin@MirrorVMInternalDSG1PyPI:~$ /pypiserver/bin/pip install sismic --index http://localhost:3128
[...]
pip._vendor.requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://localhost:3128/simple/sismic/

jemrobinson commented 5 years ago

Possible cause

systemctl service is set incorrectly

Possible solution

Test pypi-server directly.

Test A:

[first uninstall pypiserver[cache] and watchdog, then reinstall pypiserver]

[as atiadmin]
/pypiserver/bin/pypi-server -p 8080 /datadrive/mirrordaemon/pypi/web &
curl http://localhost:8080/simple/sismic/ --verbose

Result

Success

*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /simple/sismic/ HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.58.0
> Accept: */*
>
*127.0.0.1 - - [03/Apr/2019 12:02:13] "GET /simple/sismic/ HTTP/1.1" 200 611
 HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Date: Wed, 03 Apr 2019 12:02:13 GMT
< Server: WSGIServer/0.2 CPython/3.6.7
< Content-Length: 611
< Content-Type: text/html; charset=UTF-8
<
    <html>
        <head>
            <title>Links for sismic</title>
        </head>
        <body>
            <h1>Links for sismic</h1>
                 <a href="/packages/packages/00/53/320c853e31823badeb5dd1000c6dc29e228320fe84c5fd5a6b5df765e9cc/sismic-0.15.0-py3-none-any.whl#md5=88c724c9e94d8fe3f2674392b9becc57">sismic-0.15.0-py3-none-any.whl</a><br>
                 <a href="/packages/packages/00/00/11993e25347ac141e9c7977b5c3594a7a5318388621eb6f5c3fa099f9308/sismic-0.22.7-py3-none-any.whl#md5=36feeb9c836c788f4c7254176a438540">sismic-0.22.7-py3-none-any.whl</a><br>
        </body>
    </html>
* Closing connection 0

Test /pypiserver/bin/pip install sismic --index http://localhost:8080

Collecting sismic
127.0.0.1 - - [03/Apr/2019 12:06:46] "GET /sismic/ HTTP/1.1" 303 0
127.0.0.1 - - [03/Apr/2019 12:06:47] "GET /simple/sismic/ HTTP/1.1" 200 611
127.0.0.1 - - [03/Apr/2019 12:06:47] "GET /packages/packages/00/00/11993e25347ac141e9c7977b5c3594a7a5318388621eb6f5c3fa099f9308/sismic-0.22.7-py3-none-any.whl HTTP/1.1" 200 44609
  Downloading http://localhost:8080/packages/packages/00/00/11993e25347ac141e9c7977b5c3594a7a5318388621eb6f5c3fa099f9308/sismic-0.22.7-py3-none-any.whl (44kB)
    100% |████████████████████████████████| 51kB 24.1MB/s
Collecting typing>=3.5.1 (from sismic)
127.0.0.1 - - [03/Apr/2019 12:06:47] "GET /typing/ HTTP/1.1" 303 0
127.0.0.1 - - [03/Apr/2019 12:06:47] "GET /simple/typing/ HTTP/1.1" 303 0
  Using cached https://files.pythonhosted.org/packages/4a/bd/eee1157fc2d8514970b345d69cb9975dcd1e42cd7e61146ed841f6e68309/typing-3.6.6-py3-none-any.whl
Collecting pykwalify>=1.5.1 (from sismic)
127.0.0.1 - - [03/Apr/2019 12:06:47] "GET /pykwalify/ HTTP/1.1" 303 0
127.0.0.1 - - [03/Apr/2019 12:06:48] "GET /simple/pykwalify/ HTTP/1.1" 303 0
  Downloading https://files.pythonhosted.org/packages/36/9f/612de8ca540bd24d604f544248c4c46e9db76f6ea5eb75fb4244da6ebbf0/pykwalify-1.7.0-py2.py3-none-any.whl (40kB)
    100% |████████████████████████████████| 40kB 10.3MB/s
Collecting pyparsing>=2.1.1 (from sismic)
127.0.0.1 - - [03/Apr/2019 12:06:48] "GET /pyparsing/ HTTP/1.1" 303 0
127.0.0.1 - - [03/Apr/2019 12:06:48] "GET /simple/pyparsing/ HTTP/1.1" 200 621
  Could not find a version that satisfies the requirement pyparsing>=2.1.1 (from sismic) (from versions: )
No matching distribution found for pyparsing>=2.1.1 (from sismic)

Test B:

[as mirrordaemon]
/pypiserver/bin/pypi-server -p 8080 /datadrive/mirrordaemon/pypi/web &
curl http://localhost:8080/simple/sismic/ --verbose

Result

Failure: curl hangs

jemrobinson commented 5 years ago

Possible cause

Using ExecStart=/datadrive/mirrordaemon/pypi/web test different values of WorkingDirectory in /etc/systemd/system/pypiserver.service

Possible solution

Test each of /datadrive/mirrordaemon/pypi/web/packages, /datadrive/mirrordaemon/pypi/web/ and /datadrive/mirrordaemon/pypi/

Test A:

Use /datadrive/mirrordaemon/pypi/web/packages.

sudo systemctl daemon-reload
sudo systemctl restart pypiserver.service
sudo systemctl status pypiserver.service
curl http://localhost:3128/simple/sismic/ --verbose
/pypiserver/bin/pip install sismic --index http://localhost:3128

Result A

Failure: curl hangs

Test B:

Use /datadrive/mirrordaemon/pypi/web.

sudo systemctl daemon-reload
sudo systemctl restart pypiserver.service
sudo systemctl status pypiserver.service
curl http://localhost:3128/simple/sismic/ --verbose
/pypiserver/bin/pip install sismic --index http://localhost:3128

Result B

Failure: curl hangs

Test C:

Use /datadrive/mirrordaemon/pypi.

sudo systemctl daemon-reload
sudo systemctl restart pypiserver.service
sudo systemctl status pypiserver.service
curl http://localhost:3128/simple/sismic/ --verbose
/pypiserver/bin/pip install sismic --index http://localhost:3128

Result C

Failure: curl hangs

jemrobinson commented 5 years ago

Possible cause

Permissions on /datadrive since a local pypi-server instance from atiadmin worked, but one from mirrordaemon did not

Possible solution

Add a new user pypisrv and see whether they can start a pypi-server instance.

Test A:

sudo adduser pypisrv
groups pypisrv
su pypisrv
/pypiserver/bin/pypi-server -p 8080 /datadrive/mirrordaemon/pypi/web &
curl http://localhost:8080/simple/sismic/ --verbose
/pypiserver/bin/pip install sismic --index http://localhost:8080

Result A

curl works
pip works

Is the fact that atiadmin can't see /datadrive/mirrordaemon/pypi/web/simple an advantage?

Test B:

Rename /datadrive/mirrordaemon/pypi/web/simple to /datadrive/mirrordaemon/pypi/web/simple_full and see whether files are served on 3128

sudo mv /datadrive/mirrordaemon/pypi/web/simple /datadrive/mirrordaemon/pypi/web/simple_full
ls -alh /datadrive/mirrordaemon/pypi/web/
sudo systemctl daemon-reload
sudo systemctl restart pypiserver.service
sudo systemctl status pypiserver.service
curl http://localhost:3128/simple/sismic/ --verbose
/pypiserver/bin/pip install sismic --index http://localhost:3128

Result

curl still hangs => change directory name back

Test C:

Make pypisrv the user in /etc/systemd/system/pypiserver.service

sudo vi /etc/systemd/system/pypiserver.service
sudo systemctl daemon-reload
sudo systemctl restart pypiserver.service
sudo systemctl status pypiserver.service
curl http://localhost:3128/simple/sismic/ --verbose
/pypiserver/bin/pip install sismic --index http://localhost:3128

jemrobinson commented 5 years ago

Possible cause

Loading the package list is slow because pypi-server has to read every directory in order to generate it.

With watchdog this might only have to happen once (per instance of pypi-server including a reset if the service is restarted).

Possible solution

Set up pypiserver with caching (again) and leave an update job running for a long time. NB. also reset the user to mirrordaemon.

Test

NB. no [cache] as this breaks search irreversibly

source /pypiserver/bin/activate
pip uninstall pypiserver
pip install pypiserver
cd /pypiserver/lib/python3.6/site-packages/pypiserver
patch < /pypiserver.app.py.patch
cd /
systemctl restart pypiserver.service
systemctl status pypiserver.service

Debug code.py and cache.py in pypiserver using

sudo systemctl daemon-reload
sudo systemctl restart pypiserver.service
sudo systemctl status pypiserver.service
curl http://localhost:3128/simple/sismic/ --verbose &
tail -f /datadrive/mirrordaemon/pypiserver.log

vollmersj commented 5 years ago

@jemrobinson @martintoreilly Are you using bandersnatch? Also there is a way to blacklist large packages (most of them are just pretrained deepnets with millions of parameters) - I think pypi stats allows you to black list by ussage

jemrobinson commented 5 years ago

SOLVED

However, interactive patching is needed for the PyPI internal mirror in production. The steps needed are as follows

Log into the mirror (easiest with the Serial console in the Azure portal)
Upload the patch script to /pypiserver.app.py.patch (small enough to be done with copy & paste)
Run the following commands

sudo bash # to become root
source /pypiserver/bin/activate
cd /pypiserver/lib/python3.6/site-packages/pypiserver
cp _app.py /_app.py.bak
patch < /pypiserver.app.py.patch
systemctl restart pypiserver.service
systemctl status pypiserver.service

We should also update the external mirrors with the new rsync settings: in /home/mirrordaemon/push_to_internal_mirrors.sh make the following change (adding a p)

rsync -rtlzv --delete --progress /datadrive/mirrordaemon/pypi/* mirrordaemon@${IP_ADDRESS}:/datadrive/mirrordaemon/pypi

rsync -prtlzv --delete --progress /datadrive/mirrordaemon/pypi/* mirrordaemon@${IP_ADDRESS}:/datadrive/mirrordaemon/pypi

This should be tested from DSG10

jemrobinson commented 5 years ago

@vollmersj : yes, we do use bandersnatch to populate the initial (external) mirror with all packages from PyPI. We could consider blacklisting specific packages in future, but the problem here isn't about the size of any individual package, it's about the number of them. If we start restricting the available packages too much then we're not really fulfilling the stated aim of Tier-2 environments which is that you can install whatever packages you want to.

jemrobinson commented 5 years ago

Patching production

patch applied with no problems
found that internal sync was incomplete (nothing in the /simple directory)
forcing new push from PyPI external mirror:

[update /home/mirrordaemon/push_to_internal_mirrors.sh]
sudo -u mirrordaemon bash
source ~mirrordaemon/push_to_internal_mirrors.sh &

also ran the same thing for CRAN

Rerunning push to internal

this is taking a long time (as not very much had been completed before the initial kernel panic)
2019-04-04 1330: started
2019-04-05 1830 synchronised up to directory /4e/ out of /ff/ [approx 30% complete after 3hrs] (NB. this includes an unknown percentage that was already complete)
2019-04-05 1000 synchronised up to directory /a9/ out of /ff/ [approx 66% complete after 20.5hrs] => 37hrs total
2019-04-05 1420 synchronised up to directory /c3/ out of /ff/ [approx 76% complete after 25hrs] => 36.5hrs total
2019-04-05 1530 synchronised up to directory /ca/ out of /ff/ [approx 79% complete after 26hrs] => 36hrs total

current estimated end time: `2019-04-06 0130` [after 36hrs]

martintoreilly commented 5 years ago

See my comment in issue #285 for what I think is a successful test from DSG10 that the production internal PyPI mirror is up to date with its external mirror.

martintoreilly commented 5 years ago

Comment moved to issue #285, which is dealing with mirror synchronisation reliability.

@jemrobinson How confident are we that the bandersnatch synchronisation from the public PyPI site to the external mirror is complete?

If it is more reliable than rsync, then could we also use it for our external->internal mirror sync?

martintoreilly commented 5 years ago

Comment moved to issue #285, which is dealing with mirror synchronisation reliability.

I note that bandersnatch also supports a blacklist and whitelist though I guess it is designed to be run from the internal mirror in our arrangement, which we would not want.

jemrobinson commented 5 years ago

Confirming that:

[x] pip search matplotlib gives expected output
[x] pip install sismic fails due to permission errors
[x] pip install --user sismic works
[x] pip install --user uncertainties works

on the 10.250.74.160 machine in DSG10

martintoreilly commented 5 years ago

@getcarter21 Please can your team test that pip install --user sismic works on DSGs 11-15? Thanks.

martintoreilly commented 5 years ago

@jemrobinson Can you open a PR to master with the fix for this, tagging this issue to autoclose when merged?

jemrobinson commented 5 years ago

Mostly done in #281. Additional speed-up fixes by removing compression will go into a new PR.

alan-turing-institute / data-safe-haven

PyPI mirror keeps failing #278

Possible cause

Possible solution

Test

Result

Possible cause

Possible solution

Test

Result

Possible cause

Possible solution

Test

Result

Possible cause

Possible solution

Test

Result

Possible cause

Possible solution

Test

Result

Possible cause

Possible solution

Test

Result

Possible cause

Possible solution

Test A:

Result A

Test B:

Result B

Test C:

Result C

Possible cause

Possible solution

Test A:

Result

Test B:

Result

Possible cause

Possible solution

Test A:

Result A

Test B:

Result B

Test C:

Result C

Possible cause

Possible solution

Test A:

Result A

Test B:

Result

Test C:

Possible cause

Possible solution

Test

SOLVED

Patching production

Rerunning push to internal

current estimated end time: 2019-04-06 0130 [after 36hrs]

current estimated end time: `2019-04-06 0130` [after 36hrs]