Closed martintoreilly closed 5 years ago
Unknown configuration/caching issues
Restart. The pypiserver
service should already be configured to auto-run on restart
/pypiserver/bin/pip install a2svm --index http://localhost:3128
Failed at (1) with the following output:
atiadmin@MirrorVMInternalDSG1PyPI:~$ pypiserver/bin/pip install a2svm --index http://localhost:3128
Collecting a2svm
Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='localhost', port=3128): Read timed out. (read timeout=15)",)': /simple/a2svm/
Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='localhost', port=3128): Read timed out. (read timeout=15)",)': /simple/a2svm/
Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='localhost', port=3128): Read timed out. (read timeout=15)",)': /simple/a2svm/
Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='localhost', port=3128): Read timed out. (read timeout=15)",)': /simple/a2svm/
Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='localhost', port=3128): Read timed out. (read timeout=15)",)': /simple/a2svaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=3128): Max retries exceeded with url: /simple/a2svm/ (Caused by ReadTimeoutError("HTTPConnectionPool(host='localhost', port=3128): Read timed out. (read timeout=15)",))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/pypiserver/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/pypiserver/lib/python3.6/site-packages/pip/commands/install.py", line 353, in run
wb.build(autobuilding=True)
File "/pypiserver/lib/python3.6/site-packages/pip/wheel.py", line 749, in build
self.requirement_set.prepare_files(self.finder)
File "/pypiserver/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/pypiserver/lib/python3.6/site-packages/pip/req/req_set.py", line 554, in _prepare_file
require_hashes
File "/pypiserver/lib/python3.6/site-packages/pip/req/req_install.py", line 278, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "/pypiserver/lib/python3.6/site-packages/pip/index.py", line 465, in find_requirement
all_candidates = self.find_all_candidates(req.name)
File "/pypiserver/lib/python3.6/site-packages/pip/index.py", line 423, in find_all_candidates
for page in self._get_pages(url_locations, project_name):
File "/pypiserver/lib/python3.6/site-packages/pip/index.py", line 568, in _get_pages
page = self._get_page(location)
File "/pypiserver/lib/python3.6/site-packages/pip/index.py", line 683, in _get_page
return HTMLPage.get_page(link, session=self.session)
File "/pypiserver/lib/python3.6/site-packages/pip/index.py", line 792, in get_page
"Cache-Control": "max-age=600",
File "/pypiserver/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/sessions.py", line 521, in get
return self.request('GET', url, **kwargs)
File "/pypiserver/lib/python3.6/site-packages/pip/download.py", line 386, in request
return super(PipSession, self).request(method, url, *args, **kwargs)
File "/pypiserver/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/pypiserver/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/sessions.py", line 640, in send
history = [resp for resp in gen] if allow_redirects else []
File "/pypiserver/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/sessions.py", line 640, in <listcomp>
history = [resp for resp in gen] if allow_redirects else []
File "/pypiserver/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/sessions.py", line 218, in resolve_redirects
**adapter_kwargs
File "/pypiserver/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "/pypiserver/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/adapters.py", line 508, in send
raise ConnectionError(e, request=request)
pip._vendor.requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=3128): Max retries exceeded with url: /simple/a2svm/ (Caused by ReadTimeoutError("HTTPConnectionPool(host='localhost', port=3128): Readtimed out. (read timeout=15)",))
From the pypiserver issue tracker, it seems that memory issues could cause this issue.
By default, pypiserver scans the entire packages directory each time an incoming HTTP request occurs. This isn't a problem for a small number of packages, but causes noticeable slow-downs when serving thousands of packages.
Source: https://github.com/pypiserver/pypiserver#serving-thousands-of-packages
More memory. Increase VM size: currently we use F4s_v2
which has 4 vCores and 8GB RAM. Test with F72s_v2
which has 72 vCores and 144 GB RAM.
As above
Timeout error still occurs.
atiadmin@MirrorVMInternalDSG1PyPI:~$ pypiserver/bin/pip install a2svm --index http://localhost:3128
Collecting a2svm
Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='localhost', port=3128): Read timed out. (read timeout=15)",)': /simple/a2svm/
Package search is taking too long
Use a package that will be found quickly. Try again with a package from the 00/00
directory. We will use sismic-0.22.7-py3-none-any.whl
for this test
/pypiserver/bin/pip install sismic --index http://localhost:3128
Timeout error still occurs.
> http://localhost:3128rnalDSG1PyPI:~$ /pypiserver/bin/pip install sismic --index h
Collecting sismic
Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='localhost', port=3128): Read timed out. (read timeout=15)",)': /sismic/
From documentation
If you run into this problem, significant speedups can be gained by enabling pypiserver's directory caching functionality. The only requirement is to install the watchdog package, or it can be installed during pypiserver installation, by specifying the cache extras option:
pip install pypiserver[cache]
Uninstall pypiserver
and reinstall with caching enabled
[Add a temporary NSG rule allowing outbound access to pypi. Then, as root]
source /pypiserver/bin/activate
pip uninstall pypiserver
pip install pypiserver[cache]
systemctl restart pypiserver.service
systemctl status pypiserver.service
/pypiserver/bin/pip install sismic --index http://localhost:3128
HTTP error:
> atiadmin@MirrorVMInternalDSG1PyPI:~$ pypiserver/bin/pip install a2svm --index http://localhost:3128
The directory '/home/atiadmin/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, youmay want sudo's -H flag.
The directory '/home/atiadmin/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting sismic
Exception:
Traceback (most recent call last):
File "/pypiserver/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/pypiserver/lib/python3.6/site-packages/pip/commands/install.py", line 342, in run
requirement_set.prepare_files(finder)
File "/pypiserver/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/pypiserver/lib/python3.6/site-packages/pip/req/req_set.py", line 554, in _prepare_file
require_hashes
File "/pypiserver/lib/python3.6/site-packages/pip/req/req_install.py", line 278, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "/pypiserver/lib/python3.6/site-packages/pip/index.py", line 465, in find_requirement
all_candidates = self.find_all_candidates(req.name)
File "/pypiserver/lib/python3.6/site-packages/pip/index.py", line 423, in find_all_candidates
for page in self._get_pages(url_locations, project_name):
File "/pypiserver/lib/python3.6/site-packages/pip/index.py", line 568, in _get_pages
page = self._get_page(location)
File "/pypiserver/lib/python3.6/site-packages/pip/index.py", line 683, in _get_page
return HTMLPage.get_page(link, session=self.session)
File "/pypiserver/lib/python3.6/site-packages/pip/index.py", line 795, in get_page
resp.raise_for_status()
File "/pypiserver/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
pip._vendor.requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://localhost:3128/simple/sismic/
Webserver isn't configured correctly
Try to curl
a page that should be on the server
curl http://localhost:3128/simple/sismic/ --verbose
Page hangs (http://pypi.org
returns the a page that looks like the static index.html
page that is in the simple/sismic
directory, but 404s when we ask for the page directly)
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 3128 (#0)
> GET /simple/sismic/ HTTP/1.1
> Host: localhost:3128
> User-Agent: curl/7.58.0
> Accept: */*
>
Built-in webserver can't cope.
Switch to Apache. From the Pypiserver docs, which references the bottle docs.
To use your Apache2 with pypiserver, prefer to utilize mod_wsgi as explained in bottle's documentation.
sudo apt-get install apache2
pypisrv user
pypisrv` and group for the PyPI server to run underConfigure Apache (adapted from Apache section of pypiserver docs with umask changed from 0007
to 0022
)
/etc/apache2/sites-enabled/000-default.conf
to be as follows:<VirtualHost *:3128>
DocumentRoot /datadrive/mirrordaemon/pypi/web
ErrorLog /datadrive/mirrordaemon/apache2_error.log=
CustomLog /datadrive/mirrordaemon/apache2_access.log combined
WSGIScriptAlias / /wsgi/pypiserver-wsgi.py
WSGIDaemonProcess pypisrv user=pypisrv group=pypisrv umask=0027 \
processes=1 threads=5 maximum-requests=500 \
display-name=wsgi-pypisrv inactivity-timeout=300
WSGIProcessGroup pypisrv
WSGIPassAuthorization On ## (Optional) Use also apache's authentication.
<Directory /wsgi >
Require all granted
</Directory>
</VirtualHost>
/datadrive/mirrordaemon/pypi/web/wsgi/pypiserver.wsgi
file with contents as below:import pypiserver
conf = pypiserver.default_config(
root = "/packages",
port=3128,
password_file = "/htpasswd", )
application = pypiserver.app(**conf)
sudo chown root:pypisrv /datadrive/mirrordaemon/pypi/web/wsgi/pypiserver.wsgi
sudo chmod ug=rx,o= /datadrive/mirrordaemon/pypi/web/wsgi/pypiserver.wsgi
Test not yet run
Misconfiguration of pypiserver.service
. In 42039465d ExecStart
and WorkingDirectory
became out of sync. File reads as follows (cat /etc/systemd/system/pypiserver.service
)
Description=A minimal PyPI server for use with pip/easy_install.
After=network.target
[Service]
Type=simple
PIDFile=/var/run/pypiserver.pid
User=mirrordaemon
Group=mirrordaemon
ExecStart=/pypiserver/bin/pypi-server -p 3128 -P . -a . --log-file /datadrive/mirrordaemon/pypiserver.log /datadrive/mirrordaemon/pypi/web/packages
ExecStop=/bin/kill -TERM $MAINPID
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
WorkingDirectory=/datadrive/mirrordaemon/pypi
TimeoutStartSec=3
RestartSec=5
[Install]
WantedBy=multi-user.target
Put them back in sync, testing each of /datadrive/mirrordaemon/pypi/web/packages
, /datadrive/mirrordaemon/pypi/web/
and /datadrive/mirrordaemon/pypi/
Use /datadrive/mirrordaemon/pypi/web/packages
.
sudo systemctl daemon-reload
sudo systemctl restart pypiserver.service
sudo systemctl status pypiserver.service
/pypiserver/bin/pip install sismic --index http://localhost:3128
Failure
atiadmin@MirrorVMInternalDSG1PyPI:~$ /pypiserver/bin/pip install sismic --index http://localhost:3128
[...]
pip._vendor.requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://localhost:3128/simple/sismic/
Use /datadrive/mirrordaemon/pypi/web
.
sudo systemctl daemon-reload
sudo systemctl restart pypiserver.service
sudo systemctl status pypiserver.service
/pypiserver/bin/pip install sismic --index http://localhost:3128
Failure
atiadmin@MirrorVMInternalDSG1PyPI:~$ /pypiserver/bin/pip install sismic --index http://localhost:3128
[...]
pip._vendor.requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://localhost:3128/simple/sismic/
Use /datadrive/mirrordaemon/pypi
.
sudo systemctl daemon-reload
sudo systemctl restart pypiserver.service
sudo systemctl status pypiserver.service
/pypiserver/bin/pip install sismic --index http://localhost:3128
Failure
atiadmin@MirrorVMInternalDSG1PyPI:~$ /pypiserver/bin/pip install sismic --index http://localhost:3128
[...]
pip._vendor.requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://localhost:3128/simple/sismic/
systemctl
service is set incorrectly
Test pypi-server directly.
[first uninstall pypiserver[cache] and watchdog, then reinstall pypiserver]
[as atiadmin]
/pypiserver/bin/pypi-server -p 8080 /datadrive/mirrordaemon/pypi/web &
curl http://localhost:8080/simple/sismic/ --verbose
Success
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /simple/sismic/ HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.58.0
> Accept: */*
>
*127.0.0.1 - - [03/Apr/2019 12:02:13] "GET /simple/sismic/ HTTP/1.1" 200 611
HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Date: Wed, 03 Apr 2019 12:02:13 GMT
< Server: WSGIServer/0.2 CPython/3.6.7
< Content-Length: 611
< Content-Type: text/html; charset=UTF-8
<
<html>
<head>
<title>Links for sismic</title>
</head>
<body>
<h1>Links for sismic</h1>
<a href="/packages/packages/00/53/320c853e31823badeb5dd1000c6dc29e228320fe84c5fd5a6b5df765e9cc/sismic-0.15.0-py3-none-any.whl#md5=88c724c9e94d8fe3f2674392b9becc57">sismic-0.15.0-py3-none-any.whl</a><br>
<a href="/packages/packages/00/00/11993e25347ac141e9c7977b5c3594a7a5318388621eb6f5c3fa099f9308/sismic-0.22.7-py3-none-any.whl#md5=36feeb9c836c788f4c7254176a438540">sismic-0.22.7-py3-none-any.whl</a><br>
</body>
</html>
* Closing connection 0
Test /pypiserver/bin/pip install sismic --index http://localhost:8080
Collecting sismic
127.0.0.1 - - [03/Apr/2019 12:06:46] "GET /sismic/ HTTP/1.1" 303 0
127.0.0.1 - - [03/Apr/2019 12:06:47] "GET /simple/sismic/ HTTP/1.1" 200 611
127.0.0.1 - - [03/Apr/2019 12:06:47] "GET /packages/packages/00/00/11993e25347ac141e9c7977b5c3594a7a5318388621eb6f5c3fa099f9308/sismic-0.22.7-py3-none-any.whl HTTP/1.1" 200 44609
Downloading http://localhost:8080/packages/packages/00/00/11993e25347ac141e9c7977b5c3594a7a5318388621eb6f5c3fa099f9308/sismic-0.22.7-py3-none-any.whl (44kB)
100% |████████████████████████████████| 51kB 24.1MB/s
Collecting typing>=3.5.1 (from sismic)
127.0.0.1 - - [03/Apr/2019 12:06:47] "GET /typing/ HTTP/1.1" 303 0
127.0.0.1 - - [03/Apr/2019 12:06:47] "GET /simple/typing/ HTTP/1.1" 303 0
Using cached https://files.pythonhosted.org/packages/4a/bd/eee1157fc2d8514970b345d69cb9975dcd1e42cd7e61146ed841f6e68309/typing-3.6.6-py3-none-any.whl
Collecting pykwalify>=1.5.1 (from sismic)
127.0.0.1 - - [03/Apr/2019 12:06:47] "GET /pykwalify/ HTTP/1.1" 303 0
127.0.0.1 - - [03/Apr/2019 12:06:48] "GET /simple/pykwalify/ HTTP/1.1" 303 0
Downloading https://files.pythonhosted.org/packages/36/9f/612de8ca540bd24d604f544248c4c46e9db76f6ea5eb75fb4244da6ebbf0/pykwalify-1.7.0-py2.py3-none-any.whl (40kB)
100% |████████████████████████████████| 40kB 10.3MB/s
Collecting pyparsing>=2.1.1 (from sismic)
127.0.0.1 - - [03/Apr/2019 12:06:48] "GET /pyparsing/ HTTP/1.1" 303 0
127.0.0.1 - - [03/Apr/2019 12:06:48] "GET /simple/pyparsing/ HTTP/1.1" 200 621
Could not find a version that satisfies the requirement pyparsing>=2.1.1 (from sismic) (from versions: )
No matching distribution found for pyparsing>=2.1.1 (from sismic)
[as mirrordaemon]
/pypiserver/bin/pypi-server -p 8080 /datadrive/mirrordaemon/pypi/web &
curl http://localhost:8080/simple/sismic/ --verbose
Failure: curl
hangs
Using ExecStart=/datadrive/mirrordaemon/pypi/web
test different values of WorkingDirectory
in /etc/systemd/system/pypiserver.service
Test each of /datadrive/mirrordaemon/pypi/web/packages
, /datadrive/mirrordaemon/pypi/web/
and /datadrive/mirrordaemon/pypi/
Use /datadrive/mirrordaemon/pypi/web/packages
.
sudo systemctl daemon-reload
sudo systemctl restart pypiserver.service
sudo systemctl status pypiserver.service
curl http://localhost:3128/simple/sismic/ --verbose
/pypiserver/bin/pip install sismic --index http://localhost:3128
Failure: curl
hangs
Use /datadrive/mirrordaemon/pypi/web
.
sudo systemctl daemon-reload
sudo systemctl restart pypiserver.service
sudo systemctl status pypiserver.service
curl http://localhost:3128/simple/sismic/ --verbose
/pypiserver/bin/pip install sismic --index http://localhost:3128
Failure: curl
hangs
Use /datadrive/mirrordaemon/pypi
.
sudo systemctl daemon-reload
sudo systemctl restart pypiserver.service
sudo systemctl status pypiserver.service
curl http://localhost:3128/simple/sismic/ --verbose
/pypiserver/bin/pip install sismic --index http://localhost:3128
Failure: curl
hangs
Permissions on /datadrive
since a local pypi-server
instance from atiadmin
worked, but one from mirrordaemon
did not
Add a new user pypisrv
and see whether they can start a pypi-server
instance.
sudo adduser pypisrv
groups pypisrv
su pypisrv
/pypiserver/bin/pypi-server -p 8080 /datadrive/mirrordaemon/pypi/web &
curl http://localhost:8080/simple/sismic/ --verbose
/pypiserver/bin/pip install sismic --index http://localhost:8080
curl
workspip
worksIs the fact that atiadmin
can't see /datadrive/mirrordaemon/pypi/web/simple
an advantage?
Rename /datadrive/mirrordaemon/pypi/web/simple
to /datadrive/mirrordaemon/pypi/web/simple_full
and see whether files are served on 3128
sudo mv /datadrive/mirrordaemon/pypi/web/simple /datadrive/mirrordaemon/pypi/web/simple_full
ls -alh /datadrive/mirrordaemon/pypi/web/
sudo systemctl daemon-reload
sudo systemctl restart pypiserver.service
sudo systemctl status pypiserver.service
curl http://localhost:3128/simple/sismic/ --verbose
/pypiserver/bin/pip install sismic --index http://localhost:3128
curl
still hangs => change directory name back
Make pypisrv
the user in /etc/systemd/system/pypiserver.service
sudo vi /etc/systemd/system/pypiserver.service
sudo systemctl daemon-reload
sudo systemctl restart pypiserver.service
sudo systemctl status pypiserver.service
curl http://localhost:3128/simple/sismic/ --verbose
/pypiserver/bin/pip install sismic --index http://localhost:3128
Loading the package list is slow because pypi-server
has to read every directory in order to generate it.
With watchdog
this might only have to happen once (per instance of pypi-server
including a reset if the service is restarted).
Set up pypiserver with caching (again) and leave an update job running for a long time. NB. also reset the user to mirrordaemon.
NB. no [cache] as this breaks search irreversibly
source /pypiserver/bin/activate
pip uninstall pypiserver
pip install pypiserver
cd /pypiserver/lib/python3.6/site-packages/pypiserver
patch < /pypiserver.app.py.patch
cd /
systemctl restart pypiserver.service
systemctl status pypiserver.service
Debug code.py
and cache.py
in pypiserver
using
sudo systemctl daemon-reload
sudo systemctl restart pypiserver.service
sudo systemctl status pypiserver.service
curl http://localhost:3128/simple/sismic/ --verbose &
tail -f /datadrive/mirrordaemon/pypiserver.log
@jemrobinson @martintoreilly Are you using bandersnatch? Also there is a way to blacklist large packages (most of them are just pretrained deepnets with millions of parameters) - I think pypi stats allows you to black list by ussage
However, interactive patching is needed for the PyPI internal
mirror in production. The steps needed are as follows
/pypiserver.app.py.patch
(small enough to be done with copy & paste)sudo bash # to become root
source /pypiserver/bin/activate
cd /pypiserver/lib/python3.6/site-packages/pypiserver
cp _app.py /_app.py.bak
patch < /pypiserver.app.py.patch
systemctl restart pypiserver.service
systemctl status pypiserver.service
We should also update the external mirrors with the new rsync settings:
in /home/mirrordaemon/push_to_internal_mirrors.sh
make the following change (adding a p
)
rsync -rtlzv --delete --progress /datadrive/mirrordaemon/pypi/* mirrordaemon@${IP_ADDRESS}:/datadrive/mirrordaemon/pypi
to
rsync -prtlzv --delete --progress /datadrive/mirrordaemon/pypi/* mirrordaemon@${IP_ADDRESS}:/datadrive/mirrordaemon/pypi
@vollmersj : yes, we do use bandersnatch
to populate the initial (external) mirror with all packages from PyPI. We could consider blacklisting specific packages in future, but the problem here isn't about the size of any individual package, it's about the number of them. If we start restricting the available packages too much then we're not really fulfilling the stated aim of Tier-2 environments which is that you can install whatever packages you want to.
/simple
directory)[update /home/mirrordaemon/push_to_internal_mirrors.sh]
sudo -u mirrordaemon bash
source ~mirrordaemon/push_to_internal_mirrors.sh &
2019-04-04 1330
: started2019-04-05 1830
synchronised up to directory /4e/
out of /ff/
[approx 30% complete after 3hrs] (NB. this includes an unknown percentage that was already complete)2019-04-05 1000
synchronised up to directory /a9/
out of /ff/
[approx 66% complete after 20.5hrs] => 37hrs total2019-04-05 1420
synchronised up to directory /c3/
out of /ff/
[approx 76% complete after 25hrs] => 36.5hrs total2019-04-05 1530
synchronised up to directory /ca/
out of /ff/
[approx 79% complete after 26hrs] => 36hrs total2019-04-06 0130
[after 36hrs]See my comment in issue #285 for what I think is a successful test from DSG10 that the production internal PyPI mirror is up to date with its external mirror.
Comment moved to issue #285, which is dealing with mirror synchronisation reliability.
@jemrobinson How confident are we that the bandersnatch synchronisation from the public PyPI site to the external mirror is complete?
If it is more reliable than rsync, then could we also use it for our external->internal mirror sync?
Comment moved to issue #285, which is dealing with mirror synchronisation reliability.
I note that bandersnatch also supports a blacklist and whitelist though I guess it is designed to be run from the internal mirror in our arrangement, which we would not want.
Confirming that:
pip search matplotlib
gives expected outputpip install sismic
fails due to permission errorspip install --user sismic
workspip install --user uncertainties
workson the 10.250.74.160 machine in DSG10
@getcarter21 Please can your team test that pip install --user sismic
works on DSGs 11-15? Thanks.
@jemrobinson Can you open a PR to master with the fix for this, tagging this issue to autoclose when merged?
Mostly done in #281. Additional speed-up fixes by removing compression will go into a new PR.
On two occasions connecting from the DSG9 compute VM to the PyPI mirror in the test SHM has failed with a
ReadTimeoutError
when trying to install packages via pip.The first time, this was solved by deleting the internal mirror's package directory and restarting the rsync job to push the external mirror's packages over to the internal mirror.