bandersnatch同步pypi失败

[shitou1992]

我这边使用bandersnatch同步pypi失败的时候会在packages中产生隐藏文件，不知道你们是怎么解决的

[shitou1992]

我现在的方法是find然后删除，不过不是长久之计

[shankerwangmiao]

据反映，bandersnatch 并不能成功从 tuna 同步 pypi 镜像。bandersnatch 只能用于从官方上游同步数据。

[shitou1992]

是的，我是从官方同步的，同步过程中发现了这个问题，所以想咨询下你们是否遇到过这个问题，是如何处理的

[shankerwangmiao]

具体是怎么失败的，我们遇到了失败的情况，就直接重新强行多同步几次就好了。

[shitou1992]

基本是timeout导致的，具体体现是这样的：

[root@theThree pypi]# find web/packages/21/ -name ".*"| xargs ls -lh

-rw------- 1 root root 128K Feb 16 10:42 web/packages/21/1a/3d8d5303cc0d44d7ffc49ba2b4abb9223689a3557d0e9c812ef973a9629f/.cesium-0.9.9-cp27-cp27mu-manylinux1_x86_64.whl._zqvsbfh -rw-------. 1 root root 8.2M Jul 23 2018 web/packages/21/1e/b168e2fdba93962d9c39fc29808b82318245bdc13c3b2b9e8288078a671f/.mxnet_mkl-1.3.0b20180719-cp27-cp27m-macosx_10_12_x86_64.whl.7rapv8dk -rw-------. 1 root root 12M Jul 23 2018 web/packages/21/1e/b168e2fdba93962d9c39fc29808b82318245bdc13c3b2b9e8288078a671f/.mxnet_mkl-1.3.0b20180719-cp27-cp27m-macosx_10_12_x86_64.whl.nf101clo -rw-------. 1 root root 9.3M Jul 23 2018 web/packages/21/1e/b168e2fdba93962d9c39fc29808b82318245bdc13c3b2b9e8288078a671f/.mxnet_mkl-1.3.0b20180719-cp27-cp27m-macosx_10_12_x86_64.whl.posbj6os -rw-------. 1 root root 2.0M Jul 23 2018 web/packages/21/1e/b168e2fdba93962d9c39fc29808b82318245bdc13c3b2b9e8288078a671f/.mxnet_mkl-1.3.0b20180719-cp27-cp27m-macosx_10_12_x86_64.whl.w1k2547p

[huiyiqun]

贴一下错误信息，挂在 mxnet 的话，我们之前的经验是，不是超时，而是 CDN 不一致导致的，是否是还得看错误信息才能确定。

[shitou1992]

错误如下： ''' 2019-03-05 15:16:16,435 ERROR: Continuing to next file after error downloading: https://files.pythonhosted.org/packages/95/e4/58f55649b0fa535097adc76d0accaf647b05bb2f468459b51e53bec529e8/pydrobert_kaldi-0.5.2-cp36-cp36m-manylinux1_i686.whl Traceback (most recent call last): File "/root/anaconda3/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 280, in recv_into return self.connection.recv_into(*args, **kwargs) File "/root/anaconda3/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1715, in recv_into self._raise_ssl_error(self._ssl, result) File "/root/anaconda3/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1521, in _raise_ssl_error raise WantReadError() OpenSSL.SSL.WantReadError

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/root/anaconda3/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 280, in recv_into return self.connection.recv_into(*args, **kwargs) File "/root/anaconda3/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1715, in recv_into self._raise_ssl_error(self._ssl, result) File "/root/anaconda3/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1521, in _raise_ssl_error raise WantReadError() OpenSSL.SSL.WantReadError

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/root/anaconda3/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 280, in recv_into return self.connection.recv_into(*args, **kwargs) File "/root/anaconda3/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1715, in recv_into self._raise_ssl_error(self._ssl, result) File "/root/anaconda3/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1521, in _raise_ssl_error raise WantReadError() OpenSSL.SSL.WantReadError

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/root/anaconda3/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 280, in recv_into return self.connection.recv_into(*args, **kwargs) File "/root/anaconda3/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1715, in recv_into self._raise_ssl_error(self._ssl, result) File "/root/anaconda3/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1521, in _raise_ssl_error raise WantReadError() OpenSSL.SSL.WantReadError

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/root/anaconda3/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 280, in recv_into return self.connection.recv_into(*args, **kwargs) File "/root/anaconda3/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1715, in recv_into self._raise_ssl_error(self._ssl, result) File "/root/anaconda3/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1521, in _raise_ssl_error raise WantReadError() OpenSSL.SSL.WantReadError

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/root/anaconda3/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 280, in recv_into return self.connection.recv_into(*args, **kwargs) File "/root/anaconda3/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1715, in recv_into self._raise_ssl_error(self._ssl, result) File "/root/anaconda3/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1521, in _raise_ssl_error raise WantReadError() OpenSSL.SSL.WantReadError

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/root/anaconda3/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 280, in recv_into return self.connection.recv_into(*args, **kwargs) File "/root/anaconda3/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1715, in recv_into self._raise_ssl_error(self._ssl, result) File "/root/anaconda3/lib/python3.6/site-packages/OpenSSL/SSL.py", line 1521, in _raise_ssl_error raise WantReadError() OpenSSL.SSL.WantReadError

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/root/anaconda3/lib/python3.6/site-packages/urllib3/response.py", line 302, in _error_catcher yield File "/root/anaconda3/lib/python3.6/site-packages/urllib3/response.py", line 384, in read data = self._fp.read(amt) File "/root/anaconda3/lib/python3.6/http/client.py", line 449, in read n = self.readinto(b) File "/root/anaconda3/lib/python3.6/http/client.py", line 493, in readinto n = self.fp.readinto(b) File "/root/anaconda3/lib/python3.6/socket.py", line 586, in readinto return self._sock.recv_into(b) File "/root/anaconda3/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 296, in recv_into return self.recv_into(*args, kwargs) File "/root/anaconda3/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 296, in recv_into return self.recv_into(*args, *kwargs) File "/root/anaconda3/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 296, in recv_into return self.recv_into(args, kwargs) [Previous line repeated 2 more times] File "/root/anaconda3/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 294, in recv_into raise timeout('The read operation timed out') socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/root/anaconda3/lib/python3.6/site-packages/requests/models.py", line 745, in generate for chunk in self.raw.stream(chunk_size, decode_content=True): File "/root/anaconda3/lib/python3.6/site-packages/urllib3/response.py", line 436, in stream data = self.read(amt=amt, decode_content=decode_content) File "/root/anaconda3/lib/python3.6/site-packages/urllib3/response.py", line 401, in read raise IncompleteRead(self._fp_bytes_read, self.length_remaining) File "/root/anaconda3/lib/python3.6/contextlib.py", line 99, in exit self.gen.throw(type, value, traceback) File "/root/anaconda3/lib/python3.6/site-packages/urllib3/response.py", line 307, in _error_catcher raise ReadTimeoutError(self._pool, None, 'Read timed out.') urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/root/anaconda3/lib/python3.6/site-packages/bandersnatch/package.py", line 175, in sync_release_files release_file["url"], release_file["digests"]["sha256"] File "/root/anaconda3/lib/python3.6/site-packages/bandersnatch/package.py", line 356, in download_file for chunk in r.iter_content(chunk_size=64 * 1024): File "/root/anaconda3/lib/python3.6/site-packages/requests/models.py", line 752, in generate raise ConnectionError(e) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out. 2019-03-05 15:16:16,437 INFO: Downloading: https://files.pythonhosted.org/packages/72/f3/d8a466615411ccbb88cbdb0a2cf385b8719bc12c0b3694304e82ba39e08e/pydrobert_kaldi-0.5.2-cp36-cp36m-manylinux1_x86_64.whl '''

[shankerwangmiao]

嗯，我们也遇到过类似问题，处理方法就是多同步几次。

[shitou1992]

好吧，这种情况在对应的目录里会生成隐藏文件，建议你们也检查下