search

web-platform-tests / results-collection

Other
41 stars 46 forks source link

Chrome builds interrupted by module compilation failure #649

Open jugglinmike opened 5 years ago

jugglinmike commented 5 years ago

Yesterday, 2 Chrome stable builds and 1 Chrome experimental builds failed as the WPT CLI installed Python dependencies. Specifically, the workers were unable to compile the native extensions provided by the cffi module (error log included below).

I haven't been able to find a conclusive explanation for this failure, but I suspect it was a transitory packaging issue. Here's why:

I'll keep my eyes peeled for future occurrences, but I'm hopeful that it was a passing glitch. I've manually re-triggered the failed builds; the corresponding data should be available on wpt.fyi shortly.

Partial error log 2019-02-13 13:39:57,247 INFO validate-wpt-results python ./wpt run --log-raw /tmp/tmpPTpDNe/log-raw.txt --log-wptreport /tmp/tmpPTpDNe/report.json --log-mach - --this-chunk 8 --total-chunks 20 --binary /usr/bin/google-chrome --webdriver-binary /home/jade/worker/GNU_Linux_Chunked_Runner/build/chromedriver --install-fonts --no-headless --binary-arg=--use-fake-ui-for-media-stream --binary-arg=--use-fake-device-for-media-stream --binary-arg=--enable-experimental-web-platform-features chrome 2019-02-13 13:40:00,123 INFO validate-wpt-results wpt-run:stderr DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. 2019-02-13 13:40:01,010 INFO validate-wpt-results wpt-run:stderr DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. 2019-02-13 13:40:03,706 INFO validate-wpt-results wpt-run:stderr Failed building wheel for cffi 2019-02-13 13:40:04,626 INFO validate-wpt-results wpt-run:stderr Command "/home/jade/worker/GNU_Linux_Chunked_Runner/build/_venv/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-FhQJPN/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-LhHMGn/install-record.txt --single-version-externally-managed --compile --install-headers /home/jade/worker/GNU_Linux_Chunked_Runner/build/_venv/include/site/python2.7/cffi" failed with error code 1 in /tmp/pip-install-FhQJPN/cffi/ 2019-02-13 13:40:04,676 INFO validate-wpt-results wpt-run:stderr CRITICAL:tools.wpt.utils:('/home/jade/worker/GNU_Linux_Chunked_Runner/build/_venv/bin/pip', 'install', '-r', u'/home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt') exited with return code 1 2019-02-13 13:40:04,677 INFO validate-wpt-results wpt-run:stderr CRITICAL:tools.wpt.utils:Collecting html5lib==1.0.1 (from -r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 1)) 2019-02-13 13:40:04,677 INFO validate-wpt-results wpt-run:stderr Using cached https://files.pythonhosted.org/packages/a5/62/bbd2be0e7943ec8504b517e62bab011b4946e1258842bc159e5dfde15b96/html5lib-1.0.1-py2.py3-none-any.whl 2019-02-13 13:40:04,677 INFO validate-wpt-results wpt-run:stderr Collecting mozinfo==0.10 (from -r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 2)) 2019-02-13 13:40:04,677 INFO validate-wpt-results wpt-run:stderr Collecting mozlog==4.0 (from -r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 3)) 2019-02-13 13:40:04,677 INFO validate-wpt-results wpt-run:stderr Using cached https://files.pythonhosted.org/packages/18/7b/37831f9ae29ea950dd2dd332ae49a684ffda3aaf30a72c7e6b275b080315/mozlog-4.0-py2.py3-none-any.whl 2019-02-13 13:40:04,677 INFO validate-wpt-results wpt-run:stderr Collecting mozdebug==0.1.1 (from -r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 4)) 2019-02-13 13:40:04,677 INFO validate-wpt-results wpt-run:stderr Using cached https://files.pythonhosted.org/packages/cb/fb/cd673c7e9f7d4911d93c7aacd4d464ec8465d213c1ca63fcd4f59cc24ef7/mozdebug-0.1.1-py2.py3-none-any.whl 2019-02-13 13:40:04,677 INFO validate-wpt-results wpt-run:stderr Requirement already satisfied: urllib3[secure]==1.24.1 in ./_venv/lib/python2.7/site-packages (from -r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 5)) (1.24.1) 2019-02-13 13:40:04,678 INFO validate-wpt-results wpt-run:stderr Collecting six>=1.9 (from html5lib==1.0.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 1)) 2019-02-13 13:40:04,678 INFO validate-wpt-results wpt-run:stderr Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl 2019-02-13 13:40:04,678 INFO validate-wpt-results wpt-run:stderr Collecting webencodings (from html5lib==1.0.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 1)) 2019-02-13 13:40:04,678 INFO validate-wpt-results wpt-run:stderr Using cached https://files.pythonhosted.org/packages/f4/24/2a3e3df732393fed8b3ebf2ec078f05546de641fe1b667ee316ec1dcf3b7/webencodings-0.5.1-py2.py3-none-any.whl 2019-02-13 13:40:04,678 INFO validate-wpt-results wpt-run:stderr Collecting mozfile>=0.12 (from mozinfo==0.10->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 2)) 2019-02-13 13:40:04,678 INFO validate-wpt-results wpt-run:stderr Using cached https://files.pythonhosted.org/packages/85/ed/4f8bec5ac04373ef0c79428bdd50ce039e91e4b53660d08e9a45988b74dc/mozfile-2.0.0-py2.py3-none-any.whl 2019-02-13 13:40:04,678 INFO validate-wpt-results wpt-run:stderr Collecting mozterm (from mozlog==4.0->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 3)) 2019-02-13 13:40:04,678 INFO validate-wpt-results wpt-run:stderr Using cached https://files.pythonhosted.org/packages/ab/eb/0c53684f5821d666372c6ea03d8c64558c8f74fda0fd5b24ae3dd2ab6a95/mozterm-1.0.0-py2.py3-none-any.whl 2019-02-13 13:40:04,679 INFO validate-wpt-results wpt-run:stderr Collecting blessings>=1.3 (from mozlog==4.0->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 3)) 2019-02-13 13:40:04,679 INFO validate-wpt-results wpt-run:stderr Using cached https://files.pythonhosted.org/packages/8d/b1/a3fe6fd8a012e6d019bafd671c2fee0597ea97ff2e76c25aadfa4545fc32/blessings-1.7-py2-none-any.whl 2019-02-13 13:40:04,679 INFO validate-wpt-results wpt-run:stderr Collecting pyOpenSSL>=0.14; extra == "secure" (from urllib3[secure]==1.24.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 5)) 2019-02-13 13:40:04,679 INFO validate-wpt-results wpt-run:stderr Using cached https://files.pythonhosted.org/packages/01/c8/ceb170d81bd3941cbeb9940fc6cc2ef2ca4288d0ca8929ea4db5905d904d/pyOpenSSL-19.0.0-py2.py3-none-any.whl 2019-02-13 13:40:04,679 INFO validate-wpt-results wpt-run:stderr Requirement already satisfied: idna>=2.0.0; extra == "secure" in ./_venv/lib/python2.7/site-packages (from urllib3[secure]==1.24.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 5)) (2.8) 2019-02-13 13:40:04,679 INFO validate-wpt-results wpt-run:stderr Collecting cryptography>=1.3.4; extra == "secure" (from urllib3[secure]==1.24.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 5)) 2019-02-13 13:40:04,679 INFO validate-wpt-results wpt-run:stderr Using cached https://files.pythonhosted.org/packages/17/fd/4c2c8953a9dfe38fbe0c3adafb6355540bd98cef70cc82734acb0a4c0e2f/cryptography-2.5-cp27-cp27mu-manylinux1_x86_64.whl 2019-02-13 13:40:04,679 INFO validate-wpt-results wpt-run:stderr Requirement already satisfied: certifi; extra == "secure" in ./_venv/lib/python2.7/site-packages (from urllib3[secure]==1.24.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 5)) (2018.11.29) 2019-02-13 13:40:04,680 INFO validate-wpt-results wpt-run:stderr Collecting ipaddress; extra == "secure" (from urllib3[secure]==1.24.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 5)) 2019-02-13 13:40:04,680 INFO validate-wpt-results wpt-run:stderr Using cached https://files.pythonhosted.org/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl 2019-02-13 13:40:04,680 INFO validate-wpt-results wpt-run:stderr Collecting asn1crypto>=0.21.0 (from cryptography>=1.3.4; extra == "secure"->urllib3[secure]==1.24.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 5)) 2019-02-13 13:40:04,680 INFO validate-wpt-results wpt-run:stderr Using cached https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl 2019-02-13 13:40:04,680 INFO validate-wpt-results wpt-run:stderr Collecting enum34; python_version < "3" (from cryptography>=1.3.4; extra == "secure"->urllib3[secure]==1.24.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 5)) 2019-02-13 13:40:04,683 INFO validate-wpt-results wpt-run:stderr Using cached https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl 2019-02-13 13:40:04,684 INFO validate-wpt-results wpt-run:stderr Collecting cffi!=1.11.3,>=1.8 (from cryptography>=1.3.4; extra == "secure"->urllib3[secure]==1.24.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 5)) 2019-02-13 13:40:04,684 INFO validate-wpt-results wpt-run:stderr Downloading https://files.pythonhosted.org/packages/10/fe/b6362c613a70ac29cf7cac36307d85f08ebe4a96d9d54b895b10a807e39b/cffi-1.12.0.tar.gz (453kB) 2019-02-13 13:40:04,684 INFO validate-wpt-results wpt-run:stderr Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography>=1.3.4; extra == "secure"->urllib3[secure]==1.24.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 5)) 2019-02-13 13:40:04,684 INFO validate-wpt-results wpt-run:stderr Building wheels for collected packages: cffi 2019-02-13 13:40:04,684 INFO validate-wpt-results wpt-run:stderr Building wheel for cffi (setup.py): started 2019-02-13 13:40:04,685 INFO validate-wpt-results wpt-run:stderr Building wheel for cffi (setup.py): finished with status 'error' 2019-02-13 13:40:04,685 INFO validate-wpt-results wpt-run:stderr Complete output from command /home/jade/worker/GNU_Linux_Chunked_Runner/build/_venv/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-FhQJPN/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-9U0S4d --python-tag cp27: 2019-02-13 13:40:04,685 INFO validate-wpt-results wpt-run:stderr running bdist_wheel 2019-02-13 13:40:04,685 INFO validate-wpt-results wpt-run:stderr running build 2019-02-13 13:40:04,685 INFO validate-wpt-results wpt-run:stderr running build_py 2019-02-13 13:40:04,685 INFO validate-wpt-results wpt-run:stderr creating build 2019-02-13 13:40:04,685 INFO validate-wpt-results wpt-run:stderr creating build/lib.linux-x86_64-2.7 2019-02-13 13:40:04,686 INFO validate-wpt-results wpt-run:stderr creating build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,686 INFO validate-wpt-results wpt-run:stderr copying cffi/ffiplatform.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,686 INFO validate-wpt-results wpt-run:stderr copying cffi/error.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,686 INFO validate-wpt-results wpt-run:stderr copying cffi/lock.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,686 INFO validate-wpt-results wpt-run:stderr copying cffi/__init__.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,686 INFO validate-wpt-results wpt-run:stderr copying cffi/commontypes.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,686 INFO validate-wpt-results wpt-run:stderr copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,686 INFO validate-wpt-results wpt-run:stderr copying cffi/vengine_gen.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,687 INFO validate-wpt-results wpt-run:stderr copying cffi/verifier.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,687 INFO validate-wpt-results wpt-run:stderr copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,687 INFO validate-wpt-results wpt-run:stderr copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,687 INFO validate-wpt-results wpt-run:stderr copying cffi/recompiler.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,687 INFO validate-wpt-results wpt-run:stderr copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,687 INFO validate-wpt-results wpt-run:stderr copying cffi/api.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,687 INFO validate-wpt-results wpt-run:stderr copying cffi/pkgconfig.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,688 INFO validate-wpt-results wpt-run:stderr copying cffi/model.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,688 INFO validate-wpt-results wpt-run:stderr copying cffi/cparser.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,688 INFO validate-wpt-results wpt-run:stderr copying cffi/_cffi_include.h -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,688 INFO validate-wpt-results wpt-run:stderr copying cffi/parse_c_type.h -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,688 INFO validate-wpt-results wpt-run:stderr copying cffi/_embedding.h -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,688 INFO validate-wpt-results wpt-run:stderr copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,688 INFO validate-wpt-results wpt-run:stderr running build_ext 2019-02-13 13:40:04,688 INFO validate-wpt-results wpt-run:stderr building '_cffi_backend' extension 2019-02-13 13:40:04,689 INFO validate-wpt-results wpt-run:stderr creating build/temp.linux-x86_64-2.7 2019-02-13 13:40:04,689 INFO validate-wpt-results wpt-run:stderr creating build/temp.linux-x86_64-2.7/c 2019-02-13 13:40:04,689 INFO validate-wpt-results wpt-run:stderr x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-3hk45v/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-x86_64-2.7/c/_cffi_backend.o 2019-02-13 13:40:04,689 INFO validate-wpt-results wpt-run:stderr c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory 2019-02-13 13:40:04,689 INFO validate-wpt-results wpt-run:stderr #include 2019-02-13 13:40:04,689 INFO validate-wpt-results wpt-run:stderr ^~~~~~~ 2019-02-13 13:40:04,689 INFO validate-wpt-results wpt-run:stderr compilation terminated. 2019-02-13 13:40:04,690 INFO validate-wpt-results wpt-run:stderr error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 2019-02-13 13:40:04,690 INFO validate-wpt-results wpt-run:stderr 2019-02-13 13:40:04,690 INFO validate-wpt-results wpt-run:stderr ---------------------------------------- 2019-02-13 13:40:04,690 INFO validate-wpt-results wpt-run:stderr Running setup.py clean for cffi 2019-02-13 13:40:04,690 INFO validate-wpt-results wpt-run:stderr Failed to build cffi 2019-02-13 13:40:04,690 INFO validate-wpt-results wpt-run:stderr Installing collected packages: six, webencodings, html5lib, mozfile, mozinfo, mozterm, blessings, mozlog, mozdebug, asn1crypto, enum34, ipaddress, pycparser, cffi, cryptography, pyOpenSSL 2019-02-13 13:40:04,690 INFO validate-wpt-results wpt-run:stderr Running setup.py install for cffi: started 2019-02-13 13:40:04,690 INFO validate-wpt-results wpt-run:stderr Running setup.py install for cffi: finished with status 'error' 2019-02-13 13:40:04,691 INFO validate-wpt-results wpt-run:stderr Complete output from command /home/jade/worker/GNU_Linux_Chunked_Runner/build/_venv/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-FhQJPN/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-LhHMGn/install-record.txt --single-version-externally-managed --compile --install-headers /home/jade/worker/GNU_Linux_Chunked_Runner/build/_venv/include/site/python2.7/cffi: 2019-02-13 13:40:04,691 INFO validate-wpt-results wpt-run:stderr running install 2019-02-13 13:40:04,691 INFO validate-wpt-results wpt-run:stderr running build 2019-02-13 13:40:04,691 INFO validate-wpt-results wpt-run:stderr running build_py 2019-02-13 13:40:04,691 INFO validate-wpt-results wpt-run:stderr creating build 2019-02-13 13:40:04,691 INFO validate-wpt-results wpt-run:stderr creating build/lib.linux-x86_64-2.7 2019-02-13 13:40:04,692 INFO validate-wpt-results wpt-run:stderr creating build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,692 INFO validate-wpt-results wpt-run:stderr copying cffi/ffiplatform.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,692 INFO validate-wpt-results wpt-run:stderr copying cffi/error.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,692 INFO validate-wpt-results wpt-run:stderr copying cffi/lock.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,692 INFO validate-wpt-results wpt-run:stderr copying cffi/__init__.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,692 INFO validate-wpt-results wpt-run:stderr copying cffi/commontypes.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,692 INFO validate-wpt-results wpt-run:stderr copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,692 INFO validate-wpt-results wpt-run:stderr copying cffi/vengine_gen.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,693 INFO validate-wpt-results wpt-run:stderr copying cffi/verifier.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,693 INFO validate-wpt-results wpt-run:stderr copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,693 INFO validate-wpt-results wpt-run:stderr copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,693 INFO validate-wpt-results wpt-run:stderr copying cffi/recompiler.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,693 INFO validate-wpt-results wpt-run:stderr copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,693 INFO validate-wpt-results wpt-run:stderr copying cffi/api.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,693 INFO validate-wpt-results wpt-run:stderr copying cffi/pkgconfig.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,694 INFO validate-wpt-results wpt-run:stderr copying cffi/model.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,694 INFO validate-wpt-results wpt-run:stderr copying cffi/cparser.py -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,694 INFO validate-wpt-results wpt-run:stderr copying cffi/_cffi_include.h -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,694 INFO validate-wpt-results wpt-run:stderr copying cffi/parse_c_type.h -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,694 INFO validate-wpt-results wpt-run:stderr copying cffi/_embedding.h -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,694 INFO validate-wpt-results wpt-run:stderr copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-2.7/cffi 2019-02-13 13:40:04,694 INFO validate-wpt-results wpt-run:stderr running build_ext 2019-02-13 13:40:04,694 INFO validate-wpt-results wpt-run:stderr building '_cffi_backend' extension 2019-02-13 13:40:04,694 INFO validate-wpt-results wpt-run:stderr creating build/temp.linux-x86_64-2.7 2019-02-13 13:40:04,695 INFO validate-wpt-results wpt-run:stderr creating build/temp.linux-x86_64-2.7/c 2019-02-13 13:40:04,695 INFO validate-wpt-results wpt-run:stderr x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-3hk45v/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-x86_64-2.7/c/_cffi_backend.o 2019-02-13 13:40:04,695 INFO validate-wpt-results wpt-run:stderr c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory 2019-02-13 13:40:04,695 INFO validate-wpt-results wpt-run:stderr #include 2019-02-13 13:40:04,695 INFO validate-wpt-results wpt-run:stderr ^~~~~~~ 2019-02-13 13:40:04,695 INFO validate-wpt-results wpt-run:stderr compilation terminated. 2019-02-13 13:40:04,696 INFO validate-wpt-results wpt-run:stderr error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 2019-02-13 13:40:04,696 INFO validate-wpt-results wpt-run:stderr 2019-02-13 13:40:04,696 INFO validate-wpt-results wpt-run:stderr ---------------------------------------- 2019-02-13 13:40:04,696 INFO validate-wpt-results wpt-run:stderr 2019-02-13 13:40:04,696 INFO validate-wpt-results wpt-run:stderr Traceback (most recent call last): 2019-02-13 13:40:04,696 INFO validate-wpt-results wpt-run:stderr File "./wpt", line 5, in 2019-02-13 13:40:04,696 INFO validate-wpt-results wpt-run:stderr wpt.main() 2019-02-13 13:40:04,696 INFO validate-wpt-results wpt-run:stderr File "/home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/wpt.py", line 115, in main 2019-02-13 13:40:04,697 INFO validate-wpt-results wpt-run:stderr venv = setup_virtualenv(main_args.venv, main_args.skip_venv_setup, props) 2019-02-13 13:40:04,697 INFO validate-wpt-results wpt-run:stderr File "/home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/wpt.py", line 95, in setup_virtualenv 2019-02-13 13:40:04,697 INFO validate-wpt-results wpt-run:stderr venv.install_requirements(path) 2019-02-13 13:40:04,697 INFO validate-wpt-results wpt-run:stderr File "/home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/virtualenv.py", line 55, in install_requirements 2019-02-13 13:40:04,697 INFO validate-wpt-results wpt-run:stderr call(self.pip_path, "install", "-r", requirements_path) 2019-02-13 13:40:04,697 INFO validate-wpt-results wpt-run:stderr File "/home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/utils.py", line 43, in call 2019-02-13 13:40:04,697 INFO validate-wpt-results wpt-run:stderr return subprocess.check_output(args) 2019-02-13 13:40:04,697 INFO validate-wpt-results wpt-run:stderr File "/usr/lib/python2.7/subprocess.py", line 223, in check_output 2019-02-13 13:40:04,698 INFO validate-wpt-results wpt-run:stderr raise CalledProcessError(retcode, cmd, output=output) 2019-02-13 13:40:04,698 INFO validate-wpt-results wpt-run:stderr subprocess.CalledProcessError: Command '('/home/jade/worker/GNU_Linux_Chunked_Runner/build/_venv/bin/pip', 'install', '-r', u'/home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt')' returned non-zero exit status 1 2019-02-13 13:40:04,698 INFO validate-wpt-results WPT CLI exited with return code None
gsnedders commented 5 years ago

My guess is we don't have the libffi-dev package installed hence we couldn't build cffi from source, and the appropriate cffi wheel hadn't been uploaded yet?

jugglinmike commented 5 years ago

That sounds plausible and more satisfying than my feeble "transitory packaging issue."

jugglinmike commented 5 years ago

Surprisingly, two more builds just failed with the very same error (8 days later). This happened about 3 hours ago, and that directly correlates with the release of cffi 1.12.2.

@gsnedders @jgraham I know next to nothing about Python packaging, but this seems like something we ought to be able to avoid. Is there anything we can change in WPT to guard against invalid installations?

jgraham commented 5 years ago

Hah.

cffi seems to be a dep of pyOpenSSL, which depends on cffi [required: >=1.8,!=1.11.3, installed: 1.11.5]. I don't think that pinning transitive dependencies is supported in any good way, but what could work is this:

Adjust the requirements.txt file to specify a specific version of cffi. Assume that pip will notice that the pinned version is a match for the requirement and not try to install a newer version. Then rely on pyup.io to install newer versions which will come with enough delay to assure we don't end up with races for the wheel to be uploaded.

jugglinmike commented 5 years ago

Thanks, James! That looks like a fairly module-specific solution. Is that an indication of some bad practice in pyOpenSSL or cffi? Or is this a problem that Python developers experience commonly?

jgraham commented 5 years ago

Ooh, pip seems to have a --prefer-binary option. I think that might help here.

jugglinmike commented 5 years ago

Six more builds (3 Firefox, 2 Chrome, and 1 Edge) were interrupted today at 14:05 UTC. There's no new release of cffi to blame this time, though. @jgraham do you know how this could happen again? Could it occur when some transitive dependency transitions to the new release?

jgraham commented 5 years ago

Do you have logs?

jugglinmike commented 5 years ago

Sure do!

All for WPT commit 9f49616e7fa90489fac85dd203e86d7c83ca67da

jgraham commented 5 years ago

I'm not sure why it's trying to build it there; maybe something happened to the wheel package. In any case passing --prefer-binary or --only-binary=:all: to the pip command seems like it should avoid this failure mode.

gsnedders commented 5 years ago

At least the ones I looked at there aren't cffi, they're cryptography, which had two releases yesterday.

To expand on my previous comment: note that uploads aren't atomic on PyPI.

The common flow is:

  1. Create a new release.
  2. Upload an artifact.
  3. If you have another artifact, goto 2; else, complete.

Each of these items happens sequentially, and if a source release artifact is uploaded before a (suitable) binary release artifact is anything looking for the latest release will download the source release and build it. As noted, either of the options @jgraham suggested will passing this will change this (preferring an older release with a binary release).

jugglinmike commented 5 years ago

We got hit by both releases of the "cryptography" package (the second release interfered with six builds for Firefox stable). I'm going to look in to patching WPT as you folks recommend. I'm curious, though: have either of you seen this occur in Taskcluster?