search

bazelbuild / rules_python

Bazel Python Rules
https://rules-python.readthedocs.io
Apache License 2.0
523 stars 537 forks source link

Tests failing on Bazel CI #67

Closed katre closed 6 years ago

katre commented 6 years ago

I am seeing failures in Bazel CI for both Bazel HEAD and Bazel 0.10.1: https://ci.bazel.build/blue/organizations/jenkins/Global%2Frules_python/detail/rules_python/276/tests

I can reproduce this locally and get the same error messages.

katre commented 6 years ago

This seems to be due to 3175797bd07aac4ff35fa711f0a82285f2005e42, @mattmoor can you comment?

mattmoor commented 6 years ago

@duggelz made PAR construction deterministic, so without probing deeper I would guess that you changed something (transitively) about how the PAR is built (the same __init__.py change that broke rules_docker seems plausible?).

buchgr commented 6 years ago

How can we resolve this issue? Who would be the person to talk to?

duggelz commented 6 years ago

Bazel has apparently just switched to a new CI system:

https://groups.google.com/d/msg/bazel-discuss/8spcVZMv9yE/nz-ya0tpAQAJ

buchgr commented 6 years ago

Heya,

we are still seeing this error on our CI. //tools:par_test is failing with Bazel from HEAD but it works with the latest released Bazel.

exec ${PAGER:-/usr/bin/less} "$0" || exit 1
-----------------------------------------------------------------------------
FF
======================================================================
FAIL: test_piptool_matches (__main__.WheelTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/3dd5dfaab0d9fd8c7dd47297b1d500e7/sandbox/5266514015113038112/execroot/io_bazel_rules_python/bazel-out/k8-fastbuild/bin/tools/par_test.runfiles/io_bazel_rules_python/tools/par_test.py", line 33, in test_piptool_matches
    'The checked in tools/piptool.par does not match the latest build.')
AssertionError: The checked in tools/piptool.par does not match the latest build.

======================================================================
FAIL: test_whltool_matches (__main__.WheelTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/3dd5dfaab0d9fd8c7dd47297b1d500e7/sandbox/5266514015113038112/execroot/io_bazel_rules_python/bazel-out/k8-fastbuild/bin/tools/par_test.runfiles/io_bazel_rules_python/tools/par_test.py", line 42, in test_whltool_matches
    'The checked in tools/whltool.par does not match the latest build.')
AssertionError: The checked in tools/whltool.par does not match the latest build.

----------------------------------------------------------------------
Ran 2 tests in 0.050s

FAILED (failures=2)
duggelz commented 6 years ago

I am pretty curious what the actual difference is. I don't have access to the new CI, so I guess I'll see if I can repo it locally.

buchgr commented 6 years ago

@duggelz I have sent you an invite to your @google.com address.

lfpino commented 6 years ago

Any updates on this issue @duggelz ? Thanks!

lfpino commented 6 years ago

Friendly ping @duggelz, we're still seeing the failures in Bazel CI

duggelz commented 6 years ago

Ok, so here's my diagnosis: .par file construction is not deterministic between different Python interpreter versions. In particular, python 2.7.13+ and python 2.7.12- produce zip files with slightly different header flags, because 2.7.12 was out of spec (but the zip files still work - it's a loose spec). That seems to be what's happening here.

It's not really fixable by subpar. We could relax the test to check that the contents of the individual zip members are the same. But a malicious adversary could easily handcraft a zipfile header that passes that relaxed check. And I'm actually not comfortable with this as a "security" boundary anyway, because I certainly didn't write the code as such.

So, unless we have some kind of process where .par files are regenerated on a "blessed" image with a known Python interpreter version, and/or by a "blessed" release person, we should probably disable this test.

And we really shouldn't let untrusted contributors submit updated .par files.

duggelz commented 6 years ago

As per this thread: https://bugs.python.org/issue29094 , it appears that Python 2.7.13, 3.5.3, 3.6.0, and 3.6.1 (only) produce these subtly broken .par files. Unfortunately, those are the versions that are installed with the widely used Debian 9 (not to mention my personal workstation!). By chance, it appears that none of the BuildKite builder machines have those "bad" versions, so we should be able to rebuild the .par files, and turn the build green.

duggelz commented 6 years ago

Regenerating the files on Ubuntu 16.04 turned the build green: https://github.com/bazelbuild/rules_python/pull/83

Going forward, it seems like we can keep this test, at the cost of a bit of annoyance for people with "broken" Python versions (basically just Debian 9). Google's internal workstations will be updating from Python 2.7.13 to 2.7.14 "soon", and thus get the fix, in the future (We also have Python 3.6! Which is now entirely supported and recommended inside Google! Use Python 3 everywhere!)

vladmos commented 6 years ago

Can #83 be merged? We'd like to cut a release for Bazel 0.13 and failing rules_python builds block that.

duggelz commented 6 years ago

Merged.

philwo commented 6 years ago

@duggelz Not sure what's going on, but this still seems to fail (tested with Bazel 0.12.0rc3 just a few minutes ago):

exec ${PAGER:-/usr/bin/less} "$0" || exit 1
-----------------------------------------------------------------------------
FF
======================================================================
FAIL: test_piptool_matches (__main__.WheelTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/b5c94007a378b85b2836f44d796f101b/bazel-sandbox/3529782167307750595/execroot/io_bazel_rules_python/bazel-out/k8-fastbuild/bin/tools/par_test.runfiles/io_bazel_rules_python/tools/par_test.py", line 86, in test_piptool_matches
    TestData('tools/piptool.par'))
  File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/b5c94007a378b85b2836f44d796f101b/bazel-sandbox/3529782167307750595/execroot/io_bazel_rules_python/bazel-out/k8-fastbuild/bin/tools/par_test.runfiles/io_bazel_rules_python/tools/par_test.py", line 82, in _diff_zip
    self.assertEquals(contents1, contents2, message)
AssertionError: Files do not match.

************************************************************************
File 1: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/b5c94007a378b85b2836f44d796f101b/bazel-sandbox/3529782167307750595/execroot/io_bazel_rules_python/bazel-out/k8-fastbuild/bin/tools/par_test.runfiles/io_bazel_rules_python/rules_python/piptool.par
Length in bytes: 5879466
SHA256: 51519a9c2c8b0917c82158bdd07bd36f9db5b0e82519cb34b1ce66cc09e0a1d1
************************************************************************
File 2: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/b5c94007a378b85b2836f44d796f101b/bazel-sandbox/3529782167307750595/execroot/io_bazel_rules_python/bazel-out/k8-fastbuild/bin/tools/par_test.runfiles/io_bazel_rules_python/tools/piptool.par
Length in Bytes: 5878341
SHA256: 5deae79fc45c6b9f3cbb9943e48b24ef63ebad563b002c453bf97a2f56efaba5
************************************************************************
Zip Content Diff:
--- 

+++ 

@@ -2,7 +2,7 @@

 __init__.py                                    1980-01-01 00:00:00            0
 __main__.py                                    1980-01-01 00:00:00         7673
 io_bazel_rules_python/rules_python/__init__.py 1980-01-01 00:00:00            0
-io_bazel_rules_python/rules_python/piptool     1980-01-01 00:00:00         6847
+io_bazel_rules_python/rules_python/piptool     1980-01-01 00:00:00         5722
 io_bazel_rules_python/rules_python/piptool.py  1980-01-01 00:00:00         7342
 io_bazel_rules_python/rules_python/whl.py      1980-01-01 00:00:00         5606
 pypi__pip_9_0_3/__init__.py                    1980-01-01 00:00:00            0
************************************************************************

======================================================================
FAIL: test_whltool_matches (__main__.WheelTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/b5c94007a378b85b2836f44d796f101b/bazel-sandbox/3529782167307750595/execroot/io_bazel_rules_python/bazel-out/k8-fastbuild/bin/tools/par_test.runfiles/io_bazel_rules_python/tools/par_test.py", line 90, in test_whltool_matches
    TestData('tools/whltool.par'))
  File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/b5c94007a378b85b2836f44d796f101b/bazel-sandbox/3529782167307750595/execroot/io_bazel_rules_python/bazel-out/k8-fastbuild/bin/tools/par_test.runfiles/io_bazel_rules_python/tools/par_test.py", line 82, in _diff_zip
    self.assertEquals(contents1, contents2, message)
AssertionError: Files do not match.

************************************************************************
File 1: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/b5c94007a378b85b2836f44d796f101b/bazel-sandbox/3529782167307750595/execroot/io_bazel_rules_python/bazel-out/k8-fastbuild/bin/tools/par_test.runfiles/io_bazel_rules_python/rules_python/whltool.par
Length in bytes: 1386426
SHA256: acbf53c155c70fe36d3c1a0e0fc70c18776a99566b3a9ff3063e592445a58c5c
************************************************************************
File 2: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/b5c94007a378b85b2836f44d796f101b/bazel-sandbox/3529782167307750595/execroot/io_bazel_rules_python/bazel-out/k8-fastbuild/bin/tools/par_test.runfiles/io_bazel_rules_python/tools/whltool.par
Length in Bytes: 1385301
SHA256: f56e8128d2bd575859e483eaa9a356ec05442765a076d757fc9067cb654de833
************************************************************************
Zip Content Diff:
--- 

+++ 

@@ -3,7 +3,7 @@

 __main__.py                                    1980-01-01 00:00:00         5847
 io_bazel_rules_python/rules_python/__init__.py 1980-01-01 00:00:00            0
 io_bazel_rules_python/rules_python/whl.py      1980-01-01 00:00:00         5606
-io_bazel_rules_python/rules_python/whltool     1980-01-01 00:00:00         6806
+io_bazel_rules_python/rules_python/whltool     1980-01-01 00:00:00         5681
 pypi__setuptools_38_2_4/__init__.py            1980-01-01 00:00:00            0
 pypi__setuptools_38_2_4/easy_install.py        1980-01-01 00:00:00          126
 pypi__setuptools_38_2_4/pkg_resources/__init__.py 1980-01-01 00:00:00       105991
************************************************************************

----------------------------------------------------------------------
Ran 2 tests in 0.231s

FAILED (failures=2)

https://buildkite.com/bazel/bazel-with-downstream-projects-bazel/builds/193#143a5613-592d-4f0e-a442-ca9a9c21fc86