AddressSanitizer: Skip tests directly in Python, not with external config

[vstinner]

BPO	46633
Nosy	@vstinner, @pablogsal
PRs	python/cpython#31632 python/cpython#31634 python/cpython#31644

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = created_at = labels = ['tests', '3.11'] title = 'AddressSanitizer: Skip tests directly in Python, not with external config' updated_at = user = 'https://github.com/vstinner' ``` bugs.python.org fields: ```python activity = actor = 'vstinner' assignee = 'none' closed = True closed_date = closer = 'vstinner' components = ['Tests'] creation = creator = 'vstinner' dependencies = [] files = [] hgrepos = [] issue_num = 46633 keywords = ['patch'] message_count = 11.0 messages = ['412499', '412501', '412506', '414247', '414248', '414250', '414262', '414266', '414370', '414379', '414380'] nosy_count = 2.0 nosy_names = ['vstinner', 'pablogsal'] pr_nums = ['31632', '31634', '31644'] priority = 'normal' resolution = 'fixed' stage = 'resolved' status = 'closed' superseder = None type = None url = 'https://bugs.python.org/issue46633' versions = ['Python 3.11'] ```

Linked PRs

• gh-108286
• gh-108289

[vstinner]

It seems like bpo-45200: "Address Sanitizer: libasan dead lock in pthread_create() (test_multiprocessing_fork.test_get() hangs)" is not fixed yet.

In the GitHub Action job, test_multiprocessing_fork is skipped because it's too slow, so the job doesn't hang.

Yesterday, I modified the ASAN buildbot to double its timeout from 20 to 40 minutes: https://github.com/python/buildmaster-config/commit/5a37411e75c9475d48eabaac18102a3c9fc2d467

But it's useful, when it hangs, it hangs forever. Exmaple on the AMD64 Arch Linux Asan Debug 3.9 buildbot (with the new config):

--- (test.test_multiprocessing_fork.WithProcessesTestPicklingConnections) ... ok Timeout (0:35:00)! --- https://buildbot.python.org/all/#/builders/588/builds/332

Tests are tuned for ASAN, but the configuration is copied and inconsistent between the GitHub Actions job and the buildbot configuration.

I propose to move this configuration directly into Python.

test_decimal.py checks for "-fsanitize=address" in CFLAGS and skip some tests if it's present.

[vstinner]

Notes:

• test.support has check_sanitizer() function. Tests using it:

  • test_crypt
  • test_idle
  • test_tix
  • test_tk
  • test_ttk_guionly

• test_decimal also checks for memory sanitizer and skip some tests if it's present:

MEMORY_SANITIZER = (
    '-fsanitize=memory' in _cflags or
    '--with-memory-sanitizer' in _config_args
)   

• test_faulthandler suppress crash report, similir to support.SuppressCrashReport, directly in the C code used by tests: faulthandler_suppress_crash_report() function.

• Objects/obmalloc.c checks "#if __has_feature(address_sanitizer)": is ASAN enabled?

• GitHub Action config: .github/workflows/build.yml. Skipped tests:

  • test_all
  • test_concurrent_futures
  • test_multiprocessing_fork
  • test_multiprocessing_forkserver
  • test_multiprocessing_spawn
  • test_peg_generator
  • test_tools

Comment: --- # Skip test_tools test_peg_generator test_concurrent_futures because # there are too slow: between 5 and 20 minutes on this CI. # # Skip multiprocessing and concurrent.futures tests which are affected by # bpo-45200 bug: libasan dead lock in pthread_create(). # # test_all is skipped because importing some modules directly can trigger # known problems with ASAN (like tk or crypt). ---

• Buildbot configuration: UnixAsanBuild class of master/custom/factories.py. Options:

  ASAN_OPTIONS='detect_leaks=0:allocator_may_return_null=1:handle_segv=0'

With the comment: --- # See https://bugs.python.org/issue42985 for more context on why # SIGSEGV is ignored on purpose. ---

Skipped tests:

• test_ctypes
• test_capi
• test_crypt
• test_decimal
• test_faulthandler
• test_interpreters

With the comment: --- # These tests are currently raising false positives or are interfering with the ASAN mechanism, # so we need to skip them unfortunately. ---

[pablogsal]

I have recently added some decorators in test. support to deactivate tests if running under the sanitizers.

[vstinner]

I created a PR to no longer skip tests in buildbots: https://github.com/python/buildmaster-config/pull/314

[vstinner]

Status with the 2 pending PRs:

• python/cpython#75813
• https://github.com/python/buildmaster-config/pull/314

Tests only skipped on ASAN:

• _test_multiprocessing

Skip on ASAN and MSAN:

• test_all
• test_concurrent_futures
• test_crypt
• test_decimal.test_maxcontext_exact_arith()
• test_idle
• test_peg_generator
• test_tix
• test_tk
• test_tools
• test_ttk_guionly

No longer skipped:

• test_capi
• test_ctypes
• test_faulthandler
• test_interpreters

[vstinner]

I created bpo-46887: ./Programs/_freeze_module fails with MSAN: Uninitialized value was created by an allocation of 'stat.i'.

[vstinner]

""" No longer skipped:

• test_capi
• test_ctypes
• test_faulthandler
• test_interpreters """

I built Python manually with:

./configure --with-pydebug CC=clang LD=clang --with-address-sanitizer

These tests no longer with ASAN_OPTIONS used on our CI:

ASAN_OPTIONS='detect_leaks=0:allocator_may_return_null=1:handle_segv=0' ./python -m test (...)

[vstinner]

New changeset 9204bb72a2da5885facc747e63d2bd2d654606fe by Victor Stinner in branch 'main': bpo-46633: Skip tests on ASAN and/or MSAN builds (GH-31632) https://github.com/python/cpython/commit/9204bb72a2da5885facc747e63d2bd2d654606fe

[vstinner]

New changeset 93264452d952d9ba604bacf2947c2df5dd477931 by Victor Stinner in branch '3.10': [3.10] bpo-46633: Skip tests on ASAN and/or MSAN builds (GH-31632) (GH-31634) https://github.com/python/cpython/commit/93264452d952d9ba604bacf2947c2df5dd477931

[vstinner]

New changeset 359bc392ba2b8f2acca223426c8210bb74f724c6 by Victor Stinner in branch '3.9': [3.10] bpo-46633: Skip tests on ASAN and/or MSAN builds (GH-31632) (GH-31634) (GH-31644) https://github.com/python/cpython/commit/359bc392ba2b8f2acca223426c8210bb74f724c6

[vstinner]

https://github.com/python/buildmaster-config/pull/314

I merged this PR as well. The new buildbot configuration will be deployed soon.

[vstinner]

Lib/test/test_asyncio/test_subprocess.py and Lib/test/test_distutils.py are now skipped on ASAN tests by the commit: https://github.com/python/cpython/commit/f6dd14c65336cda4e2ebccbc6408dfe3b0a68a34 (cc @gpshead)

[gpshead]

FWIW I disabled those two because they triggered ASAN on Ubuntu 20.04's gcc 9.4 and in github's CI (which might use that?). They do not trigger ASAN on more recent Debian bullseye or Ubuntu 22.04 based systems with a more recent gcc.

In the _posixsubprocess.c case it was at a callsite that has ~25 parameters (ugh) and just gained one more. But the asan assertion would only trigger during one consistent but seemingly random test_asyncio.test_subprocess test rather than all the time; so if there is a bug in the ASAN implementation itself it isn't clear what it was.

[vstinner]

I enabled test_faulthandler on ASAN: commit 58f9c635002673e35a56eeaa122cade7a2b6bae1.

[vstinner]

I enabled more tests in test_all: commit https://github.com/python/cpython/commit/a541e01537cc07b326cc37cc29412b816bc2b4fc