search

pytest-dev / pytest

The pytest framework makes it easy to write small tests, yet scales to support complex functional testing
https://pytest.org
MIT License
11.92k stars 2.66k forks source link

Sandboxed test execution fails when any plugin is installed #2305

Closed hoefling closed 7 years ago

hoefling commented 7 years ago

I'm building a package using sandboxing (setuptools.sandbox module). I found out that I can't execute tests with pytest once I have installed any plugin. Here are the steps to reproduce from scratch:

  1. Create a fresh virtual environment and preinstall pytest.

    $ virtualenv /tmp/tstenv
$ source /tmp/tstenv/bin/activate
$ pip install pytest pytest-runner
$ pip list
Package       Version
------------- -------
appdirs       1.4.3
packaging     16.8
pip           9.0.1
py            1.4.32
pyparsing     2.2.0
pytest        3.0.6
pytest-runner 2.11.1
setuptools    34.3.2
six           1.10.0
wheel         0.29.0

  2. Add project stub to play with.

    
$ mkdir -p /tmp/pytest-issue
$ cd $_
$ cat <<EOM >setup.py
from setuptools import setup

setup( name='project-stub', version='0.1', packages=[] ) EOM $ cat <sandboxed_setup.py import setuptools.sandbox

setuptools.sandbox.run_setup('setup.py', ['pytest']) EOM $ cat <test_foo.py def test_bar(): assert True EOM


3. Now run the tests in sandboxed mode. This works as expected, one test is found and executed successfully.
```bash
$ python sandboxed_setup.py
running pytest
running egg_info
writing top-level names to project_stub.egg-info/top_level.txt
writing dependency_links to project_stub.egg-info/dependency_links.txt
writing project_stub.egg-info/PKG-INFO
reading manifest file 'project_stub.egg-info/SOURCES.txt'
writing manifest file 'project_stub.egg-info/SOURCES.txt'
running build_ext
============================== test session starts ==============================
platform linux -- Python 3.5.3, pytest-3.0.6, py-1.4.32, pluggy-0.4.0
rootdir: /tmp/pytest-issue, inifile: 
collected 1 items 

test_foo.py .

============================ 1 passed in 0.02 seconds ===========================
  1. Install any pytest plugin and rerun the tests: 
    
$ pip install pytest-variables
$ python sandboxed_setup.py
running pytest
running egg_info
writing dependency_links to project_stub.egg-info/dependency_links.txt
writing project_stub.egg-info/PKG-INFO
writing top-level names to project_stub.egg-info/top_level.txt
reading manifest file 'project_stub.egg-info/SOURCES.txt'
writing manifest file 'project_stub.egg-info/SOURCES.txt'
running build_ext
error: SandboxViolation: mkdir('/tmp/tstenv/lib/python3.5/site-packages/pytest_variables/__pycache__',) {}

The package setup script has attempted to modify files on your system that are not within the EasyInstall build area, and has been aborted.

This package cannot be safely installed by EasyInstall, and may not support alternate installation locations even if you run its setup script by hand. Please inform the package's author and the EasyInstall maintainers to find out if a fix or workaround is available.


<details>
  <summary>Sys info</summary>
  <p>
```bash
$ emerge --info
Portage 2.3.4 (python 3.5.3-final-0, default/linux/amd64/13.0/desktop/plasma, gcc-5.4.0, glibc-2.23-r3, 4.10.1-gentoo x86_64)
=================================================================
System uname: Linux-4.10.1-gentoo-x86_64-Intel-R-_Core-TM-_i3-4100M_CPU_@_2.50GHz-with-gentoo-2.3
KiB Mem:     8055596 total,   1400008 free
KiB Swap:    8388604 total,   8388604 free
Timestamp of repository gentoo: Mon, 13 Mar 2017 20:00:01 +0000
sh bash 4.3_p48-r1
ld GNU ld (Gentoo 2.25.1 p1.1) 2.25.1
distcc 3.2rc1 x86_64-pc-linux-gnu [disabled]
app-shells/bash:          4.3_p48-r1::gentoo
dev-java/java-config:     2.2.0-r3::gentoo
dev-lang/perl:            5.22.3_rc4::gentoo
dev-lang/python:          2.7.12::gentoo, 3.5.3::gentoo, 3.6.0::gentoo
dev-util/cmake:           3.7.2::gentoo
dev-util/pkgconfig:       0.28-r2::gentoo
sys-apps/baselayout:      2.3::gentoo
sys-apps/openrc:          0.23.2::gentoo
sys-apps/sandbox:         2.10-r3::gentoo
sys-devel/autoconf:       2.13::gentoo, 2.69::gentoo
sys-devel/automake:       1.14.1::gentoo, 1.15::gentoo
sys-devel/binutils:       2.25.1-r1::gentoo
sys-devel/gcc:            5.4.0-r3::gentoo
sys-devel/gcc-config:     1.7.3::gentoo
sys-devel/libtool:        2.4.6-r3::gentoo
sys-devel/make:           4.2.1::gentoo
sys-kernel/linux-headers: 4.4::gentoo (virtual/os-headers)
sys-libs/glibc:           2.23-r3::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000

crossdev
    location: /usr/local/portage/overlays/crossdev
    masters: gentoo
    priority: 10

brother-overlay
    location: /usr/local/portage/overlays/brother-overlay
    sync-type: git
    sync-uri: https://github.com/stefan-langenmaier/brother-overlay.git
    masters: gentoo
    priority: 50

hering-overlay
    location: /usr/local/portage/overlays/hering-overlay
    sync-type: git
    sync-uri: https://github.com/internethering/hering-overlay.git
    masters: gentoo
    priority: 50

hoefling-private
    location: /usr/local/portage/overlays/hoefling-private
    sync-type: git
    sync-uri: https://github.com/hoefling/hoefling-private-overlay.git
    masters: gentoo
    priority: 50

pypi
    location: /var/lib/layman/pypi
    masters: gentoo
    priority: 50

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe -march=native"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/maven-bin-3.3/conf /var/lib/hsqldb"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release 
/etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d 
/etc/texmf/web2c"
CXXFLAGS="-O2 -pipe -march=native"
DISTDIR="/usr/portage/distfiles"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news 
parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch 
userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://mirror.eu.oneandone.net/linux/distributions/gentoo/gentoo/ rsync://mirror.eu.oneandone.net/gentoo/ 
http://mirror.netcologne.de/gentoo/ rsync://mirror.netcologne.de/gentoo/ 
http://linux.rz.ruhr-uni-bochum.de/download/gentoo-mirror/ http://ftp.halifax.rwth-aachen.de/gentoo/ 
rsync://ftp.halifax.rwth-aachen.de/gentoo/ http://ftp.uni-erlangen.de/pub/mirrors/gentoo 
rsync://ftp-stud.hs-esslingen.de/gentoo/ http://ftp-stud.hs-esslingen.de/pub/Mirrors/gentoo/"
LANG="en_US.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j5"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete 
--stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="X a52 aac acl acpi alsa amd64 berkdb bluetooth branding bzip2 cairo cdda cdr cli consolekit cracklib crypt cups cxx dbus 
declarative dri dts dvd dvdr emboss encode exif fam ffmpeg firefox flac fortran gdbm gif glamor gpm iconv icu ipv6 jpeg kipi 
kwallet lcms ldap libnotify mad mmx mng modules mp3 mp4 mpeg multilib ncurses networkmanager nls nptl ogg opengl openmp pam 
pango pcre pdf phonon plasma png policykit ppds pulseaudio qml qt5 readline sdl seccomp semantic-desktop session spell sqlite 
sse sse2 ssl startup-notification svg tcpd tiff truetype udev udisks unicode upower usb vorbis wayland widgets wxwidgets x264 
xattr xcb xcomposite xinerama xml xscreensaver xv xvid zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem 
bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio 
via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias 
authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user 
autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include 
info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack 
vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" COLLECTD_PLUGINS="df 
interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm 
earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 
rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="evdev synaptics" KERNEL="linux" 
LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console 
presenter-minimizer" NETBEANS_MODULES="apisupport java javafx profiler websvccommon webcommon enterprise" 
OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6" PYTHON_SINGLE_TARGET="python3_5" PYTHON_TARGETS="python3_5 python3_6 
pypy3" QEMU_SOFTMMU_TARGETS="i386 x86_64" RUBY_TARGETS="ruby21" USERLAND="GNU" VIDEO_CARDS="intel i965" XTABLES_ADDONS="quota2 
psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark 
dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, 
PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON

nicoddemus commented 7 years ago

It seems the problem is that assertion rewriting is writing the cached .pyc files to disk, and sandboxing doesn't like that. You can set PYTHONDONTWRITEBYTECODE to disable this behavior.

RonnyPfannschmidt commented 7 years ago

its also a wont-fix, the setuptools sandbox is broken by design and will eventually disappear

hoefling commented 7 years ago

@RonnyPfannschmidt: interesting, didn't know that. Can you point me on any official info about its removal? As far as I can see, there's no deprecation warnings in the module itself, its code is being maintained (last modified on Feb 24th).