search

PolicyStat / terrarium

Package and ship relocatable python virtualenvs, like a boss.
https://terrarium.readthedocs.io/en/latest
BSD 3-Clause "New" or "Revised" License
171 stars 14 forks source link

A failed build can still register a hash, dooming subsequent runs on the same machine #2

Closed winhamwr closed 12 years ago

winhamwr commented 12 years ago

The first Jenkins build failed on the lxml requirement (http timeout), but then the next build saw the first hash somehow and used it. This is despite the fact that the hash was totally busted.

When run from a different machine, it works fine (since it wasn't on S3).

Output from first Jenkins run:

Started by user Hubot Botson
Building in workspace /var/lib/jenkins/jobs/pstat_ticket/workspace
Checkout:workspace / /var/lib/jenkins/jobs/pstat_ticket/workspace - hudson.remoting.LocalChannel@a68bf1
Using strategy: Default
Last Built Revision: Revision d5b2e6ec5de7260c4d7d39462530418e28f045bf (origin/issue_254)
Fetching changes from 1 remote Git repository
Fetching upstream changes from git@github.com:PolicyStat/PolicyStat.git
Commencing build of Revision 094ac57b21256f42d08293ea97ce6f8eb81d7d01 (origin/issue_254)
Checking out Revision 094ac57b21256f42d08293ea97ce6f8eb81d7d01 (origin/issue_254)
[workspace] $ /bin/bash /tmp/hudson6392217377009833731.sh
error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the
installation directory:

    [Errno 13] Permission denied: '/usr/local/lib/python2.6/dist-packages/test-easy-install-20879.pth'

The installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

    /usr/local/lib/python2.6/dist-packages/

Perhaps your account does not have write access to this directory?  If the
installation directory is a system-owned directory, you may need to sign in
as the administrator or "root" account.  If you do not have administrative
access to this machine, you may wish to choose a different installation
directory, preferably one that is listed in your PYTHONPATH environment
variable.

For information on other options, you may wish to consult the
documentation at:

  http://packages.python.org/distribute/easy_install.html

Please make the appropriate changes for your system and try again.

[terrarium]INFO:Building fresh environment at /var/lib/jenkins/virtualenvs/d6a0e239c19f6593cef3981325a74f98
New python executable in /var/lib/jenkins/virtualenvs/d6a0e239c19f6593cef3981325a74f98/bin/python
Installing distribute.............................................................................................................................................................................................done.
Installing pip...............done.
Downloading/unpacking uWSGI==1.1.2 (from -r config/ec2_scripts/tpl/pstat/sys_requirements.txt (line 1))
  Running setup.py egg_info for package uWSGI
Downloading/unpacking newrelic==1.2.0.246 (from -r config/ec2_scripts/tpl/pstat/sys_requirements.txt (line 2))
  Running setup.py egg_info for package newrelic
Downloading/unpacking lxml==2.2.4 (from -r config/ec2_scripts/tpl/pstat/sys_requirements.txt (line 3))
Exception:
Traceback (most recent call last):
  File "/var/lib/jenkins/virtualenvs/d6a0e239c19f6593cef3981325a74f98/lib/python2.6/site-packages/pip-1.1-py2.6.egg/pip/basecommand.py", line 104, in main
    status = self.run(options, args)
  File "/var/lib/jenkins/virtualenvs/d6a0e239c19f6593cef3981325a74f98/lib/python2.6/site-packages/pip-1.1-py2.6.egg/pip/commands/install.py", line 245, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/var/lib/jenkins/virtualenvs/d6a0e239c19f6593cef3981325a74f98/lib/python2.6/site-packages/pip-1.1-py2.6.egg/pip/req.py", line 985, in prepare_files
    self.unpack_url(url, location, self.is_download)
  File "/var/lib/jenkins/virtualenvs/d6a0e239c19f6593cef3981325a74f98/lib/python2.6/site-packages/pip-1.1-py2.6.egg/pip/req.py", line 1109, in unpack_url
    retval = unpack_http_url(link, location, self.download_cache, self.download_dir)
  File "/var/lib/jenkins/virtualenvs/d6a0e239c19f6593cef3981325a74f98/lib/python2.6/site-packages/pip-1.1-py2.6.egg/pip/download.py", line 451, in unpack_http_url
    download_hash = _download_url(resp, link, temp_location)
  File "/var/lib/jenkins/virtualenvs/d6a0e239c19f6593cef3981325a74f98/lib/python2.6/site-packages/pip-1.1-py2.6.egg/pip/download.py", line 368, in _download_url
    chunk = resp.read(4096)
  File "/usr/lib/python2.6/socket.py", line 353, in read
    data = self._sock.recv(left)
  File "/usr/lib/python2.6/httplib.py", line 538, in read
    s = self.fp.read(amt)
  File "/usr/lib/python2.6/socket.py", line 353, in read
    data = self._sock.recv(left)
timeout: timed out

Storing complete log in /var/lib/jenkins/.pip/pip.log
after install
creating symlinks
Traceback (most recent call last):
  File "/tmp/pstat_bootstrap_P25F0O.py", line 2380, in <module>
    main()
  File "/tmp/pstat_bootstrap_P25F0O.py", line 932, in main
    after_install(options, home_dir)
  File "/tmp/pstat_bootstrap_P25F0O.py", line 1848, in after_install
    pip_install(options, **kwargs)
  File "/tmp/pstat_bootstrap_P25F0O.py", line 1827, in pip_install
    call_subprocess(command)
  File "/tmp/pstat_bootstrap_P25F0O.py", line 1008, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /var/lib/jenkins/vir...981325a74f98/bin/pip install -r config/ec2_scripts/t...sys_requirements.txt -r config/ec2_scripts/t...dev_requirements.txt -r config/ec2_scripts/t...src_requirements.txt -r config/ec2_scripts/tpl/pstat/requirements.txt failed with error code 2
Traceback (most recent call last):
  File "/var/lib/jenkins/jobs/pstat_ticket/workspace/scripts/terrarium.py", line 512, in <module>
    main()
  File "/var/lib/jenkins/jobs/pstat_ticket/workspace/scripts/terrarium.py", line 387, in main
    archive = env.build_fresh()
  File "/var/lib/jenkins/jobs/pstat_ticket/workspace/scripts/terrarium.py", line 137, in build_fresh
    call_subprocess(['python', bootstrap] + requirements + [self.path])
  File "/usr/local/lib/python2.6/dist-packages/virtualenv.py", line 1006, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command python /tmp/pstat_bootstrap_P25F0O.py -r config/ec2_scripts/t...sys_requirements.txt -r config/ec2_scripts/t...dev_requirements.txt -r config/ec2_scripts/t...src_requirements.txt -r config/ec2_scripts/tpl/pstat/requirements.txt /var/lib/jenkins/vir...6593cef3981325a74f98 failed with error code 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording fingerprints
Finished: FAILURE

Ouput from the subsequent Jenkins run:

Started by user Hubot Botson
Building in workspace /var/lib/jenkins/jobs/pstat_ticket/workspace
Checkout:workspace / /var/lib/jenkins/jobs/pstat_ticket/workspace - hudson.remoting.LocalChannel@a68bf1
Using strategy: Default
Last Built Revision: Revision 094ac57b21256f42d08293ea97ce6f8eb81d7d01 (origin/issue_254)
Fetching changes from 1 remote Git repository
Fetching upstream changes from git@github.com:PolicyStat/PolicyStat.git
Commencing build of Revision 094ac57b21256f42d08293ea97ce6f8eb81d7d01 (origin/issue_254)
Checking out Revision 094ac57b21256f42d08293ea97ce6f8eb81d7d01 (origin/issue_254)
[workspace] $ /bin/bash /tmp/hudson7390782146878434149.sh
error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the
installation directory:

    [Errno 13] Permission denied: '/usr/local/lib/python2.6/dist-packages/test-easy-install-20973.pth'

The installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

    /usr/local/lib/python2.6/dist-packages/

Perhaps your account does not have write access to this directory?  If the
installation directory is a system-owned directory, you may need to sign in
as the administrator or "root" account.  If you do not have administrative
access to this machine, you may wish to choose a different installation
directory, preferably one that is listed in your PYTHONPATH environment
variable.

For information on other options, you may wish to consult the
documentation at:

  http://packages.python.org/distribute/easy_install.html

Please make the appropriate changes for your system and try again.

[terrarium]INFO:Using existing environment at /var/lib/jenkins/virtualenvs/d6a0e239c19f6593cef3981325a74f98 (md5 matches)
Archiving artifacts
Recording fingerprints
Triggering a new build of pstat_ticket_selenium #2130
Triggering a new build of pstat_ticket_unittest #2023
Finished: SUCCESS
kylegibson commented 12 years ago

1 may have fixed this.

kylegibson commented 12 years ago

This is no longer an issue since terrarium always rebuilds each time