Error removing toil/src directory from python path

[adderan]

When installed from pip (pip install git+http://github.com/bd2kgenomics/toil) into a virtualenv:

Added job: b/k/jobHAlTgv to active jobs Built the jobs list, currently have 1 jobs to update and 0 jobs issued Issuing the command: /home/alden/progressiveCactus/python/bin/python -E /home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py /home/alden/progressiveCactus/submodules/cactus/pipeline/toil b/k/jobHAlTgv with memory: 2147483648, cores: >1, disk: 2147483648 Issued job with job store ID: b/k/jobHAlTgv and job batch system ID: 98 and cores: 1, disk: >2147483648, and memory: 2147483648 Acquiring 2147483648 bytes of memory from pool of 9223372036854775807. Attempting to acquire 10 threads for 1 cpus submitted Acquiring semaphore blockingly. Semaphore acquired. Executing command: '/home/alden/progressiveCactus/python/bin/python -E /home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py /home/alden/progressiveCactus/submodules/cactus/pipeline/toil b/k/jobHAlTgv'. Traceback (most recent call last): File "/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py", line >24, in sys.path.remove(os.path.dirname(os.path.abspath(file))) ValueError: list.remove(x): x not in list Got exit code 1 (indicating failure) from command '/home/alden/progressiveCactus/python/bin/python -E /home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py /home/alden/progressiveCactus/submodules/cactus/pipeline/toil b/k/jobHAlTgv'.

[hannes-ucsc]

I want to get rid of that hack but for now we should just catch the ValueError and move on. I am puzzled though as to why this started happening. When running /foo/bar.py python always adds /foo to sys.path.

BTW: Not a big fan of installing from git. Not every commit to master passes the tests. Does "pip install --pre toil" not work? That way you'd get a build that is guaranteed to have passed the test suite.

[adderan]

Okay, I'll switch to the pre-release version.

[adderan]

Actually it looks like this fix won't be good enough. I'm getting the conflict with toil.resource now:

n/n/jobZpx21Q: File "/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py", line 221, in main n/n/jobZpx21Q: startClock = getTotalCpuTime() n/n/jobZpx21Q: File "/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/lib/bioio.py", line 179, in getTotalCpuTime n/n/jobZpx21Q: return getTotalCpuTimeAndMemoryUsage()[0] n/n/jobZpx21Q: File "/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/lib/bioio.py", line 170, in getTotalCpuTimeAndMemoryUsage n/n/jobZpx21Q: me = resource.getrusage(resource.RUSAGE_SELF) n/n/jobZpx21Q: AttributeError: 'module' object has no attribute 'getrusage'

[hannes-ucsc]

Could you please print file and sys.path and post the sys.path right before this happens and post the output here?

[adderan]

0/H/job6n3SVo:  ---TOIL WORKER OUTPUT LOG---
0/H/job6n3SVo:  Next available file descriptor: 7
0/H/job6n3SVo:  DEBUG:__main__:Next available file descriptor: 7
0/H/job6n3SVo:  Parsed job
0/H/job6n3SVo:  DEBUG:__main__:Parsed job
0/H/job6n3SVo:  INFO:toil.lib.bioio:Path = ['/home/alden/progressiveCactus/python/lib/python2.7/site-packages/toil', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/psutil-0.5.0-py2.7-linux-x86_64.egg', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages/psutil-0.5.0-py2.7-linux-x86_64.egg', '/home/alden/progressiveCactus/python/lib/python2.7', '/home/alden/progressiveCactus/python/lib/python2.7/plat-x86_64-linux-gnu', '/home/alden/progressiveCactus/python/lib/python2.7/lib-tk', '/home/alden/progressiveCactus/python/lib/python2.7/lib-old', '/home/alden/progressiveCactus/python/lib/python2.7/lib-dynload', '/usr/lib/python2.7', '/usr/lib/python2.7/lib-tk', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages', '/home/alden/progressiveCactus/submodules']
0/H/job6n3SVo:  INFO:toil.lib.bioio:__file__ = /home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/lib/bioio.pyc
0/H/job6n3SVo:  Traceback (most recent call last):
0/H/job6n3SVo:    File "/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py", line 221, in main
0/H/job6n3SVo:      startClock = getTotalCpuTime()
0/H/job6n3SVo:    File "/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/lib/bioio.py", line 174, in getTotalCpuTime
0/H/job6n3SVo:      return getTotalCpuTimeAndMemoryUsage()[0]
0/H/job6n3SVo:    File "/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/lib/bioio.py", line 165, in getTotalCpuTimeAndMemoryUsage
0/H/job6n3SVo:      me = resource.getrusage(resource.RUSAGE_SELF)
0/H/job6n3SVo:  AttributeError: 'module' object has no attribute 'getrusage'
0/H/job6n3SVo:  Exiting the worker because of a failed job on host thinkpad
0/H/job6n3SVo:  ERROR:__main__:Exiting the worker because of a failed job on host thinkpad
0/H/job6n3SVo:  WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job 0/H/job6n3SVo to 0

[hannes-ucsc]

Sorry, I meant printing sys.path and __file__ in worker.py before the sys.path modification is done.

[adderan]

>Traceback (most recent call last):
>  File "/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py", line >28, in <module>
>    sys.path.remove(os.path.dirname(os.path.abspath(__file__)))
>ValueError: list.remove(x): x not in list
>sys.path: ['/home/alden/progressiveCactus/python/lib/python2.7/site-packages/toil', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/psutil-0.5.0-py2.7-linux-x86_64.egg', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages/psutil-0.5.0-py2.7-linux-x86_64.egg', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg', '/home/alden/progressiveCactus/python/lib/python2.7', '/home/alden/progressiveCactus/python/lib/python2.7/plat-x86_64-linux-gnu', '/home/alden/progressiveCactus/python/lib/python2.7/lib-tk', '/home/alden/progressiveCactus/python/lib/python2.7/lib-old', '/home/alden/progressiveCactus/python/lib/python2.7/lib-dynload', '/usr/lib/python2.7', '/usr/lib/python2.7/lib-tk', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages']
>__file__: /home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py

[hannes-ucsc]

I think I know what it is. Can you print the result of os.path.abspath(file) as well. I think one of the directories that make up the path /home/alden/progressiveCactus/python/lib/python2.7/site-packages/toil is a symlink.

[adderan]

sys.path: ['/home/alden/progressiveCactus/python/lib/python2.7/site-packages/toil', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/psutil-0.5.0-py2.7-linux-x86_64.egg', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages/psutil-0.5.0-py2.7-linux-x86_64.egg', '/home/alden/progressiveCactus/python/lib/python2.7', '/home/alden/progressiveCactus/python/lib/python2.7/plat-x86_64-linux-gnu', '/home/alden/progressiveCactus/python/lib/python2.7/lib-tk', '/home/alden/progressiveCactus/python/lib/python2.7/lib-old', '/home/alden/progressiveCactus/python/lib/python2.7/lib-dynload', '/usr/lib/python2.7', '/usr/lib/python2.7/lib-tk', '/home/alden/progressiveCactus/python/local/lib/python2.7/site-packages', '/home/alden/progressiveCactus/python/lib/python2.7/site-packages']

os.path.aspath(file): /home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil/worker.py

os.path.dirname(os.path.abspath(file)): /home/alden/progressiveCactus/python/local/lib/python2.7/site-packages/toil

[hannes-ucsc]

It seems that /home/alden/progressiveCactus/python/local is a symlink to /home/alden/progressiveCactus/python

Could you verify that by running ls -l /home/alden/progressiveCactus/python

[adderan]

alden@thinkpad:~$ ls -l /home/alden/progressiveCactus/python total 20 drwxrwxr-x 2 alden alden 4096 Aug 28 11:36 bin drwxrwxr-x 2 alden alden 4096 Aug 27 16:50 include drwxrwxr-x 3 alden alden 4096 Aug 27 16:50 lib drwxrwxr-x 2 alden alden 4096 Aug 27 16:50 local drwxrwxr-x 3 alden alden 4096 Aug 27 16:50 share

On Fri, Aug 28, 2015 at 2:35 PM, Hannes Schmidt notifications@github.com wrote:

It seems that /home/alden/progressiveCactus/python/local is a symlink to /home/alden/progressiveCactus/python

Could you verify that by running ls -l /home/alden/progressiveCactus/python

— Reply to this email directly or view it on GitHub https://github.com/BD2KGenomics/toil/issues/290#issuecomment-135894036.

[hannes-ucsc]

Just for posterity: Alden and I confirmed that he's got symlinks in /home/alden/progressiveCactus/python/local. Solution will be to apply abspath to each sys.path before comparing and removing.