Test - Githubissues

francicco commented 6 years ago

Hi, I just installed the toolkit and I tried the test to verity if everything's fine

luigid --background --logdir luigi_logs

luigi --module cat RunCat --hal=test_data/vertebrates.hal --ref-genome=mm10 --workers=10 --config=test_data/test.config --work-dir test_install --out-dir test_install --local-scheduler --augustus --augustus-cgp --augustus-pb --assembly-hub --binary-mode=local > log.txt

I got errors:

DEBUG:matplotlib:$HOME=/home/fc464
DEBUG:matplotlib:matplotlib data path /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/matplotlib/mpl-data
DEBUG:matplotlib:loaded rc file /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/matplotlib/mpl-data/matplotlibrc
DEBUG:matplotlib:matplotlib version 2.2.3
DEBUG:matplotlib:interactive is False
DEBUG:matplotlib:platform is linux2
DEBUG:matplotlib:loaded modules: ['requests.Cookie', 'pandas._libs.numpy', 'pandas.errors.pandas', 'luigi.cmdline', 'distutils.sysconfig', 'pandas.plotting.contextlib', 'sqlalchemy.engine.contextlib', 'numpy.core.info', 'luigi.time', 'logging.weakref', 'dateutil.tz.os', 'pprint', 'past.translation.os', 'dbm.gnu', 'luigi.target', 'dill.objtypes', 'SocketServer', 'docker.models.copy', 'numpy.core._multiarray_tests', 'chardet.mbcssm', 'xml.etree.sys', 'requests.packages.urllib3.util.wait', 'requests.packages.urllib3.util.collections', 'requests.packages.chardet.hebrewprober', 'shlex', 'sqlalchemy.sql.operators', 'pandas._libs.pytz', 'zlib', 'websocket.hashlib', 'pkg_resources._vendor.pprint', 'docker.models', 'matplotlib.cbook.warnings', 'numpy.testing._private.pytesttester', 'pkg_resources.extern.appdirs', 'past.builtins.__builtin__', 'tarfile', 'UserList', 'email.random', 'libfuturize.fixes.lib2to3', 'pandas.io.formats.format', 'lib2to3.pgen2', 'pandas.core.types', 'codecs', 'docker.constants', 'cat.logging', 'numpy.core._internal', 'luigi.types', 'pandas.io', 'distutils.sys', 'toil.lib.past', 'requests.packages.chardet.langbulgarianmodel', 'urllib3.util.math', 'future.utils.sys', 'pkg_resources._vendor.datetime', 'toil.fileStore', 'ctypes._ctypes', '_heapq', 'requests.packages.urllib3.util', 'docker.transport', 'luigi.configuration', 'pandas._libs.tslibs.timedeltas', 'email.sys', 'urllib3.packages._abcoll', 'tokenize', 'dateutil.tz._common', 'requests.chardet', 'pandas.compat.StringIO', 'chardet.langturkishmodel', 'cPickle', 'unittest.sys', 'requests.packages.urllib3.filepost', 'pandas.core.accessor', 'pandas.core.dtypes.pandas', 'anydbm', 'docker.six', 'future.backports.email.iterators', 'pkg_resources._vendor.copy', 'dateutil.tz', 'requests.packages.chardet.langcyrillicmodel', 'fnmatch', 'pandas.io.formats.locale', 'requests.packages.chardet.charsetprober', 'logging.cStringIO', 'luigi.configuration.toml', 'past.builtins.past', 'numpy.compat', 'dateutil.parser.__future__', 'chardet.version', 'sqlalchemy.weakref', 'numpy.core.records', 'email.message', 'toil.batchSystems', 'urllib3.fields', 'strop', 'pandas.core.sparse.__future__', 'pandas.core.arrays.pandas', 'requests.packages.chardet.gb2312freq', 'pandas._libs.hashing', 'urllib.request', 'lib2to3.itertools', 'luigi.rpc', 'pandas.compat.unicodedata', 'future.types.collections', 'docker.utils.fnmatch', 'os.path', 'docker.transport.unixconn', 'pandas.compat.types', 'email.base64mime', 'pkg_resources._vendor.packaging.version', 'luigi.cPickle', 'cat.datetime', 'idna.idnadata', 'luigi.task_status', 'functools', 'pkg_resources._vendor.weakref', 'pandas.core.groupby.copy', 'distutils.fancy_getopt', 'pkg_resources._vendor._thread', 'pandas.plotting.warnings', 'uuid', 'sqlalchemy.cresultproxy', 'lib2to3.pgen2.pgen', 'pandas.core.operator', 'imp', 'multiprocessing.os', 'docker.api.json', 'docker.models.plugins', 'enum.sys', 'websocket.ssl', 'pandas.util.pandas', 'unittest.time', 'sqlalchemy.sql.crud', 'lib2to3.pgen2.logging', 'pkg_resources.errno', 'dill.source', 'pandas._libs.properties', 'pandas.io.common', 'sqlalchemy.engine.util', 'docker.utils.datetime', 'dateutil.tz.struct', 'luigi.abc', 'numpy.add_newdocs', 'docker.api.volume', 'docker.utils.os', 'requests.packages.chardet.universaldetector', 'pandas.core.sparse.array', 'docker.transport.requests', 'pytz.pytz', 'numpy.random.threading', 're', 'email.quopri', 'docker.models.resource', 'sqlalchemy.sql.selectable', 'numpy.testing._private', 'matplotlib.json', '_locale', 'logging', 'docker.api.os', 'traceback', 'sqlalchemy.sql.dml', 'pandas.core.series', 'sqlalchemy.cutils', 'future.types.newbytes', 'pandas.core.pandas', 'ctypes.re', 'sqlalchemy.util.hashlib', 'future.backports.email.base64mime', 'luigi.argparse', 'dateutil.six', 'pandas.io.urllib', 'sqlalchemy.util.cStringIO', 'pandas._libs.tslibs.nattype', 'pkg_resources._vendor.packaging._compat', 'numpy.core.arrayprint', 'pandas.tseries.pandas', 'pandas.io.codecs', 'pandas._libs.hashtable', 'pandas.core.tools.datetimes', 'keyword', 'websocket._abnf', 'pandas._libs.tslibs.strptime', 'stringprep', 'docker.api.image', 'pandas.io.zipfile', 'pandas.core.nanops', 'future.backports.email.parser', 'bd2k', 'reprlib', 'dill.types', 'websocket', 'requests.packages.chardet.euckrprober', 'pandas.util._validators', 'requests.packages.urllib3.packages.thread', '__main__', 'numpy.sys', 'requests.os', 'pandas.core.gc', 'email.MIMEMultipart', 'unittest.difflib', 'numpy.lib.index_tricks', 'future.utils.codecs', 'dockerpycreds.store', 'pandas.core.groupby.functools', 'toil.lib.misc', 'numpy.__config__', 'requests.packages.chardet.logging', 'sqlalchemy.util.cPickle', 'future.types.string', 'websocket._exceptions', 'requests.cookielib', 'dbhash', 'numpy.lib._iotools', 'numpy.lib.npyio', 'random', 'websocket.__future__', 'sqlalchemy.util.time', 'websocket.array', 'requests.urlparse', 'pandas._libs.pandas', 'email.time', 'luigi.io', 'sqlalchemy.engine.re', 'requests.packages.urllib3.packages.six.moves', 'pkg_resources._vendor.warnings', 'xml.etree.ElementTree', 'requests.packages.urllib3.fields', 'numpy.lib._version', 'websocket.threading', 'UserDict', 'pytz.tzinfo', 'pytz.UserDict', 'numpy.lib.type_check', 'pandas.core.indexes.interval', 'pandas.core.groupby.numpy', 'pyexpat.model', 'pkg_resources._vendor.six.moves', 'dateutil.easter', 'docker.transport.six', 'pandas.compat.chainmap', 'sqlalchemy.sql.inspect', 'pandas.core.indexes.__future__', 'dateutil.parser.io', 'requests.status_codes', 'sqlalchemy.engine', 'sqlalchemy.engine.interfaces', 'urllib.response', 'pandas.io.urlparse', 'luigi.itertools', 'pandas._libs.indexing', 'urllib', 'pandas.plotting.operator', 'toil.provisioners.aws.toil', 'lib2to3.pgen2.os', 'pandas.io.httplib', 'future.backports.email.header', 'chardet.mbcharsetprober', 'sqlalchemy.log', 'Queue', 'numpy.lib.info', 'ctypes', 'pandas._libs.tslibs', 'pandas.plotting', 'unittest.signal', 'luigi.worker', 'requests', 'sqlalchemy.sql.base', 'sqlalchemy.sql', 'toil.provisioners', 'six.moves', 'luigi.operator', 'email.uu', 'urllib3.poolmanager', 'email.calendar', 'future.types.future', 'encodings.unicode_escape', 'sqlalchemy.re', 'pandas._libs.distutils', 'pkgutil', 'luigi.six.moves', 'requests.packages.idna.core', 'numpy.core._methods', 'collections', 'numpy', 'subprocess32', 'requests.packages.urllib3.util.connection', 'pytz', 'docker.transport.socket', 'test.re', 'websocket.select', 'urllib3.packages.ssl_match_hostname.backports', 'requests.packages.chardet.euctwprober', 'pandas.core.contextlib', 'getpass', 'encodings.re', 'pandas._libs.internals', 'chardet.re', 'toil.provisioners.aws.six', 'toil.lib.threading', 'json.decoder', 'pkg_resources._vendor.appdirs', 'sqlalchemy.engine.sys', 'zipimport', 'io', 'luigi.tools.itertools', 'sqlalchemy.util.base64', 'luigi.configuration.ConfigParser', 'requests.packages.idna.idnadata', 'idna.package_data', 'bsddb.dbutils', 'numpy.core', 'requests.packages.chardet.sbcharsetprober', 'rfc822', 'test.warnings', 'xml.dom.xmlbuilder', 'pkg_resources._vendor.packaging', '_weakref', 'pkg_resources.extern.pyparsing', 'urlparse', 'sqlalchemy.sql.re', 'pandas._libs.tslibs.re', 'gzip', 'requests.packages.urllib3.packages.six', '_bisect', 'lib2to3.codecs', 'websocket.hmac', 'matplotlib.colors', 'toil.lib.__future__', 'numpy.random.info', 'docker.ssl', 'docker.types.warnings', 'toil.resource', 'docker.models.secrets', 'luigi.random', 'unittest.main', 'test.importlib', 'pkg_resources._vendor.packaging._structures', 'pandas._libs.tslibs.calendar', 'textwrap', 'certifi.warnings', 'pandas.tseries.frequencies', 'ConfigParser', 'chardet.logging', 'websocket._url', 'requests.packages.urllib3.util.math', 'requests.socket', 'pandas.compat.cPickle', 'pkg_resources.py31compat', 'cat.itertools', 'websocket._cookiejar', 'pandas.core.groupby.types', 'numpy.core.getlimits', 'pandas.core.indexes.numeric', 'pandas.datetime', 'logging.traceback', 'toil.realtimeLogger', 'pandas.plotting._converter', 'numpy.lib.mixins', 'dill.os', 'test.UserDict', 'pandas.core.config_init', 'docker.utils.tempfile', 'requests.packages.chardet.eucjpprober', 'requests.packages.chardet.langgreekmodel', 'mpl_toolkits', 'chardet.big5freq', 'websocket._http', 'chardet.codecs', 'urllib3.util.ssl_', 'quopri', 'pandas.tseries.functools', 'numpy.linalg.lapack_lite', 'marshal', 'libfuturize.fixes', 'matplotlib.compat', 'pandas.core.api', 'requests.packages.urllib3.packages.ssl_match_hostname', 'logging.thread', 'sqlalchemy.sql.type_api', 'pandas.core.indexes', '_socket', 'email.Utils', 'sqlalchemy', 'past.builtins.__future__', 'pandas.io.formats.console', '_bsddb', 'pandas.compat.cStringIO', '_pyio', 'toil.provisioners.aws', 'sqlalchemy.util.threading', 'sqlalchemy.engine.sqlalchemy', 'future.types.copy', 'numpy.fft', 'numpy.random.numpy', 'pandas._libs.tslibs.thread', 'requests.contextlib', 'mimetools', 'lib2to3.warnings', 'urllib3.contrib', 'pandas.errors', 'dill.operator', 'pandas.core.indexes.sys', 'sqlalchemy.util.topological', 'pandas._libs.datetime', 'numpy.polynomial.polynomial', 'future.types.newint', 'test', 'lib2to3.pgen2.re', 'genericpath', 'urllib3', 'urllib2', 'ctypes.ctypes', 'future.types.newlist', 'urllib3.packages.ssl_match_hostname', 'chardet.hebrewprober', 'websocket.sys', 'time', 'idna', 'toil.lib.retry', 'pandas.core.indexes.pandas', 'pandas.core.indexes.frozen', 'lib2to3.collections', 'dbm', 'luigi.ast', 'docker.utils.tarfile', 'requests.packages.urllib3.packages.six.moves.http_client', 'requests.packages.chardet.re', 'docker.api.struct', 'sqlalchemy.processors', 'sqlalchemy.pool', '_posixsubprocess32', 'luigi.sys', 'pandas.core.tokenize', 'string', 'toil.lib', '_elementtree', 'sqlalchemy.util.operator', 'future.backports.email.encoders', 'email.quoprimime', 'numpy.ctypeslib', 'test.os', 'datetime', 'numpy.core.machar', 'pandas.core.sparse.pandas', 'lib2to3.pygram', 'urllib3.util.retry', 'sqlalchemy.engine.weakref', 'sqlalchemy.engine.result', 'pandas.core.dtypes.numbers', 'numpy.ma.extras', 'luigi.contrib.luigi', 'requests.logging', 'requests.packages.chardet.escsm', 'multiprocessing', 'dis', 'requests.packages.chardet.sbcsgroupprober', 'pandas._libs.tslibs.textwrap', 'pytz.datetime', 'atexit', 'docker.utils.struct', 'sqlalchemy.sys', 'numpy.fft.helper', '_ctypes', 'exceptions', 'docker.api', 'pandas._libs.join', 'email.socket', 'sqlalchemy.collections', 'requests.packages.urllib3.packages.six.moves.urllib.parse', '_version', 'websocket._socket', 'weakref', 'numpy.lib.arraypad', 'pandas.core.groupby', 'encodings.binascii', 'luigi.tools.re', 'docker.utils.errno', 'pandas.core.functools', 'email.FeedParser', 'future.types.newdict', 'email.charset', 'logging.re', 'pandas.core.util.numpy', 'six.moves.urllib', 'email.MIMEMessage', 'email._parseaddr', 'pandas.plotting.dateutil', 'pandas.core.tools.pandas', 'luigi.tools.range', 'future.builtins.future_builtins', 'docker.transport.ssladapter', 'lib2to3.logging', 'pandas.core.tools.timedeltas', 'email.iterators', 'past.translation.imp', 'numpy.core.fromnumeric', '_ast', 'pandas.io.formats.sys', 'pandas._libs.tslib', 'cat', 'sqlalchemy.event.registry', 'pickle', 'pandas.core.indexes.textwrap', 'chardet.sbcsgroupprober', 'dill.threading', 'pandas.io.mmap', 'luigi.task_register', 'numpy.lib.polynomial', 'docker.utils.six', 'pkg_resources._vendor.os', 'urllib3.util.collections', 'distutils.email', 'chardet.euctwfreq', 'xml.etree.ElementPath', 'dumbdbm', 'pandas._libs.tslibs.pandas', 'matplotlib.testing', 'cat.luigi', 'lib2to3.operator', 'luigi.threading', 'future.backports.email._policybase', 'sqlalchemy.engine.base', 'numpy.lib.utils', 'pandas._libs.tslibs.conversion', 'dateutil.relativedelta', 'toil.provisioners.aws.logging', 'numpy.lib.arrayterator', 'email.parser', 'luigi.configuration.os', 'pandas.core.collections', '_weakrefset', 'test.struct', 'docker.logging', 'unittest.traceback', 'unittest.os', 'pandas._libs.tslibs.period', 'urllib3.exceptions', 'email.MIMEImage', 'luigi.configuration.logging', 'pandas._libs.tslibs.timestamps', 'sqlalchemy.sql.default_comparator', 'pandas.core.groupby.pandas', 'urllib3.util.request', 'luigi.local_target', 'numpy.polynomial.legendre', 'builtins', 'platform', 'chardet.charsetprober', 'docker.utils.base64', 'pandas.io.os', 'unittest.functools', 'dill.ctypes', 'unittest.util', 'backports.ssl_match_hostname.re', 'urllib3.packages.six', 'pandas.core.dtypes.numpy', 'csv', 'pytz.collections', 'bisect', 'email.encoders', 'lib2to3.pgen2.driver', 'dateutil.warnings', 'docker.api.plugin', 'docker.api.websocket', 'distutils.dist', 'pandas.core.dtypes.concat', 'pydoc', 'pkg_resources._vendor.packaging.__about__', 'requests.packages.chardet.langthaimodel', 'luigi.uuid', 'luigi.contrib.sqlalchemy', 'lib2to3.patcomp', 'lib2to3.pgen2.string', 'requests.packages.urllib3.util.retry', 'pandas._libs.tslibs.resolution', 'docker.api.swarm', 'sqlalchemy.sql.util', 'pandas.compat.numpy.distutils', 'test.socket', 'pandas.core.indexes.multi', 'pandas.core.sys', 'posix', 'pkg_resources._vendor.packaging.markers', 'chardet.universaldetector', 'past.utils.sys', 'pandas.compat.numpy.pandas', 'dateutil.parser.string', 'numpy.random', 'luigi.warnings', 'array', 'requests.packages.urllib3.util.url', 'pkg_resources._vendor.sys', 'pytz.exceptions', 'luigi.getpass', 'types', '_codecs', 'past.builtins.inspect', 'pandas.io.formats.terminal', 'distutils.imp', 'copy', 'toil.lib.expando', 'pkg_resources._vendor.pyparsing', 'pandas.util.types', 'unittest.weakref', 'future', 'pkg_resources.sys', 'websocket.struct', 'chardet.escsm', 'pandas.core.groupby.groupby', 'chardet.utf8prober', 'cat.configobj', 'dockerpycreds.sys', 'pandas.tseries.offsets', 'numpy.lib.shape_base', 'numpy._import_tools', 'backports_abc', 'numpy.fft.info', 'docker.api.warnings', 'xml.etree.cElementTree', 'docker.types.base', 'unittest.result', 'future.backports.urllib.response', 'docker.version', 'toil.version', 'dateutil.parser.six', 'pandas.core.indexes.datetimelike', 'pandas.compat.struct', 'pandas.core.dtypes.dtypes', 'future_builtins', 'pandas.core.dtypes.common', 'sqlalchemy.util.ConfigParser', 'requests.packages.urllib3.util.selectors', 'dill.dill', 'linecache', 'requests.packages.chardet.compat', 'hmac', 'future.standard_library', 'urllib3.packages.thread', 'lib2to3.StringIO', 'subprocess', 'distutils.getopt', 'luigi.six', 'xml', 'logging.stat', 'pandas.core.ops', '_cython_0_28_4', 'dateutil.tz.tz', '_json', 'pandas.core.arrays.numpy', 'pyexpat.errors', 'requests.packages.urllib3.packages.ssl_match_hostname.sys', 'requests.packages.chardet.jisfreq', 'pkg_resources.extern.packaging.version', 'requests.packages.chardet.codecs', 'urllib3.packages.six.moves.urllib', 'pandas.util', 'zipfile', 'repr', 'requests.packages.urllib3', 'ssl', 'numpy.version', 'luigi.locale', 'resource', 'luigi.tempfile', 'chardet.compat', 'unittest.re', 'threading', 'dateutil.tz.bisect', 'email.Message', 'docker.api.secret', 'future.backports.misc', 'chardet.sys', 'docker.transport.distutils', 'pkg_resources._vendor.re', 'toil.lib.objects', 'future.backports.email._encoded_words', 'calendar', 'plistlib', 'pandas.tseries.numpy', 'requests.collections', '_MozillaCookieJar', 'requests.urllib', 'fcntl', 'unittest.case', 'cat.os', 'docker.api.service', 'distutils.os', 'chardet.escprober', 'websocket.base64', 'luigi.tools', 'requests.__version__', 'pandas.util._decorators', 'json.re', 'requests.time', 'requests.packages.urllib3.connection', 'pandas.core.missing', 'distutils.spawn', 'opcode', 'docker.utils.select', 'requests.warnings', 'frozendict.functools', 'unittest', 'docker.client', 'logging.errno', 'luigi.traceback', 'xml.dom.domreg', 'luigi.contrib', 'numpy.testing._private.utils', 'libfuturize.fixes.sys', 'sqlalchemy.sql.json', 'numpy.polynomial.laguerre', 'future.utils', 'sre_constants', 'numpy.core.function_base', '_random', 'certifi', 'urllib3.filepost', 'requests.packages.urllib3._collections', 'dateutil.parser.time', 'pandas.plotting._tools', 'logging.atexit', 'logging.cPickle', 'future.backports.urllib.parse', 'pandas.core.indexes.datetimes', 'luigi.subprocess', 'email.MIMENonMultipart', 'pandas.io.formats.__future__', 'sqlalchemy.util.urllib', 'websocket.logging', 'docker.models.networks', 'past.translation.sys', 'pandas.core.dtypes.generic', 'pkg_resources._vendor.packaging.specifiers', 'cat.random', 'requests.packages.urllib3.packages._abcoll', 'pandas.plotting.pandas', 'pandas.util.inspect', 'pyexpat', 'shutil', 'requests.packages.urllib3.util.sys', 'pandas._libs.tslibs.warnings', 'pandas.io.formats', 'frozendict.operator', 'pandas.tseries.re', 'past.builtins.noniterators', 'dbm.ndbm', 'chardet.euctwprober', 'pandas.core.tools.numpy', 'multiprocessing.subprocess', 'importlib.sys', 'requests.packages.chardet.chardistribution', 'matplotlib.fontconfig_pattern', 'dill.weakref', 'urllib.error', 'encodings.idna', 'idna.sys', 'pandas.core.dtypes.re', 'heapq', 'docker.utils.urllib', 'pandas._libs.tslibs.sys', 'requests.packages.chardet', 'requests.packages.urllib3.util.errno', 'struct', 'past.builtins.future', 'requests.packages.chardet.langhebrewmodel', 'email.Parser', 'distutils.types', 'future.backports.urllib.error', 'future.backports.urllib.request', 'pkg_resources.extern.packaging.markers', 'pandas._libs.tslibs.fields', 'pandas.core.dtypes.warnings', 'future.backports.datetime', 'cat.collections', 'future.backports', 'websocket._utils', 'numpy.matrixlib.defmatrix', 'email.Generator', 'docker.utils.warnings', 'requests.re', 'pandas.core.sparse.numpy', 'email.MIMEAudio', 'pandas.core.tools', 'email.errors', 'pandas.compat.sys', 'inspect', 'docker.api.six', 'pandas.core.index', 'email.Iterators', 'numpy.core.memmap', 'bsddb.db', 'cython_runtime', 'pandas.compat.re', 'past.translation.lib2to3', 'dateutil.math', 'luigi', 'os', 'backports.functools_lru_cache', 'numpy.core.shape_base', 'pandas.io.functools', 'requests.certs', 'pandas._libs.tslibs.offsets', 'operator', 'luigi.tools.logging', 'sqlalchemy.sql.naming', 'luigi.configuration.cfg_parser', 'pandas._libs.missing', 'urllib.robotparser', 'pandas.core.indexes.api', 'Cookie', '_warnings', 'pandas._libs.algos', 'sqlalchemy.sql.codecs', 'sqlalchemy.codecs', 'cat.toil', 'numpy._distributor_init', 'sqlalchemy.util.sys', 'pandas.core.groupby.collections', 'future.moves.dbm.dumb', 'sqlalchemy.sql.expression', 'dateutil.parser.dateutil', 'lib2to3.__future__', 'docker.api.exec_api', 'requests.packages.chardet.mbcharsetprober', 'logging.SocketServer', 'requests.struct', 'lib2to3.sys', 'dateutil.datetime', 'xml.dom.NodeFilter', 'requests.packages.chardet.escprober', 'docker.utils.utils', 'requests.threading', 'logging.types', 'docker.types.networks', 'test.errno', 'numpy.core.defchararray', 'gettext', 'pandas.core.arrays.warnings', '_abcoll', 'future.moves.test', 'requests.packages.urllib3.request', 'mimetypes', 'websocket.six', 'sqlalchemy.sql.__future__', 'docker.models.services', 'luigi.six.moves.urllib.error', 'unittest.signals', 'termios', 'dill.temp', 'lib2to3.pgen2.sys', 'numpy.testing._private.decorators', 'pandas.core.groupby.textwrap', 'encodings.stringprep', 'lib2to3.refactor', 'pandas.core.arrays.categorical', 'sqlalchemy.engine.reflection', 'pandas.core.indexes.numpy', 'gdbm', 'dockerpycreds.os', 'toil.lib.random', 'pandas._libs.tslibs.parsing', 'sqlalchemy.sql.decimal', 'past.translation.marshal', 'numpy.core.umath', 'luigi.luigi', 'luigi.contrib.datetime', 'chardet.euckrprober', 'encodings.utf_8', 'dockerpycreds.distutils', 'requests.packages.urllib3.util.request', 'json.encoder', 'pandas.plotting.re', 'future.backports.email', 'unittest.StringIO', 'future.types.newobject', 'numpy.fft.fftpack_lite', 'sqlalchemy.sqlalchemy', 'dockerpycreds.six', 'sqlalchemy.schema', 'past', 'abc', 'urllib3.packages', 'docker.utils.socket', 'past.utils.numbers', 'optparse', 'glob', 'xml.etree', 'websocket._core', 'sqlalchemy.engine.default', 'docker.utils.io', 'multiprocessing.multiprocessing', 'StringIO', 'pandas.core.sparse', 'requests.packages.urllib3.util.time', 'pandas.core.codecs', 'requests.platform', 'pandas.core.frame', 'sqlalchemy.engine.url', 'pandas.compat.numpy.numpy', 'bsddb.thread', 'select', 'pandas._libs.interval', 'numpy.lib.financial', 'luigi.json', 'pkg_resources._vendor.six.moves.urllib', 'requests.structures', 'docker.models.images', 'numpy.polynomial.hermite_e', 'pandas.compat.collections', 'future.moves.dbm.ndbm', 'pandas._libs.tslibs.timezones', 'toil.provisioners.aws.os', 'pandas.core.indexes.operator', 'pandas._libs.operator', 'unicodedata', 'sqlalchemy.util.collections', 'pkg_resources.os', 'requests.packages.idna.bisect', 'encodings.aliases', 'luigi.contrib.sqla', 'pandas.core.common', 'pandas.tseries', 'numpy.random.warnings', 'pandas._libs.tslibs.ccalendar', 'past.past', 'pandas.io.formats.functools', 'numbers', 'distutils.opcode', 'requests.exceptions', 'pandas.core.keyword', 'future.builtins.iterators', 'numpy.core.numeric', 'luigi.os', 'docker.models.docker', 'pandas.core.inspect', 'pandas.core.dtypes.collections', 'past.builtins.collections', 'dill.settings', 'argparse', 'pandas.plotting.numpy', 'urllib3.util.select', 'bsddb.weakref', 'future.moves.dbm.gnu', 'urllib3.connectionpool', 'docker.api.base64', 'sqlalchemy.sql.functions', 'toil.provisioners.aws.collections', 'dill.pickle', 'pytz.lazy', 'chardet.cp949prober', 'luigi.tools.functools', 'requests.urllib2', 'requests.urllib3', 'luigi.inspect', 'idna.unicodedata', 'bsddb._bsddb', 'xml.dom.copy', 'numpy.linalg.info', 'requests.packages.chardet.version', 'pkg_resources._vendor.six', 'matplotlib.rcsetup', 'past.translation.logging', 'pandas._libs.tslibs.time', 'email.urllib', 'pandas.core.indexes.accessors', 'httplib', 'urllib3.util.time', 'luigi.logging', 'requests.packages.idna.intranges', 'luigi.six.moves.urllib.parse', 'decimal', 'websocket._handshake', 'token', 'backports.ssl_match_hostname', 'pkg_resources.extern', 'dill.thread', 'six.moves.urllib.parse', 'multiprocessing.process', 'distutils.warnings', 'future.types.struct', 'websocket.socket', 'future.backports.http.client', 'matplotlib.cbook', 'requests.packages.chardet.jpcntx', 'ast', 'dateutil.tz.six', 'sqlalchemy.sql.ddl', 'chardet.langhebrewmodel', 'ctypes.struct', 'requests.packages.chardet.codingstatemachine', 'pandas.core.strings', 'pandas.core.distutils', 'thread', 'pandas.compat.functools', 'websocket.time', 'future.backports.email.feedparser', 'multiprocessing.util', '_collections', 'sqlalchemy.engine.operator', 'pandas.compat.httplib', 'future.builtins', 'pandas.core.tools.collections', 'email.Errors', 'ctypes.sys', 'pandas.tseries.dateutil', 'posixpath', 'pandas.compat.pandas', 'ipaddress', 'requests.packages.urllib3.packages.ssl_match_hostname.backports', 'requests.json', 'json._json', 'pandas.tseries.datetime', 'json.sys', 'future.builtins.__builtin__', 'requests.packages.chardet.sjisprober', 'dill.imp', 'hashlib', 'luigi.task', 'pandas.compat.distutils', 'numpy.lib.nanfunctions', 'requests.cgi', 'requests.models', 'pandas.core.arrays.textwrap', 'luigi.event', 'idna.re', 'sre_compile', 'pkg_resources._vendor.threading', 'dateutil.tz._factories', 'pandas.core.dtypes.datetime', 'websocket._logging', 'email.base64', 'toml', 'docker.base64', 'future.utils.types', 'multiprocessing.atexit', 'dateutil._version', 'lib2to3', 'requests.packages.chardet.sys', 'sqlalchemy.util.re', 'xml.dom.minicompat', 'pandas.plotting._misc', 'pandas._libs.decimal', 'warnings', 'encodings.ascii', 'urllib3.packages.six.moves', 'past.translation.libfuturize', 'pandas.core.algorithms', 'multiprocessing._multiprocessing', 'dateutil.parser.calendar', '_virtualenv_distutils', 'pkg_resources.extern.six.moves', 'frozendict.collections', 'numpy.linalg.linalg', 'future.backports.email.message', 'urllib.parse', 'bsddb.os', 'ctypes._endian', 'bsddb.bsddb', 'unittest.pprint', 'lib2to3.btm_matcher', 'pandas.core.util.itertools', 'six.moves.urllib.request', 'numpy.linalg', 'luigi.task_history', 'pandas.core.re', 'requests.utils', 'toil.batchSystems.options', 'docker.api.container', 'future.backports.urllib.robotparser', 'sqlalchemy.util.types', 'pandas.util.warnings', 'docker.models.volumes', 'cat.json', 'luigi.socket', 'toil.lib.processes', 'requests.hashlib', 'pandas.io.formats.os', 'pytz.tzfile', 'locale', 'test.contextlib', 'docker.types', 'email.quopriMIME', 'matplotlib._color_data', 'pandas.core.arrays.base', 'docker.utils.shlex', 'dateutil', 'dill.logging', 'sqlalchemy.util.queue', 'requests.StringIO', 'pkg_resources._vendor.string', 'chardet.eucjpprober', 'luigi.textwrap', 'chardet.chardistribution', 'docker.transport.httplib', 'test.sys', 'pandas.compat.chainmap_impl', 'pandas.compat', 'urllib3.util.response', 'numpy.fft.fftpack', 'pandas.core.dtypes.cast', 'urllib3.util.sys', 'future.types.newrange', 'past.types', 'test.time', 'numpy.testing._private.nosetester', 'pandas.compat.inspect', 'past.utils', 'websocket._ssl_compat', 'unittest.collections', 'luigi.tools.operator', 'logging.struct', 'logging.config', 'pkg_resources._vendor.sre_constants', 'sqlalchemy.sql.contextlib', 'requests.packages.urllib3.connectionpool', 'lib2to3.pgen2.grammar', 'pandas.compat.numpy', 'idna.bisect', 'chardet.codingstatemachine', 'matplotlib.cbook._backports', 'future.builtins.newsuper', 'requests.packages.urllib3.util.ssl_', 'luigi.configuration.toml_parser', 'pandas.core.util.hashing', 'requests.certifi', 'pandas.io.formats.pandas', 'luigi.contextlib', 'pytz.bisect', 'requests.packages.chardet.euctwfreq', 'copy_reg', 'dill.__builtin__', 'luigi.configuration.warnings', 'requests.encodings', 'email.MIMEBase', 'dateutil.parser.datetime', 'toil.lib.iterables', 'encodings.hex_codec', 'requests.packages.urllib3.util.response', 'dbm.dumb', 'docker.api.logging', 'sqlalchemy.util.StringIO', 'json.json', 'email.re', 'chardet.enums', 'sys', 'pandas.core.indexes.timedeltas', 'luigi.configuration.base_parser', 'requests.hooks', 'xml.dom.xml', 'pandas.core.indexes.base', 'luigi.ConfigParser', 'bsddb.collections', 'sqlalchemy.util.warnings', 'toil', 'docker', 'docker.types.healthcheck', 'distutils', 'past.future', 'requests._internal_utils', 'email.mime', 'luigi.tools.datetime', 'sqlalchemy.util.langhelpers', 'requests.io', 'pandas.core.groupby.warnings', 'pandas.core.weakref', 'future.builtins.io', 'websocket.inspect', 'docker.auth', 'lib2to3.pgen2.pickle', 'logging.socket', 'past.translation', 'toil.provisioners.aws.datetime', 'docker.types.six', 'pandas.plotting._compat', 'sqlalchemy.datetime', 'luigi.requests', 'docker.utils.functools', 'lib2to3.os', 'future.backports.email.utils', 'signal', 'docker.models.containers', 'chardet.langthaimodel', 'distutils.version', 'future.backports.email._parseaddr', 'requests.packages.urllib3.contrib', 'dateutil.tz.sys', 'luigi.interface', 'requests.packages.chardet.gb2312prober', 'chardet.gb2312prober', 'requests.packages.chardet.euckrfreq', 'sqlalchemy.engine.collections', 'test.platform', 'luigi.contrib.collections', 'pandas.core.warnings', 'luigi.collections', 'dateutil.tz.functools', 'pandas._libs', 'urllib3.util.socket', 'cgi', 'sqlalchemy.event.base', 'cat.frozendict', 'luigi.six.moves.urllib.request', 'pandas.plotting.collections', 'dateutil.calendar', 'unittest.loader', 'dill.cStringIO', 'sqlalchemy.types', 'dill.marshal', 'dill.pointers', '__future__', 'pkg_resources.extern.packaging.requirements', 'pandas.core.internals', 'distutils.string', 'docker.api.re', 'pandas.plotting.datetime', 'errno', 'pkg_resources._vendor.packaging.requirements', 'pandas.compat.dateutil', 'urllib3.request', 'urllib3.response', 'docker.utils.build', 'urllib3.util', 'numpy.lib.histograms', 'luigi.errno', 'pandas.util.textwrap', 'encodings.__builtin__', 'email.os', 'bsddb.warnings', 'requests.compat', 'pwd', 'docker.errors', 'bsddb.time', '_struct', 'uu', 'requests.packages', 'xml.dom.minidom', 'luigi.multiprocessing', 'docker.types.containers', 'matplotlib.compat.subprocess', 'docker.utils.json_stream', 'configobj', 'dill', 'multiprocessing.signal', 'pandas.compat.itertools', 'logging.codecs', 'numpy.compat.py3k', 'numpy.polynomial._polybase', 'dockerpycreds.subprocess', 'future.utils.inspect', 'future.utils.future', 'future.builtins.decimal', 'pandas.core.sorting', 'urllib3.packages.six.moves.urllib.parse', 'pandas.core.groupby.datetime', 'grp', 'dateutil.parser.decimal', 'requests.packages.urllib3.util.timeout', 'lib2to3.pytree', 'email.warnings', 'docker.models.re', 'pandas', 'requests.packages.urllib3.poolmanager', 'dateutil.parser.functools', 'pandas.core.config', 'past.builtins.misc', 'sqlalchemy.util', 'pandas.io.contextlib', 'docker.api.network', 'dill.functools', 'sqlalchemy.event.attr', 'numpy.lib.format', 'requests.calendar', 'requests.packages.idna.package_data', 'frozendict', 'pandas.core.indexes.pytz', 'past.builtins.sys', 'luigi.notifications', 'dill.gc', 'cat.shutil', 'backports.ssl_match_hostname.ipaddress', 'future.utils.__builtin__', 'distutils.debug', 'docker.types.swarm', 'ctypes.os', 'gc', 'pandas._libs.tslibs.locale', 'test.functools', 'pandas.plotting.math', 'base64', 'lib2to3.pgen2.StringIO', 'requests.packages.idna.re', 'future.backports.http', 'email.Header', 'dill._pyio', 'dateutil.parser._parser', 'numpy.lib.arraysetops', 'xml.etree.warnings', 'toil.batchSystems.registry', 'requests.packages.idna.sys', 'requests.packages.chardet.charsetgroupprober', 'docker.transport.backports', 'email.codecs', 'certifi.core', 'luigi.lock', 'logging.threading', 'logging.os', 'idna.intranges', 'luigi.datetime', 'luigi.cmdline_parser', 'encodings.encodings', 'websocket._app', 'matplotlib.cbook.textwrap', 'luigi.parameter', 'websocket.os', '_thread', 'pkg_resources._vendor.collections', 'dateutil.parser.warnings', 'numpy._globals', 'matplotlib.sys', 'luigi.batch_notifier', 'pandas._libs.tslibs.dateutil', 'requests.packages.urllib3.exceptions', 'future.moves.dbm', 'chardet.big5prober', 'ctypes.tempfile', 'luigi.shutil', 'unittest.suite', 'toil.test', 'matplotlib', 'json.scanner', 'luigi.retcodes', 'pandas.compat.platform', 'numpy.os', 'urllib3.util.timeout', 'luigi.scheduler', 'pandas.core.textwrap', 'pkg_resources', 'luigi.copy', 'pandas._libs.tslibs.StringIO', 'pandas._libs.tslibs.pytz', '_sre', 'sqlalchemy.util.itertools', 'pandas.core.numpy', 'luigi.contrib.logging', 'logging.sys', 'sqlalchemy.util.deprecations', 'docker.utils.json', 'requests.api', 'binascii', 'requests.cookies', 'unittest.fnmatch', 'numpy.polynomial.chebyshev', 'pkg_resources.extern.six.moves.urllib', 'cat.string', 'urllib3.util.wait', 'sqlalchemy.interfaces', 'matplotlib._version', 'pandas.pandas', 'sre_parse', 'future.types.__future__', 'docker.api.daemon', 'sqlalchemy.sql.sqlalchemy', 'luigi.tools.warnings', 'pandas._libs.sparse', 'lib2to3.fixer_util', 'pkg_resources.extern.six', 'requests.packages.chardet.big5prober', 'docker.api.datetime', 'six', 'pandas._libs.tslibs.datetime', 'frozendict.sys', 'pandas._libs.index', 'pandas.core.copy', 'sqlalchemy.engine.threadlocal', 'requests.packages.chardet.langturkishmodel', 'sqlalchemy.engine.strategies', 'docker.dockerpycreds', 'ctypes.util', 'luigi.contrib.itertools', 'dateutil._common', 'email.Charset', 'requests.packages.chardet.latin1prober', 'docker.utils.requests', 'multiprocessing.sys', 'pkg_resources._vendor.__builtin__', 'pkg_resources.extern.packaging.specifiers', 'requests.base64', 'sqlalchemy.sql.datetime', 'luigi.format', 'lib2to3.pgen2.codecs', 'sqlalchemy.sql.compiler', 'pandas.core.util', 'docker.utils.distutils', 'sysconfig', 'whichdb', 'docker.os', 'email.MIMEText', 'dateutil.tz.time', 'distutils.dep_util', 'sqlalchemy.inspection', 'tempfile', 'sqlalchemy.util.contextlib', 'pkg_resources._vendor', 'multiprocessing.itertools', 'pandas.io.formats.printing', 'requests.adapters', 'mmap', 'pandas.core.sparse.warnings', 'future.types.numbers', 'chardet.sjisprober', 'requests.packages.urllib3.util.socket', 'future.utils.surrogateescape', 'dockerpycreds.json', 'pandas._libs.lib', 'docker.utils.decorators', 'logging.logging', 'past.translation.copy', 'test.gc', 'numpy.lib._datasource', 'toil.common', 'logging.handlers', 'dateutil.tz.datetime', 'sqlalchemy.util.urlparse', 'dateutil.operator', 'requests.packages.urllib3.packages.ordered_dict', 'numpy.linalg._umath_linalg', 'cStringIO', 'numpy.polynomial', 'requests.packages.chardet.mbcssm', 'requests.datetime', 'luigi.signal', 'sqlalchemy.time', 'encodings', 'sqlalchemy.logging', 'requests.packages.idna.unicodedata', 'numpy.lib.numpy', 'pandas.core.dtypes.missing', 'future.types', 'UserString', 'math', 'numpy.lib.ufunclike', 'docker.api.build', 'dateutil.parser.isoparser', 'docker.requests', 'chardet.jpcntx', 'toil.job', 'pandas.plotting._style', 'luigi.execution_summary', 'chardet.latin1prober', 'requests.copy', 'sqlalchemy.inspect', 'requests.packages.chardet.enums', 'pandas.util.functools', 'numpy.lib.twodim_base', 'pandas._libs.numbers', 'distutils.util', '_LWPCookieJar', 'toil.lib.bioio', 'libfuturize', 'requests.packages.urllib3.packages', 'numpy.lib.stride_tricks', 'numpy.lib.scimath', 'matplotlib.cbook.functools', 'enum.collections', 'email.feedparser', 'luigi.__future__', 'requests.codecs', 'sqlalchemy.sql.schema', 'pandas.tseries.pytz', 'lib2to3.pgen2.parse', 'docker.utils', 'sqlalchemy.sql.annotation', 'docker.api.client', '_csv', 'dockerpycreds.errors', 'luigi.enum', 'chardet', 'test.shutil', 'urllib3.packages.ordered_dict', 'sqlalchemy.util.inspect', '_hashlib', 'luigi.six.moves.urllib', 'sqlalchemy.events', 'pandas.core.groupby.contextlib', 'pandas.core.indexes.period', 'pytz.struct', 'past.types.__builtin__', 'luigi.tools.luigi', 'sqlalchemy.util._collections', 'future.utils.numbers', 'dateutil.parser', 'pandas._libs.tslibs.np_datetime', 'encodings.codecs', 'docker.models.swarm', 'sqlalchemy.sql.numbers', 'pandas.core.util.pandas', 'email.string', '_ssl', 'sqlalchemy.engine.random', 'bsddb.sys', 'pandas.core.indexes.datetime', '_io', 'future.types.newstr', 'pandas.plotting.__future__', 'dockerpycreds', 'pkg_resources._vendor.traceback', 'sqlalchemy.sql.visitors', 'sqlalchemy.engine.codecs', 'unittest.types', 'pyparsing', 'numpy.random.mtrand', 'luigi.functools', 'importlib', 'requests.sessions', 'urllib3.connection', 'pandas.compat.numpy.re', 'sqlalchemy.util.functools', 'docker.utils.re', 'urllib3.util.selectors', 'distutils.re', 'pandas.core.base', 'six.moves.queue', 'cookielib', 'websocket.errno', 'urllib3.packages.ssl_match_hostname.sys', 'pandas._libs.tslibs.frequencies', 'cycler', 'docker.tls', 'pandas.core.dtypes.base', 'docker.sys', 'chardet.euckrfreq', 'chardet.langcyrillicmodel', 'pandas.core.generic', 'toil.lib.math', 'requests.auth', 'luigi.re', 'requests.packages.idna', 'docker.types.services', 'requests.packages.urllib3.util.select', 'future.builtins.future', 'numpy.core.numerictypes', 'pandas.core.arrays', 'urllib3.util.connection', 'email', 'chardet.mbcsgroupprober', 'toil.provisioners.aws.operator', 'toil.lib.humanize', 'backports.ssl_match_hostname.sys', 'sqlalchemy.traceback', 'certifi.os', 'dill.sys', 'commands', 'sqlalchemy.sql.sqltypes', 'chardet.langbulgarianmodel', 'pandas.core.indexing', 'itertools', 'future.utils.copy', 'pandas._libs.ops', 'pandas.compat.numpy.function', 'pandas.core.tools.datetime', 'email.base64MIME', 'pandas.tseries.operator', 'future.backports.email.errors', 'past.builtins', 'json', 'lib2to3.pgen2.tokenize', 'past.types.past', 'toil.lib.memoize', 'pandas.core', 'email.utils', 'numpy.ma', 'future.backports.email.quoprimime', 'pandas.core.indexes.range', 'pandas.core.dtypes', 'email.email', 'urllib3.util.url', 'xml.etree.re', 'numpy.lib', 'sqlalchemy.exc', 'multiprocessing.weakref', 'pandas.core.indexes.category', 'distutils.distutils', 'chardet.sbcharsetprober', 'urllib3.packages.six.moves.http_client', 'site', 'sqlalchemy.event.legacy', 'sqlalchemy.event.api', 'chardet.gb2312freq', 'test.unittest', 'email.binascii', 'future.builtins.newround', 'pandas.io.pandas', 'sqlalchemy.util.textwrap', 'numpy.polynomial.polyutils', 'future.backports.email.charset', 'xml.etree._elementtree', 'luigi.configuration.core', 'numpy.compat._inspect', 'docker.transport.io', 'docker.models.six', 'pandas.core.indexes.warnings', 'email.cStringIO', 'sqlalchemy.util.compat', 'pandas.core.itertools', 'pandas.core.__future__', 'difflib', 'distutils.errors', 'docker.json', 'pandas.compat.thread', 'unittest.warnings', 'sqlalchemy.util.sqlalchemy', 'pandas.plotting.distutils', 'pytz.threading', 'sqlalchemy.sql.operator', 'numpy.core.einsumfunc', 'pkg_resources.extern.packaging', 'matplotlib.cbook.deprecation', 'sqlalchemy.event', 'websocket.Cookie', 'numpy.testing', 'urllib3.util.errno', 'pytz.os', 'dill.info', 'docker.api.requests', 'requests.sys', 'pkg_resources.extern.sys', 'sqlalchemy.engine.__future__', 'requests.packages.chardet.utf8prober', 'chardet.charsetgroupprober', 'pandas.core.datetime', 'luigi.hashlib', 'requests.packages.urllib3.response', 'luigi.util', 'luigi.contrib.os', 'distutils.log', 'numpy.random.operator', 'pandas.compat.__builtin__', 'pandas.io.csv', 'requests.packages.chardet.cp949prober', 'numpy.core.multiarray', 'numpy.ma.core', 'sqlalchemy.sql.collections', 'pandas._libs.tslibs.collections', 'sqlalchemy.sql.elements', 'sqlalchemy.dialects', 'pandas.io.formats.shutil', 'lib2to3.btm_utils', 'pandas.io.formats.numpy', 'numpy.matrixlib', 'sqlalchemy.sql.itertools', '_multiprocessing', 'pandas.core.dtypes.inference', 'backports', 'requests.packages.chardet.mbcsgroupprober', 'dateutil.parser.re', 'idna.core', 'requests.packages.chardet.big5freq', 'enum', 'unittest.runner', 'future.builtins.newnext', '_functools', 'docker.api.functools', 'socket', 'chardet.langgreekmodel', 'docker.models.nodes', 'lib2to3.pgen2.token', 'luigi.contrib.abc', '__builtin__', 'ctypes.errno', 'json.struct', 'sqlalchemy.cprocessors', 'pandas.core.json', 'future.moves', 'multiprocessing.threading', 'future.types.sys', 'pytz.sys', 'distutils.stat', 'urllib3._collections', 'future.builtins.misc', 'websocket.traceback', 'pandas.io.urllib2', 'pandas.plotting._core', 'lib2to3.pgen2.literals', 'logging.time', 'luigi.tools.time', 'luigi.Queue', 'numpy.lib.function_base', 'logging.warnings', 'dockerpycreds.constants', 'docker.transport.sys', 'lib2to3.pgen2.lib2to3', 'future.backports.urllib', 'luigi.importlib', 'numpy.polynomial.hermite', 'contextlib', 'requests.packages.urllib3.packages.six.moves.urllib', 'dill.detect', 'chardet.jisfreq', 'docker.transport.functools', 'bsddb', 'getopt', 'dockerpycreds.utils', 'stat', 'email.Encoders', 'xml.dom', 'future.utils.functools', 'toil.lib.six', 'pandas._libs.sys', 'encodings.unicodedata', 'pandas._libs.tslibs.numpy', 'pkg_resources._vendor.types', 'dill.__main__']
DEBUG:matplotlib:CACHEDIR=/home/fc464/.cache/matplotlib
DEBUG:matplotlib.font_manager:Using fontManager instance from /home/fc464/.cache/matplotlib/fontList.json
DEBUG:matplotlib.backends:backend agg version v2.2
INFO:luigi-interface:Loaded ['luigi.cfg']
09-20 13:51:18 luigi-interface INFO  Informed scheduler that task   RunCat_False_True_True_6347a3a45f   has status   PENDING
09-20 13:51:18 luigi-interface INFO  Informed scheduler that task   ReportStats_False_True_True_6347a3a45f   has status   PENDING
which: no wiggletools in (/home/fc464/software/luigi/luigi_env/bin:/usr/local/software/spack/spack-0.11.2/opt/spack/linux-rhel7-x86_64/gcc-5.4.0/ncurses-6.0-qe3zsbdgaja5toda6pdkjkgxhke4dgnl/bin:/usr/local/Cluster-Apps/gcc/5.2.0/bin:/usr/local/Cluster-Apps/intel/2017.4/debugger_2017/gdb/intel64_mic/bin:/usr/local/Cluster-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/mpi/intel64/bin:/usr/local/Cluster-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/bin/intel64:/usr/local/software/global/bin:/opt/singularity/bin:/usr/local/Cluster-Apps/singularity/images:/usr/local/Cluster-Apps/vgl/2.5.1/64/bin:/usr/local/Cluster-Apps/turbovnc/2.0.1/bin:/usr/local/software/slurm/current/sbin:/usr/local/software/slurm/current/bin:/usr/lib64/qt-3.3/bin:/home/fc464/perl5/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/dell/srvadmin/bin:.:/home/fc464/software//cactus/bin:/home/fc464/software//htop/bin:/home/fc464/software//canu/Linux-amd64/bin:/home/fc464/software//samtools-1.8/bin:/home/fc464/software//cd-hit-v4.6.8-2017-1208:/home/fc464/software//trinityrnaseq-Trinity-v2.8.1/trinity-plugins/BIN:/home/fc464/software//bowtie2-2.3.4.2-linux-x86_64:/home/fc464/software//jellyfish-2.2.10/bin:/home/fc464/software//maker/bin:/home/fc464/software//DAMA2:/home/fc464/software//bbmap:/home/fc464/software//gm_et_linux_64/gmes_petap:/home/fc464/software//BRAKER:/home/fc464/software//sratoolkit.2.9.1-1-centos_linux64/bin:/home/fc464/software//ncbi-blast-2.7.1+/bin:/home/fc464/software//mcl-14-137/bin:/home/fc464/software//hmmer-3.2.1/bin:/home/fc464/software//RepeatMasker:/home/fc464/software//samtools-1.8/bin:/home/fc464/software//discovardenovo/bin:/bin:/home/fc464/software//jemalloc/bin:/home/fc464/software//multi-mapping-counter:/home/fc464/software//subread-1.6.2-source/bin:/home/fc464/software//HTSeq-count/scripts:/home/fc464/software//stringtie:/home/fc464/software//ebab.lab_scripts:/home/fc464/software//mmseq/bin:/home/fc464/software//bamtools/bin:/home/fc464/software//cufflinks-2.2.1.Linux_x86_64:/home/fc464/software//exonerate-2.2.0/bin:/home/fc464/software//hmmer-3.1b2-linux-intel-x86_64/binaries:/home/fc464/software//OrthoFinder-2.2.3:APPS/FRAMA:/home/fc464/software//STAR/bin/Linux_x86_64:/home/fc464/software//HaploMerger2_20161205/bin:/home/fc464/software//MUMmer3.23:/home/fc464/software//bin:/home/fc464/software//bioscript:/rds/user/fc464/hpc-work/SoftwareTwo/blobtools:/home/fc464/software//busco/scripts:/home/fc464/software//Augustus/bin:/home/fc464/software//freebayes/bin:/home/fc464/software//SPAdes-3.12.0-Linux/bin:/home/fc464/software//lastz-distrib-1.04.00/bin:/home/fc464/software//GenomeAlignmentTools/precompiledBinary_x86_64:/home/fc464/software//RepeatMasker:/home/fc464/software//purge_haplotigs/bin:/home/fc464/software//bedtools2/bin:/rds/user/fc464/hpc-work/SoftwareTwo/kentUtils/bin:/home/fc464/software//Augustus/scripts:/home/fc464/software//Augustus/auxprogs/joingenes:/home/fc464/.local/bin:/home/fc464/software//luigi/bin:/home/fc464/software//HaploMerger2_20180603/bin)
09-20 13:51:18 luigi-interface INFO  Informed scheduler that task   AssemblyHub_False_True_True_6347a3a45f   has status   PENDING
09-20 13:51:18 luigi-interface WARNING Will not run Task: CreateTracks or any dependencies due to error in complete() method:
Traceback (most recent call last):
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 401, in check_complete
    is_complete = task.complete()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/task.py", line 827, in complete
    return all(r.complete() for r in flatten(self.requires()))
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/task.py", line 888, in flatten
    for result in iterator:
  File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/cat/__init__.py", line 2276, in requires
    self.validate()
  File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/cat/__init__.py", line 2273, in validate
    raise ToolMissingException('Tool {} not in global path.'.format(tool))
ToolMissingException: Tool wiggletools not in global path.

09-20 13:51:18 luigi-interface INFO  Informed scheduler that task   CreateTracks_False_True_True_6347a3a45f   has status   UNKNOWN
09-20 13:51:18 luigi-interface INFO  Informed scheduler that task   CreateDirectoryStructure_False_True_True_6347a3a45f   has status   PENDING
09-20 13:51:18 luigi-interface INFO  Informed scheduler that task   ReferenceFiles_False_True_True_6347a3a45f   has status   PENDING
09-20 13:51:18 luigi-interface INFO  Informed scheduler that task   FakePsl_False__home_fc464_soft__home_fc464_soft_f9fe6dd165   has status   PENDING
09-20 13:51:18 luigi-interface INFO  Informed scheduler that task   GenomeSizes_False_True_True_0106dc1bdd   has status   DONE
09-20 13:51:18 luigi-interface INFO  Informed scheduler that task   FlatTranscriptFasta_False__home_fc464_soft__home_fc464_soft_d6896d27b1   has status   PENDING
09-20 13:51:18 luigi-interface INFO  Informed scheduler that task   TranscriptGtf_False__home_fc464_soft__home_fc464_soft_55b66d18b4   has status   PENDING
09-20 13:51:18 luigi-interface INFO  Informed scheduler that task   TranscriptFasta_False__home_fc464_soft__home_fc464_soft_2394487939   has status   PENDING
09-20 13:51:18 luigi-interface INFO  Informed scheduler that task   GenomeFlatFasta_False_True_True_592d3347d7   has status   DONE
09-20 13:51:18 luigi-interface INFO  Informed scheduler that task   TranscriptBed_False__home_fc464_soft__home_fc464_soft_d1a04c3599   has status   PENDING
09-20 13:51:18 luigi-interface INFO  Informed scheduler that task   Gff3ToAttrs_False__home_fc464_soft__home_fc464_soft_ec7fb656b4   has status   PENDING
09-20 13:51:18 luigi-interface INFO  Informed scheduler that task   Gff3ToGenePred_False__home_fc464_soft__home_fc464_soft_ec7fb656b4   has status   PENDING
09-20 13:51:18 luigi-interface INFO  Informed scheduler that task   Plots_False_True_True_6347a3a45f   has status   PENDING
09-20 13:51:18 luigi-interface INFO  Informed scheduler that task   Consensus_False_True_True_6347a3a45f   has status   PENDING
09-20 13:51:18 luigi-interface INFO  Informed scheduler that task   ConsensusDriverTask_False_True_True_fa4dafb5a0   has status   PENDING
09-20 13:51:18 luigi-interface INFO  Informed scheduler that task   ConsensusDriverTask_False_True_True_fe71dc0265   has status   PENDING
09-20 13:51:18 luigi-interface INFO  Informed scheduler that task   ConsensusDriverTask_False_True_True_fa494a5425   has status   PENDING
09-20 13:51:18 luigi-interface INFO  Informed scheduler that task   ConsensusDriverTask_False_True_True_223c2e21ea   has status   PENDING
09-20 13:51:18 luigi-interface INFO  Informed scheduler that task   ConsensusDriverTask_False_True_True_2bbbfdbecd   has status   PENDING
09-20 13:51:18 luigi-interface INFO  Informed scheduler that task   ConsensusDriverTask_False_True_True_975c017f3d   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   ConsensusDriverTask_False_True_True_2d38edffaf   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   EvaluateTranscripts_False_True_True_6347a3a45f   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_fa4dafb5a0   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_fe71dc0265   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_fa494a5425   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_223c2e21ea   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_2bbbfdbecd   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_975c017f3d   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_2d38edffaf   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   AlignTranscripts_False_True_True_6347a3a45f   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   AlignTranscriptDriverTask_False_True_True_fa4dafb5a0   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   GenomeFiles_False_True_True_6347a3a45f   has status   DONE
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   AlignTranscriptDriverTask_False_True_True_fe71dc0265   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   AlignTranscriptDriverTask_False_True_True_fa494a5425   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   AlignTranscriptDriverTask_False_True_True_223c2e21ea   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   AlignTranscriptDriverTask_False_True_True_2bbbfdbecd   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   AlignTranscriptDriverTask_False_True_True_975c017f3d   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   AlignTranscriptDriverTask_False_True_True_2d38edffaf   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   Hgm_False_True_True_6347a3a45f   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   HgmDriverTask_False_True_True_b7ad204508   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   HgmDriverTask_False_True_True_fb487d60bd   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   HgmDriverTask_False_True_True_05c547cfdb   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   HgmDriverTask_False_True_True_f20b277ebc   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   HgmDriverTask_False_True_True_c3ff09aa2b   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   IsoSeqTranscripts_False_True_True_6347a3a45f   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   IsoSeqTranscriptsDriverTask_False_True_True_975c017f3d   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   FindDenovoParents_False_True_True_b7ad204508   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   AugustusPb_False_True_True_6347a3a45f   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   AugustusPbDriverTask_False_True_True_975c017f3d   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   FindDenovoParents_False_True_True_f20b277ebc   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   AugustusCgp_False_True_True_6347a3a45f   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   Augustus_False_True_True_6347a3a45f   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   AugustusDriverTask_False_True_True_fa4dafb5a0   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   AugustusDriverTask_False_True_True_fe71dc0265   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   AugustusDriverTask_False_True_True_fa494a5425   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   AugustusDriverTask_False_True_True_223c2e21ea   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   AugustusDriverTask_False_True_True_2bbbfdbecd   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   AugustusDriverTask_False_True_True_975c017f3d   has status   PENDING
09-20 13:51:19 luigi-interface INFO  Informed scheduler that task   AugustusDriverTask_False_True_True_2d38edffaf   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   EvaluateTransMap_False_True_True_6347a3a45f   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   EvaluateTransMapDriverTask_False_True_True_92c95406d4   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   EvaluateTransMapDriverTask_False_True_True_1492f1ae3e   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   EvaluateTransMapDriverTask_False_True_True_e6cf88c9f1   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   EvaluateTransMapDriverTask_False_True_True_39372c3cc4   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   EvaluateTransMapDriverTask_False_True_True_b94f62c722   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   EvaluateTransMapDriverTask_False_True_True_43b2021ad7   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   EvaluateTransMapDriverTask_False_True_True_db97f4ab71   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   TransMap_False_True_True_6347a3a45f   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   TransMapGtf_False_True_True_fa4dafb5a0   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   FilterTransMap_False_True_True_fa4dafb5a0   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   TransMapPsl_False_True_True_fa4dafb5a0   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   TransMapGtf_False_True_True_fe71dc0265   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   FilterTransMap_False_True_True_fe71dc0265   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   TransMapPsl_False_True_True_fe71dc0265   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   TransMapGtf_False_True_True_fa494a5425   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   FilterTransMap_False_True_True_fa494a5425   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   TransMapPsl_False_True_True_fa494a5425   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   TransMapGtf_False_True_True_223c2e21ea   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   FilterTransMap_False_True_True_223c2e21ea   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   TransMapPsl_False_True_True_223c2e21ea   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   TransMapGtf_False_True_True_2bbbfdbecd   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   FilterTransMap_False_True_True_2bbbfdbecd   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   TransMapPsl_False_True_True_2bbbfdbecd   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   TransMapGtf_False_True_True_975c017f3d   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   FilterTransMap_False_True_True_975c017f3d   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   TransMapPsl_False_True_True_975c017f3d   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   TransMapGtf_False_True_True_2d38edffaf   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   FilterTransMap_False_True_True_2d38edffaf   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   TransMapPsl_False_True_True_2d38edffaf   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   Chaining_False_True_True_6347a3a45f   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   BuildDb_False_True_True_6347a3a45f   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   GenerateHints_False_True_True_ff00cb1334   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   GenerateHints_False_True_True_830a9a1831   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   GenerateHints_False_True_True_3643911bff   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   GenerateHints_False_True_True_c9796a526b   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   GenerateHints_False_True_True_b8276c29f2   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   GenerateHints_False_True_True_b4484b1ddf   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   GenerateHints_False_True_True_f0ebeace9d   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   GenerateHints_False_True_True_7e58253de0   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   PrepareFiles_False_True_True_6347a3a45f   has status   PENDING
09-20 13:51:20 luigi-interface INFO  Done scheduling tasks
09-20 13:51:20 luigi-interface INFO  Running Worker with 10 processes
09-20 13:51:20 luigi-interface INFO  [pid 144800] Worker Worker(salt=558226303, workers=10, host=login-e-10, username=fc464, pid=144360) running   Task: Gff3ToGenePred
09-20 13:51:20 cat   INFO  Converting annotation gff3 to genePred.
-rnaNameAttr is not a valid option
09-20 13:51:20 luigi-interface ERROR [pid 144800] Worker Worker(salt=558226303, workers=10, host=login-e-10, username=fc464, pid=144360) failed    Task: Gff3ToGenePred
Traceback (most recent call last):
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps
    task_gen = self.task.run()
  File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/cat/__init__.py", line 817, in run
    self.run_cmd(cmd)
  File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/cat/__init__.py", line 425, in run_cmd
    tools.procOps.run_proc(cmd, stdout=outf)
  File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/procOps.py", line 60, in run_proc
    pl.wait()
  File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait
    self.raiseIfExcept()
  File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept
    p.raiseIfExcept()
  File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept
    raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2]
ProcException: process exited 255: gff3ToGenePred -rnaNameAttr=transcript_id -geneNameAttr=gene_id -honorStartStopCodons -attrsOut=/home/fc464/software/luigi/Comparative-Annotation-Toolkit/test_install/reference/GRCm38.mm10.subset.gp_attrs /home/fc464/software/luigi/Comparative-Annotation-Toolkit/test_data/GRCm38.mm10.subset.gff3 /dev/stdout
09-20 13:51:20 luigi-interface INFO  Informed scheduler that task   Gff3ToGenePred_False__home_fc464_soft__home_fc464_soft_ec7fb656b4   has status   FAILED
09-20 13:51:20 luigi-interface INFO  Worker Worker(salt=558226303, workers=10, host=login-e-10, username=fc464, pid=144360) was stopped. Shutting down Keep-Alive thread
09-20 13:51:21 luigi-interface INFO  
===== Luigi Execution Summary =====

Scheduled 103 tasks of which:
* 3 complete ones were encountered:
    - 1 GenomeFiles(...)
    - 1 GenomeFlatFasta(...)
    - 1 GenomeSizes(...)
* 1 failed:
    - 1 Gff3ToGenePred(...)
* 1 failed scheduling:
    - 1 CreateTracks(...)
* 98 were left pending, among these:
    * 98 had failed dependencies:
        - 7 AlignTranscriptDriverTask(...)
        - 1 AlignTranscripts(...)
        - 1 AssemblyHub(...)
        - 1 Augustus(...)
        - 1 AugustusCgp(...)
        ...
    * 1 had dependencies whose scheduling failed:
        - 1 AssemblyHub(...)

This progress looks :( because there were failed tasks and tasks whose scheduling failed

===== Luigi Execution Summary =====

lassancejm commented 6 years ago

Based on your error message, it seems that wiggletools is not available:

ToolMissingException: Tool wiggletools not in global path.

francicco commented 6 years ago

Yes, I figured that out, I solved it but there are others CAT.log

francicco commented 6 years ago

I'm recompiling Augustus

lassancejm commented 6 years ago

Did you try deleting the output folder before restarting the pipeline? The error seems to be associated with toil.

francicco commented 6 years ago

Now I try, also filterBam it's hard to compile!!!

francicco commented 6 years ago

I fixed all dependencies... I think. Now I get this:

[...]
09-21 12:40:46 cat   INFO  Loaded table: rn6.augTM_Hgm
09-21 12:40:47 luigi-interface ERROR [pid 10684] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) failed    ToilTask: AugustusCgp using batchSystem singleMachine
Traceback (most recent call last):
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps
    task_gen = self.task.run()
  File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/cat/__init__.py", line 1446, in run
    augustus_cgp(cgp_args, toil_options)
  File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/cat/augustus_cgp.py", line 59, in augustus_cgp
    results, stdout_file_ids, param_file_id = t.restart()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 814, in restart
    return self._runMainLoop(rootJobGraph)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 212, in run
    self.innerLoop()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 521, in innerLoop
    self._processReadyJobs()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 445, in _processReadyJobs
    self._processReadyJob(jobGraph, resultStatus)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 395, in _processReadyJob
    self.issueJob(JobNode.fromJobGraph(jobGraph))
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 593, in issueJob
    jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/singleMachine.py", line 236, in issueBatchJob
    self.checkResourceRequest(jobNode.memory, cores, jobNode.disk)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/abstractBatchSystem.py", line 251, in checkResourceRequest
    raise InsufficientSystemResources('disk', disk, self.maxDisk)
InsufficientSystemResources: Requesting more disk than either physically available, or enforced by --maxDisk. Requested: 45097156608, Available: 17179852800
09-21 12:40:47 luigi-interface INFO  Informed scheduler that task   AugustusCgp_False_True_True_4bab51e20b   has status   FAILED
09-21 12:40:47 luigi-interface INFO  [pid 11670] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) running   ToilTask: AugustusPbDriverTask for hg38 using batchSystem singleMachine
09-21 12:40:47 cat   INFO  Loaded table: bosTau8.augTM_Hgm
09-21 12:40:47 cat   INFO  Evaluating transcript alignments for hg38.
09-21 12:40:48 cat   INFO  Loaded table: canFam3.augTM_Hgm
09-21 12:40:49 cat   INFO  Evaluating transcript alignments for canFam3.
09-21 12:40:49 cat   INFO  Evaluating transcript alignments for galGal4.
09-21 12:40:49 cat   INFO  Evaluating transcript alignments for bosTau8.
09-21 12:40:49 cat   INFO  Loaded table: galGal4.augTM_Hgm
09-21 12:40:49 cat   INFO  Evaluating transcript alignments for rheMac3.
09-21 12:40:49 cat   INFO  Evaluating transcript alignments for rn6.
09-21 12:40:49 cat   INFO  Evaluating transcript alignments for monDom5.
09-21 12:40:49 luigi-interface INFO  [pid 10326] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done      Task: HgmDriverTask for augTM
09-21 12:40:49 luigi-interface INFO  Informed scheduler that task   HgmDriverTask_False_True_True_6a1c9d2ccf   has status   DONE
09-21 12:40:49 luigi-interface INFO  [pid 11942] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) running   Task: EvaluateTransMapDriverTask for bosTau8
09-21 12:40:49 cat   INFO  Loaded table: hg38.mRNA_augTM_Evaluation
09-21 12:40:50 cat   INFO  Launching homGeneMapping for transMap.
09-21 12:40:51 cat   INFO  Loaded table: hg38.mRNA_augTMR_Metrics
09-21 12:40:51 cat   INFO  Loaded table: bosTau8.mRNA_augTM_Evaluation
09-21 12:40:51 cat   INFO  Loaded table: canFam3.mRNA_augTM_Evaluation
09-21 12:40:51 cat   INFO  Loaded table: rn6.mRNA_augTM_Evaluation
09-21 12:40:51 cat   INFO  Launching AugustusPB toil pipeline on hg38.
09-21 12:40:51 cat   INFO  Loaded table: galGal4.mRNA_augTM_Evaluation
09-21 12:40:51 cat   INFO  Loaded table: monDom5.mRNA_augTM_Evaluation
09-21 12:40:52 cat   INFO  Loaded table: rheMac3.mRNA_augTM_Evaluation
09-21 12:40:52 cat   INFO  Loaded table: rn6.CDS_augTM_Metrics
09-21 12:40:52 cat   INFO  Loaded table: canFam3.CDS_augTM_Metrics
09-21 12:40:52 cat   INFO  Loaded table: bosTau8.CDS_augTM_Metrics
09-21 12:40:52 cat   INFO  Loaded table: hg38.mRNA_transMap_Evaluation
09-21 12:40:52 cat   INFO  Loaded table: galGal4.mRNA_augTMR_Metrics
09-21 12:40:52 cat   INFO  Loaded table: monDom5.CDS_augTM_Metrics
09-21 12:40:53 cat   INFO  Loaded table: rheMac3.mRNA_augTMR_Metrics
09-21 12:40:53 cat   INFO  Loaded table: rn6.mRNA_transMap_Evaluation
09-21 12:40:53 cat   INFO  Loaded table: canFam3.mRNA_transMap_Evaluation
09-21 12:40:53 luigi-interface ERROR [pid 11670] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) failed    ToilTask: AugustusPbDriverTask for hg38 using batchSystem singleMachine
Traceback (most recent call last):
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps
    task_gen = self.task.run()
  File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/cat/__init__.py", line 1508, in run
    augustus_pb(augustus_pb_args, toil_options)
  File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/cat/augustus_pb.py", line 44, in augustus_pb
    raw_gtf_file_id, gtf_file_id, joined_gp_file_id = t.restart()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 814, in restart
    return self._runMainLoop(rootJobGraph)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 212, in run
    self.innerLoop()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 521, in innerLoop
    self._processReadyJobs()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 445, in _processReadyJobs
    self._processReadyJob(jobGraph, resultStatus)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 395, in _processReadyJob
    self.issueJob(JobNode.fromJobGraph(jobGraph))
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 593, in issueJob
    jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/singleMachine.py", line 236, in issueBatchJob
    self.checkResourceRequest(jobNode.memory, cores, jobNode.disk)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/abstractBatchSystem.py", line 251, in checkResourceRequest
    raise InsufficientSystemResources('disk', disk, self.maxDisk)
InsufficientSystemResources: Requesting more disk than either physically available, or enforced by --maxDisk. Requested: 34359738368, Available: 17179852800
09-21 12:40:53 luigi-interface INFO  Informed scheduler that task   AugustusPbDriverTask_False_True_True_a83bd75adf   has status   FAILED
09-21 12:40:53 cat   INFO  Loaded table: galGal4.mRNA_transMap_Evaluation
09-21 12:40:53 luigi-interface INFO  [pid 12169] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) running   Task: EvaluateTransMapDriverTask for canFam3
09-21 12:40:53 cat   INFO  Loaded table: bosTau8.mRNA_transMap_Evaluation
09-21 12:40:53 cat   INFO  Loaded table: hg38.CDS_transMap_Metrics
09-21 12:40:53 cat   INFO  Loaded table: monDom5.mRNA_transMap_Evaluation
09-21 12:40:54 cat   INFO  Loaded table: rheMac3.mRNA_transMap_Evaluation
09-21 12:40:54 cat   INFO  Loaded table: bosTau8.CDS_transMap_Metrics
09-21 12:40:54 cat   INFO  Loaded table: canFam3.CDS_transMap_Metrics
09-21 12:40:54 cat   INFO  Loaded table: rn6.CDS_transMap_Metrics
09-21 12:40:54 cat   INFO  Loaded table: galGal4.CDS_transMap_Metrics
09-21 12:40:55 cat   INFO  Loaded table: monDom5.CDS_transMap_Metrics
09-21 12:40:55 cat   INFO  Loaded table: hg38.CDS_augTM_Metrics
09-21 12:40:55 cat   INFO  Loaded table: rheMac3.CDS_transMap_Metrics
09-21 12:40:55 cat   INFO  Loaded table: bosTau8.mRNA_augTM_Metrics
09-21 12:40:55 cat   INFO  Loaded table: canFam3.mRNA_augTM_Metrics
09-21 12:40:55 cat   INFO  Loaded table: galGal4.CDS_augTM_Metrics
09-21 12:40:56 cat   INFO  Loaded table: rheMac3.CDS_augTM_Metrics
09-21 12:40:56 cat   INFO  Loaded table: hg38.CDS_augTMR_Evaluation
09-21 12:40:56 cat   INFO  Loaded table: rn6.mRNA_augTM_Metrics
09-21 12:40:56 cat   INFO  Loaded table: monDom5.mRNA_augTM_Metrics
09-21 12:40:56 cat   INFO  Loaded table: galGal4.CDS_augTMR_Evaluation
09-21 12:40:56 cat   INFO  Loaded table: bosTau8.CDS_transMap_Evaluation
09-21 12:40:56 cat   INFO  Loaded table: canFam3.CDS_transMap_Evaluation
09-21 12:40:57 cat   INFO  Loaded table: rheMac3.CDS_augTMR_Evaluation
09-21 12:40:57 cat   INFO  Loaded table: hg38.CDS_augTMR_Metrics
09-21 12:40:57 cat   INFO  Loaded table: rn6.CDS_transMap_Evaluation
09-21 12:40:57 cat   INFO  Loaded table: galGal4.CDS_augTMR_Metrics
09-21 12:40:57 cat   INFO  Loaded table: monDom5.transMap_Hgm
09-21 12:40:57 cat   INFO  Loaded table: canFam3.mRNA_transMap_Metrics
09-21 12:40:57 cat   INFO  Loaded table: bosTau8.mRNA_transMap_Metrics
09-21 12:40:57 cat   INFO  Loaded table: rn6.mRNA_transMap_Metrics
09-21 12:40:58 cat   INFO  Loaded table: hg38.mRNA_augTMR_Evaluation
09-21 12:40:58 cat   INFO  Loaded table: monDom5.CDS_transMap_Evaluation
09-21 12:40:58 cat   INFO  Loaded table: galGal4.mRNA_augTMR_Evaluation
09-21 12:40:58 cat   INFO  Loaded table: rheMac3.CDS_augTMR_Metrics
09-21 12:40:58 cat   INFO  Loaded table: rn6.CDS_augTM_Evaluation
09-21 12:40:58 cat   INFO  Loaded table: canFam3.CDS_augTM_Evaluation
09-21 12:40:58 luigi-interface INFO  [pid 10686] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done      Task: EvaluateDriverTask for rn6
09-21 12:40:58 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_70026afda7   has status   DONE
09-21 12:40:58 cat   INFO  Loaded table: bosTau8.CDS_augTM_Evaluation
09-21 12:40:58 luigi-interface INFO  [pid 12352] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) running   Task: EvaluateTransMapDriverTask for hg38
09-21 12:40:58 cat   INFO  Loaded table: hg38.mRNA_augTM_Metrics
09-21 12:40:58 cat   INFO  Loaded table: galGal4.mRNA_augTM_Metrics
09-21 12:40:58 cat   INFO  Loaded table: rheMac3.mRNA_augTMR_Evaluation
09-21 12:40:59 luigi-interface INFO  [pid 10687] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done      Task: EvaluateDriverTask for canFam3
09-21 12:40:59 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_f4715e1d3c   has status   DONE
09-21 12:40:59 luigi-interface INFO  [pid 12382] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) running   Task: EvaluateTransMapDriverTask for rn6
09-21 12:40:59 cat   INFO  Loaded table: monDom5.mRNA_transMap_Metrics
09-21 12:40:59 luigi-interface INFO  [pid 10711] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done      Task: EvaluateDriverTask for bosTau8
09-21 12:40:59 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_902022f711   has status   DONE
09-21 12:40:59 luigi-interface INFO  [pid 12416] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) running   Task: EvaluateTransMapDriverTask for monDom5
09-21 12:40:59 cat   INFO  Loaded table: galGal4.CDS_transMap_Evaluation
09-21 12:40:59 cat   INFO  Evaluating transMap results for bosTau8.
09-21 12:41:00 cat   INFO  Loaded table: rheMac3.mRNA_augTM_Metrics
09-21 12:41:00 cat   INFO  Loaded table: monDom5.CDS_augTM_Evaluation
09-21 12:41:00 cat   INFO  Loaded table: hg38.transMap_Hgm
09-21 12:41:00 cat   INFO  Loaded table: galGal4.mRNA_transMap_Metrics
09-21 12:41:00 cat   INFO  Loaded table: hg38.CDS_transMap_Evaluation
09-21 12:41:00 luigi-interface INFO  [pid 10721] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done      Task: EvaluateDriverTask for monDom5
09-21 12:41:00 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_41c87a08f0   has status   DONE
09-21 12:41:00 luigi-interface INFO  [pid 12492] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) running   Task: EvaluateTransMapDriverTask for galGal4
09-21 12:41:01 cat   INFO  Loaded table: bosTau8.TransMapEvaluation
09-21 12:41:01 luigi-interface INFO  [pid 11942] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done      Task: EvaluateTransMapDriverTask for bosTau8
09-21 12:41:01 luigi-interface INFO  Informed scheduler that task   EvaluateTransMapDriverTask_False_True_True_b217dbbb13   has status   DONE
09-21 12:41:01 luigi-interface INFO  [pid 12528] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) running   Task: EvaluateTransMapDriverTask for rheMac3
09-21 12:41:01 cat   INFO  Loaded table: hg38.mRNA_transMap_Metrics
09-21 12:41:01 cat   INFO  Loaded table: galGal4.CDS_augTM_Evaluation
09-21 12:41:02 luigi-interface INFO  [pid 10702] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done      Task: EvaluateDriverTask for galGal4
09-21 12:41:02 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_9c9943f49b   has status   DONE
09-21 12:41:02 cat   INFO  Loaded table: hg38.CDS_augTM_Evaluation
09-21 12:41:02 cat   INFO  Loaded table: rheMac3.transMap_Hgm
09-21 12:41:02 luigi-interface INFO  [pid 10717] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done      Task: EvaluateDriverTask for hg38
09-21 12:41:02 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_a83bd75adf   has status   DONE
09-21 12:41:02 cat   INFO  Loaded table: rheMac3.CDS_transMap_Evaluation
09-21 12:41:03 cat   INFO  Loaded table: rn6.transMap_Hgm
09-21 12:41:03 cat   INFO  Loaded table: rheMac3.mRNA_transMap_Metrics
09-21 12:41:03 cat   INFO  Evaluating transMap results for canFam3.
09-21 12:41:04 cat   INFO  Loaded table: bosTau8.transMap_Hgm
09-21 12:41:04 cat   INFO  Loaded table: rheMac3.CDS_augTM_Evaluation
09-21 12:41:05 luigi-interface INFO  [pid 10690] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done      Task: EvaluateDriverTask for rheMac3
09-21 12:41:05 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_8b824cb1d0   has status   DONE
09-21 12:41:05 luigi-interface INFO  [pid 12835] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) running   Task: EvaluateTranscripts
09-21 12:41:05 cat   INFO  Evaluating transMap results for monDom5.
09-21 12:41:05 cat   INFO  Evaluating transMap results for hg38.
09-21 12:41:05 cat   INFO  Loaded table: canFam3.transMap_Hgm
09-21 12:41:06 cat   INFO  Loaded table: canFam3.TransMapEvaluation
09-21 12:41:06 luigi-interface INFO  [pid 12169] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done      Task: EvaluateTransMapDriverTask for canFam3
09-21 12:41:06 luigi-interface INFO  Informed scheduler that task   EvaluateTransMapDriverTask_False_True_True_b6b58a1a87   has status   DONE
09-21 12:41:06 cat   INFO  Loaded table: monDom5.TransMapEvaluation
09-21 12:41:06 cat   INFO  Loaded table: hg38.TransMapEvaluation
09-21 12:41:06 cat   INFO  Loaded table: galGal4.transMap_Hgm
09-21 12:41:06 luigi-interface INFO  [pid 12416] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done      Task: EvaluateTransMapDriverTask for monDom5
09-21 12:41:06 cat   INFO  Evaluating transMap results for galGal4.
09-21 12:41:06 luigi-interface INFO  Informed scheduler that task   EvaluateTransMapDriverTask_False_True_True_0c8a652f49   has status   DONE
09-21 12:41:07 cat   INFO  Evaluating transMap results for rn6.
09-21 12:41:07 cat   INFO  Evaluating transMap results for rheMac3.
09-21 12:41:07 luigi-interface INFO  [pid 11302] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done      Task: HgmDriverTask for transMap
09-21 12:41:07 luigi-interface INFO  Informed scheduler that task   HgmDriverTask_False_True_True_66bc76859d   has status   DONE
09-21 12:41:07 luigi-interface INFO  [pid 12352] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done      Task: EvaluateTransMapDriverTask for hg38
09-21 12:41:07 luigi-interface INFO  Informed scheduler that task   EvaluateTransMapDriverTask_False_True_True_62f08feeb1   has status   DONE
09-21 12:41:08 cat   INFO  Loaded table: rn6.TransMapEvaluation
09-21 12:41:08 luigi-interface INFO  [pid 12382] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done      Task: EvaluateTransMapDriverTask for rn6
09-21 12:41:08 luigi-interface INFO  Informed scheduler that task   EvaluateTransMapDriverTask_False_True_True_fc7d9960f5   has status   DONE
09-21 12:41:08 cat   INFO  Loaded table: rheMac3.TransMapEvaluation
09-21 12:41:08 cat   INFO  Loaded table: galGal4.TransMapEvaluation
09-21 12:41:08 luigi-interface INFO  [pid 12528] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done      Task: EvaluateTransMapDriverTask for rheMac3
09-21 12:41:08 luigi-interface INFO  Informed scheduler that task   EvaluateTransMapDriverTask_False_True_True_cd6004532f   has status   DONE
09-21 12:41:08 luigi-interface INFO  [pid 12492] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done      Task: EvaluateTransMapDriverTask for galGal4
09-21 12:41:08 luigi-interface INFO  Informed scheduler that task   EvaluateTransMapDriverTask_False_True_True_01193b493b   has status   DONE
09-21 12:41:08 luigi-interface INFO  [pid 13019] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) running   Task: EvaluateTransMap
09-21 12:41:11 luigi-interface INFO  [pid 13019] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done      Task: EvaluateTransMap
09-21 12:41:11 luigi-interface INFO  Informed scheduler that task   EvaluateTransMap_False_True_True_4bab51e20b   has status   DONE
09-21 12:41:11 luigi-interface INFO  [pid 12835] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done      Task: EvaluateTranscripts
09-21 12:41:11 luigi-interface INFO  Informed scheduler that task   EvaluateTranscripts_False_True_True_4bab51e20b   has status   DONE
09-21 12:41:11 luigi-interface INFO  Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) was stopped. Shutting down Keep-Alive thread
09-21 12:41:13 luigi-interface INFO  
===== Luigi Execution Summary =====

Scheduled 140 tasks of which:
* 25 complete ones were encountered:
    - 1 AlignTranscripts(...)
    - 1 Augustus(...)
    - 1 BuildDb(...)
    - 1 Chaining(...)
    - 1 FakePsl(...)
    ...
* 32 ran successfully:
    - 7 EvaluateDriverTask(...)
    - 1 EvaluateTransMap(...)
    - 7 EvaluateTransMapDriverTask(...)
    - 1 EvaluateTranscripts(...)
    - 7 FilterTransMap(...)
    ...
* 2 failed:
    - 1 AugustusCgp(...)
    - 1 AugustusPbDriverTask(...)
* 81 were left pending, among these:
    * 81 had failed dependencies:
        - 1 AssemblyHub(...)
        - 1 AugustusPb(...)
        - 3 AugustusTrack(...)
        - 8 BgpTrack(...)
        - 1 Consensus(...)
        ...

This progress looks :( because there were failed tasks

===== Luigi Execution Summary =====

ifiddes commented 6 years ago

Your temporary disk doesn’t have enough space. Perhaps you are using a machine with a small temporary drive? You can set —workDir to a path that exists on a drive with more space.

Ian Fiddes

On Sep 21, 2018, at 12:43 PM, Francesco Cicconardi notifications@github.com wrote:

I fixed all dependencies... I think. Now I get this:

[...] 09-21 12:40:46 cat INFO Loaded table: rn6.augTM_Hgm 09-21 12:40:47 luigi-interface ERROR [pid 10684] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) failed ToilTask: AugustusCgp using batchSystem singleMachine Traceback (most recent call last): File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run new_deps = self._run_get_new_deps() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps task_gen = self.task.run() File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/cat/init.py", line 1446, in run augustus_cgp(cgp_args, toil_options) File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/cat/augustus_cgp.py", line 59, in augustus_cgp results, stdout_file_ids, param_file_id = t.restart() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 814, in restart return self._runMainLoop(rootJobGraph) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop jobCache=self._jobCache).run() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 212, in run self.innerLoop() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 521, in innerLoop self._processReadyJobs() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 445, in _processReadyJobs self._processReadyJob(jobGraph, resultStatus) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 395, in _processReadyJob self.issueJob(JobNode.fromJobGraph(jobGraph)) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 593, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/singleMachine.py", line 236, in issueBatchJob self.checkResourceRequest(jobNode.memory, cores, jobNode.disk) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/abstractBatchSystem.py", line 251, in checkResourceRequest raise InsufficientSystemResources('disk', disk, self.maxDisk) InsufficientSystemResources: Requesting more disk than either physically available, or enforced by --maxDisk. Requested: 45097156608, Available: 17179852800 09-21 12:40:47 luigi-interface INFO Informed scheduler that task AugustusCgp_False_True_True_4bab51e20b has status FAILED 09-21 12:40:47 luigi-interface INFO [pid 11670] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) running ToilTask: AugustusPbDriverTask for hg38 using batchSystem singleMachine 09-21 12:40:47 cat INFO Loaded table: bosTau8.augTM_Hgm 09-21 12:40:47 cat INFO Evaluating transcript alignments for hg38. 09-21 12:40:48 cat INFO Loaded table: canFam3.augTM_Hgm 09-21 12:40:49 cat INFO Evaluating transcript alignments for canFam3. 09-21 12:40:49 cat INFO Evaluating transcript alignments for galGal4. 09-21 12:40:49 cat INFO Evaluating transcript alignments for bosTau8. 09-21 12:40:49 cat INFO Loaded table: galGal4.augTM_Hgm 09-21 12:40:49 cat INFO Evaluating transcript alignments for rheMac3. 09-21 12:40:49 cat INFO Evaluating transcript alignments for rn6. 09-21 12:40:49 cat INFO Evaluating transcript alignments for monDom5. 09-21 12:40:49 luigi-interface INFO [pid 10326] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done Task: HgmDriverTask for augTM 09-21 12:40:49 luigi-interface INFO Informed scheduler that task HgmDriverTask_False_True_True_6a1c9d2ccf has status DONE 09-21 12:40:49 luigi-interface INFO [pid 11942] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) running Task: EvaluateTransMapDriverTask for bosTau8 09-21 12:40:49 cat INFO Loaded table: hg38.mRNA_augTM_Evaluation 09-21 12:40:50 cat INFO Launching homGeneMapping for transMap. 09-21 12:40:51 cat INFO Loaded table: hg38.mRNA_augTMR_Metrics 09-21 12:40:51 cat INFO Loaded table: bosTau8.mRNA_augTM_Evaluation 09-21 12:40:51 cat INFO Loaded table: canFam3.mRNA_augTM_Evaluation 09-21 12:40:51 cat INFO Loaded table: rn6.mRNA_augTM_Evaluation 09-21 12:40:51 cat INFO Launching AugustusPB toil pipeline on hg38. 09-21 12:40:51 cat INFO Loaded table: galGal4.mRNA_augTM_Evaluation 09-21 12:40:51 cat INFO Loaded table: monDom5.mRNA_augTM_Evaluation 09-21 12:40:52 cat INFO Loaded table: rheMac3.mRNA_augTM_Evaluation 09-21 12:40:52 cat INFO Loaded table: rn6.CDS_augTM_Metrics 09-21 12:40:52 cat INFO Loaded table: canFam3.CDS_augTM_Metrics 09-21 12:40:52 cat INFO Loaded table: bosTau8.CDS_augTM_Metrics 09-21 12:40:52 cat INFO Loaded table: hg38.mRNA_transMap_Evaluation 09-21 12:40:52 cat INFO Loaded table: galGal4.mRNA_augTMR_Metrics 09-21 12:40:52 cat INFO Loaded table: monDom5.CDS_augTM_Metrics 09-21 12:40:53 cat INFO Loaded table: rheMac3.mRNA_augTMR_Metrics 09-21 12:40:53 cat INFO Loaded table: rn6.mRNA_transMap_Evaluation 09-21 12:40:53 cat INFO Loaded table: canFam3.mRNA_transMap_Evaluation 09-21 12:40:53 luigi-interface ERROR [pid 11670] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) failed ToilTask: AugustusPbDriverTask for hg38 using batchSystem singleMachine Traceback (most recent call last): File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run new_deps = self._run_get_new_deps() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps task_gen = self.task.run() File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/cat/init.py", line 1508, in run augustus_pb(augustus_pb_args, toil_options) File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/cat/augustus_pb.py", line 44, in augustus_pb raw_gtf_file_id, gtf_file_id, joined_gp_file_id = t.restart() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 814, in restart return self._runMainLoop(rootJobGraph) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop jobCache=self._jobCache).run() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 212, in run self.innerLoop() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 521, in innerLoop self._processReadyJobs() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 445, in _processReadyJobs self._processReadyJob(jobGraph, resultStatus) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 395, in _processReadyJob self.issueJob(JobNode.fromJobGraph(jobGraph)) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 593, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/singleMachine.py", line 236, in issueBatchJob self.checkResourceRequest(jobNode.memory, cores, jobNode.disk) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/abstractBatchSystem.py", line 251, in checkResourceRequest raise InsufficientSystemResources('disk', disk, self.maxDisk) InsufficientSystemResources: Requesting more disk than either physically available, or enforced by --maxDisk. Requested: 34359738368, Available: 17179852800 09-21 12:40:53 luigi-interface INFO Informed scheduler that task AugustusPbDriverTask_False_True_True_a83bd75adf has status FAILED 09-21 12:40:53 cat INFO Loaded table: galGal4.mRNA_transMap_Evaluation 09-21 12:40:53 luigi-interface INFO [pid 12169] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) running Task: EvaluateTransMapDriverTask for canFam3 09-21 12:40:53 cat INFO Loaded table: bosTau8.mRNA_transMap_Evaluation 09-21 12:40:53 cat INFO Loaded table: hg38.CDS_transMap_Metrics 09-21 12:40:53 cat INFO Loaded table: monDom5.mRNA_transMap_Evaluation 09-21 12:40:54 cat INFO Loaded table: rheMac3.mRNA_transMap_Evaluation 09-21 12:40:54 cat INFO Loaded table: bosTau8.CDS_transMap_Metrics 09-21 12:40:54 cat INFO Loaded table: canFam3.CDS_transMap_Metrics 09-21 12:40:54 cat INFO Loaded table: rn6.CDS_transMap_Metrics 09-21 12:40:54 cat INFO Loaded table: galGal4.CDS_transMap_Metrics 09-21 12:40:55 cat INFO Loaded table: monDom5.CDS_transMap_Metrics 09-21 12:40:55 cat INFO Loaded table: hg38.CDS_augTM_Metrics 09-21 12:40:55 cat INFO Loaded table: rheMac3.CDS_transMap_Metrics 09-21 12:40:55 cat INFO Loaded table: bosTau8.mRNA_augTM_Metrics 09-21 12:40:55 cat INFO Loaded table: canFam3.mRNA_augTM_Metrics 09-21 12:40:55 cat INFO Loaded table: galGal4.CDS_augTM_Metrics 09-21 12:40:56 cat INFO Loaded table: rheMac3.CDS_augTM_Metrics 09-21 12:40:56 cat INFO Loaded table: hg38.CDS_augTMR_Evaluation 09-21 12:40:56 cat INFO Loaded table: rn6.mRNA_augTM_Metrics 09-21 12:40:56 cat INFO Loaded table: monDom5.mRNA_augTM_Metrics 09-21 12:40:56 cat INFO Loaded table: galGal4.CDS_augTMR_Evaluation 09-21 12:40:56 cat INFO Loaded table: bosTau8.CDS_transMap_Evaluation 09-21 12:40:56 cat INFO Loaded table: canFam3.CDS_transMap_Evaluation 09-21 12:40:57 cat INFO Loaded table: rheMac3.CDS_augTMR_Evaluation 09-21 12:40:57 cat INFO Loaded table: hg38.CDS_augTMR_Metrics 09-21 12:40:57 cat INFO Loaded table: rn6.CDS_transMap_Evaluation 09-21 12:40:57 cat INFO Loaded table: galGal4.CDS_augTMR_Metrics 09-21 12:40:57 cat INFO Loaded table: monDom5.transMap_Hgm 09-21 12:40:57 cat INFO Loaded table: canFam3.mRNA_transMap_Metrics 09-21 12:40:57 cat INFO Loaded table: bosTau8.mRNA_transMap_Metrics 09-21 12:40:57 cat INFO Loaded table: rn6.mRNA_transMap_Metrics 09-21 12:40:58 cat INFO Loaded table: hg38.mRNA_augTMR_Evaluation 09-21 12:40:58 cat INFO Loaded table: monDom5.CDS_transMap_Evaluation 09-21 12:40:58 cat INFO Loaded table: galGal4.mRNA_augTMR_Evaluation 09-21 12:40:58 cat INFO Loaded table: rheMac3.CDS_augTMR_Metrics 09-21 12:40:58 cat INFO Loaded table: rn6.CDS_augTM_Evaluation 09-21 12:40:58 cat INFO Loaded table: canFam3.CDS_augTM_Evaluation 09-21 12:40:58 luigi-interface INFO [pid 10686] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done Task: EvaluateDriverTask for rn6 09-21 12:40:58 luigi-interface INFO Informed scheduler that task EvaluateDriverTask_False_True_True_70026afda7 has status DONE 09-21 12:40:58 cat INFO Loaded table: bosTau8.CDS_augTM_Evaluation 09-21 12:40:58 luigi-interface INFO [pid 12352] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) running Task: EvaluateTransMapDriverTask for hg38 09-21 12:40:58 cat INFO Loaded table: hg38.mRNA_augTM_Metrics 09-21 12:40:58 cat INFO Loaded table: galGal4.mRNA_augTM_Metrics 09-21 12:40:58 cat INFO Loaded table: rheMac3.mRNA_augTMR_Evaluation 09-21 12:40:59 luigi-interface INFO [pid 10687] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done Task: EvaluateDriverTask for canFam3 09-21 12:40:59 luigi-interface INFO Informed scheduler that task EvaluateDriverTask_False_True_True_f4715e1d3c has status DONE 09-21 12:40:59 luigi-interface INFO [pid 12382] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) running Task: EvaluateTransMapDriverTask for rn6 09-21 12:40:59 cat INFO Loaded table: monDom5.mRNA_transMap_Metrics 09-21 12:40:59 luigi-interface INFO [pid 10711] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done Task: EvaluateDriverTask for bosTau8 09-21 12:40:59 luigi-interface INFO Informed scheduler that task EvaluateDriverTask_False_True_True_902022f711 has status DONE 09-21 12:40:59 luigi-interface INFO [pid 12416] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) running Task: EvaluateTransMapDriverTask for monDom5 09-21 12:40:59 cat INFO Loaded table: galGal4.CDS_transMap_Evaluation 09-21 12:40:59 cat INFO Evaluating transMap results for bosTau8. 09-21 12:41:00 cat INFO Loaded table: rheMac3.mRNA_augTM_Metrics 09-21 12:41:00 cat INFO Loaded table: monDom5.CDS_augTM_Evaluation 09-21 12:41:00 cat INFO Loaded table: hg38.transMap_Hgm 09-21 12:41:00 cat INFO Loaded table: galGal4.mRNA_transMap_Metrics 09-21 12:41:00 cat INFO Loaded table: hg38.CDS_transMap_Evaluation 09-21 12:41:00 luigi-interface INFO [pid 10721] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done Task: EvaluateDriverTask for monDom5 09-21 12:41:00 luigi-interface INFO Informed scheduler that task EvaluateDriverTask_False_True_True_41c87a08f0 has status DONE 09-21 12:41:00 luigi-interface INFO [pid 12492] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) running Task: EvaluateTransMapDriverTask for galGal4 09-21 12:41:01 cat INFO Loaded table: bosTau8.TransMapEvaluation 09-21 12:41:01 luigi-interface INFO [pid 11942] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done Task: EvaluateTransMapDriverTask for bosTau8 09-21 12:41:01 luigi-interface INFO Informed scheduler that task EvaluateTransMapDriverTask_False_True_True_b217dbbb13 has status DONE 09-21 12:41:01 luigi-interface INFO [pid 12528] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) running Task: EvaluateTransMapDriverTask for rheMac3 09-21 12:41:01 cat INFO Loaded table: hg38.mRNA_transMap_Metrics 09-21 12:41:01 cat INFO Loaded table: galGal4.CDS_augTM_Evaluation 09-21 12:41:02 luigi-interface INFO [pid 10702] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done Task: EvaluateDriverTask for galGal4 09-21 12:41:02 luigi-interface INFO Informed scheduler that task EvaluateDriverTask_False_True_True_9c9943f49b has status DONE 09-21 12:41:02 cat INFO Loaded table: hg38.CDS_augTM_Evaluation 09-21 12:41:02 cat INFO Loaded table: rheMac3.transMap_Hgm 09-21 12:41:02 luigi-interface INFO [pid 10717] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done Task: EvaluateDriverTask for hg38 09-21 12:41:02 luigi-interface INFO Informed scheduler that task EvaluateDriverTask_False_True_True_a83bd75adf has status DONE 09-21 12:41:02 cat INFO Loaded table: rheMac3.CDS_transMap_Evaluation 09-21 12:41:03 cat INFO Loaded table: rn6.transMap_Hgm 09-21 12:41:03 cat INFO Loaded table: rheMac3.mRNA_transMap_Metrics 09-21 12:41:03 cat INFO Evaluating transMap results for canFam3. 09-21 12:41:04 cat INFO Loaded table: bosTau8.transMap_Hgm 09-21 12:41:04 cat INFO Loaded table: rheMac3.CDS_augTM_Evaluation 09-21 12:41:05 luigi-interface INFO [pid 10690] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done Task: EvaluateDriverTask for rheMac3 09-21 12:41:05 luigi-interface INFO Informed scheduler that task EvaluateDriverTask_False_True_True_8b824cb1d0 has status DONE 09-21 12:41:05 luigi-interface INFO [pid 12835] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) running Task: EvaluateTranscripts 09-21 12:41:05 cat INFO Evaluating transMap results for monDom5. 09-21 12:41:05 cat INFO Evaluating transMap results for hg38. 09-21 12:41:05 cat INFO Loaded table: canFam3.transMap_Hgm 09-21 12:41:06 cat INFO Loaded table: canFam3.TransMapEvaluation 09-21 12:41:06 luigi-interface INFO [pid 12169] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done Task: EvaluateTransMapDriverTask for canFam3 09-21 12:41:06 luigi-interface INFO Informed scheduler that task EvaluateTransMapDriverTask_False_True_True_b6b58a1a87 has status DONE 09-21 12:41:06 cat INFO Loaded table: monDom5.TransMapEvaluation 09-21 12:41:06 cat INFO Loaded table: hg38.TransMapEvaluation 09-21 12:41:06 cat INFO Loaded table: galGal4.transMap_Hgm 09-21 12:41:06 luigi-interface INFO [pid 12416] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done Task: EvaluateTransMapDriverTask for monDom5 09-21 12:41:06 cat INFO Evaluating transMap results for galGal4. 09-21 12:41:06 luigi-interface INFO Informed scheduler that task EvaluateTransMapDriverTask_False_True_True_0c8a652f49 has status DONE 09-21 12:41:07 cat INFO Evaluating transMap results for rn6. 09-21 12:41:07 cat INFO Evaluating transMap results for rheMac3. 09-21 12:41:07 luigi-interface INFO [pid 11302] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done Task: HgmDriverTask for transMap 09-21 12:41:07 luigi-interface INFO Informed scheduler that task HgmDriverTask_False_True_True_66bc76859d has status DONE 09-21 12:41:07 luigi-interface INFO [pid 12352] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done Task: EvaluateTransMapDriverTask for hg38 09-21 12:41:07 luigi-interface INFO Informed scheduler that task EvaluateTransMapDriverTask_False_True_True_62f08feeb1 has status DONE 09-21 12:41:08 cat INFO Loaded table: rn6.TransMapEvaluation 09-21 12:41:08 luigi-interface INFO [pid 12382] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done Task: EvaluateTransMapDriverTask for rn6 09-21 12:41:08 luigi-interface INFO Informed scheduler that task EvaluateTransMapDriverTask_False_True_True_fc7d9960f5 has status DONE 09-21 12:41:08 cat INFO Loaded table: rheMac3.TransMapEvaluation 09-21 12:41:08 cat INFO Loaded table: galGal4.TransMapEvaluation 09-21 12:41:08 luigi-interface INFO [pid 12528] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done Task: EvaluateTransMapDriverTask for rheMac3 09-21 12:41:08 luigi-interface INFO Informed scheduler that task EvaluateTransMapDriverTask_False_True_True_cd6004532f has status DONE 09-21 12:41:08 luigi-interface INFO [pid 12492] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done Task: EvaluateTransMapDriverTask for galGal4 09-21 12:41:08 luigi-interface INFO Informed scheduler that task EvaluateTransMapDriverTask_False_True_True_01193b493b has status DONE 09-21 12:41:08 luigi-interface INFO [pid 13019] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) running Task: EvaluateTransMap 09-21 12:41:11 luigi-interface INFO [pid 13019] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done Task: EvaluateTransMap 09-21 12:41:11 luigi-interface INFO Informed scheduler that task EvaluateTransMap_False_True_True_4bab51e20b has status DONE 09-21 12:41:11 luigi-interface INFO [pid 12835] Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) done Task: EvaluateTranscripts 09-21 12:41:11 luigi-interface INFO Informed scheduler that task EvaluateTranscripts_False_True_True_4bab51e20b has status DONE 09-21 12:41:11 luigi-interface INFO Worker Worker(salt=755637362, workers=10, host=cpu-e-396, username=fc464, pid=9582) was stopped. Shutting down Keep-Alive thread 09-21 12:41:13 luigi-interface INFO
===== Luigi Execution Summary =====

Scheduled 140 tasks of which:

25 complete ones were encountered:

1 AlignTranscripts(...)

1 Augustus(...)

1 BuildDb(...)

1 Chaining(...)

1 FakePsl(...) ...

32 ran successfully:

7 EvaluateDriverTask(...)

1 EvaluateTransMap(...)

7 EvaluateTransMapDriverTask(...)

1 EvaluateTranscripts(...)

7 FilterTransMap(...) ...

2 failed:

1 AugustusCgp(...)

1 AugustusPbDriverTask(...)

81 were left pending, among these:

81 had failed dependencies:

1 AssemblyHub(...)

1 AugustusPb(...)

3 AugustusTrack(...)

8 BgpTrack(...)

1 Consensus(...) ...

This progress looks :( because there were failed tasks

===== Luigi Execution Summary ===== — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

francicco commented 6 years ago

Ok, some progress. Now I have this:

[...]
09-21 13:31:40 cat   INFO  Loaded table: rn6.CDS_augTM_Metrics
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 765, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/fileStore.py", line 1471, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 765, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/fileStore.py", line 1471, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method CachingFileStore.__del__ of <toil.fileStore.CachingFileStore object at 0x24b5e50>> ignored
09-21 13:31:40 cat   INFO  Loaded table: rheMac3.mRNA_augTMR_Metrics
09-21 13:31:40 toil.leader WARNING The job seems to have left a log file, indicating failure: 'cgp' u/c/jobx9IR3A
09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A    INFO:toil:Running Toil version 3.17.0-585383ed5c4453b556269818571d3c7419c613b0.
09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A    WARNING:toil.resource:'JTRES_3a2fc781691739384afee22d074d5828' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A    WARNING:toil.resource:'JTRES_3a2fc781691739384afee22d074d5828' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A    
09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A    augustus: ERROR
09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A        Comparative gene prediction not possible with this compiled version. Please recompile with flag COMPGENEPRED set in common.mk.
09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A    
09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A    Traceback (most recent call last):
09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A      File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A      File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/job.py", line 1350, in _runner
09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A        returnValues = self._run(jobGraph, fileStore)
09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A      File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/job.py", line 1295, in _run
09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A        return self.run(fileStore)
09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A      File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/job.py", line 1479, in run
09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A        rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A      File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/cat/augustus_cgp.py", line 247, in cgp
09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A        tools.procOps.run_proc(cmd, stdout=stdout)
09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A      File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/procOps.py", line 60, in run_proc
09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A        pl.wait()
09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A      File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait
09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A        self.raiseIfExcept()
09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A      File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept
09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A        p.raiseIfExcept()
09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A      File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept
09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A        raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2]
09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A    ProcException: process exited 1: augustus --dbhints=1 --allow_hinted_splicesites=atac --extrinsicCfgFile=/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-26e535de-e224-4749-9f27-4454d05a21e8-e16d32a4-25b2-41b6-8d60-768dd5639220/tmpGHsv8r/a31df534-97b2-4866-b429-ea3ea1ed96bd/tmpCDsQgR.tmp --species=human --treefile=/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-26e535de-e224-4749-9f27-4454d05a21e8-e16d32a4-25b2-41b6-8d60-768dd5639220/tmpGHsv8r/a31df534-97b2-4866-b429-ea3ea1ed96bd/tmpbKcJSM.tmp --alnfile=/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-26e535de-e224-4749-9f27-4454d05a21e8-e16d32a4-25b2-41b6-8d60-768dd5639220/tmpGHsv8r/a31df534-97b2-4866-b429-ea3ea1ed96bd/tmpRSh8tE.tmp --dbaccess=/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-26e535de-e224-4749-9f27-4454d05a21e8-e16d32a4-25b2-41b6-8d60-768dd5639220/tmpGHsv8r/a31df534-97b2-4866-b429-ea3ea1ed96bd/tmpjl3Ec_.tmp --speciesfilenames=/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-26e535de-e224-4749-9f27-4454d05a21e8-e16d32a4-25b2-41b6-8d60-768dd5639220/tmpGHsv8r/a31df534-97b2-4866-b429-ea3ea1ed96bd/login-e-16.197146.0476613898.tmp --softmasking=1 --exoncands=1 --alternatives-from-evidence=0 --/CompPred/logreg=on --printOEs=1 --/CompPred/outdir=/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-26e535de-e224-4749-9f27-4454d05a21e8-e16d32a4-25b2-41b6-8d60-768dd5639220/tmpGHsv8r/a31df534-97b2-4866-b429-ea3ea1ed96bd
09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A    ERROR:toil.worker:Exiting the worker because of a failed job on host login-e-16
09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'cgp' u/c/jobx9IR3A with ID u/c/jobx9IR3A to 1
09-21 13:31:40 cat   INFO  Loaded table: galGal4.mRNA_augTMR_Metrics
09-21 13:31:40 cat   INFO  Loaded table: bosTau8.mRNA_transMap_Evaluation
09-21 13:31:40 cat   INFO  Loaded table: rn6.mRNA_transMap_Evaluation
09-21 13:31:41 cat   INFO  Loaded table: galGal4.mRNA_transMap_Evaluation
09-21 13:31:41 cat   INFO  Loaded table: rheMac3.mRNA_transMap_Evaluation
09-21 13:31:41 cat   INFO  Loaded table: bosTau8.CDS_transMap_Metrics
09-21 13:31:41 cat   INFO  Loaded table: galGal4.CDS_transMap_Metrics
09-21 13:31:41 cat   INFO  Loaded table: rn6.CDS_transMap_Metrics
09-21 13:31:42 cat   INFO  Loaded table: galGal4.CDS_augTM_Metrics
09-21 13:31:42 cat   INFO  Loaded table: rheMac3.CDS_transMap_Metrics
09-21 13:31:42 cat   INFO  Loaded table: galGal4.CDS_augTMR_Evaluation
09-21 13:31:42 cat   INFO  Loaded table: rn6.augTM_Hgm
09-21 13:31:42 cat   INFO  Loaded table: bosTau8.mRNA_augTM_Metrics
09-21 13:31:42 cat   INFO  Loaded table: rn6.mRNA_augTM_Metrics
09-21 13:31:43 cat   INFO  Loaded table: rheMac3.CDS_augTM_Metrics
09-21 13:31:43 cat   INFO  Loaded table: galGal4.CDS_augTMR_Metrics
09-21 13:31:43 cat   INFO  Loaded table: rn6.CDS_transMap_Evaluation
09-21 13:31:43 cat   INFO  Loaded table: bosTau8.CDS_transMap_Evaluation
09-21 13:31:43 cat   INFO  Loaded table: rheMac3.CDS_augTMR_Evaluation
09-21 13:31:43 cat   INFO  Loaded table: galGal4.mRNA_augTMR_Evaluation
09-21 13:31:44 cat   INFO  Loaded table: rn6.mRNA_transMap_Metrics
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 765, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/fileStore.py", line 1471, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 765, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/fileStore.py", line 1471, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method CachingFileStore.__del__ of <toil.fileStore.CachingFileStore object at 0x20b3e50>> ignored
09-21 13:31:44 toil.leader WARNING The job seems to have left a log file, indicating failure: 'cgp' u/c/jobx9IR3A
09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A    INFO:toil:Running Toil version 3.17.0-585383ed5c4453b556269818571d3c7419c613b0.
09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A    WARNING:toil.resource:'JTRES_3a2fc781691739384afee22d074d5828' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A    WARNING:toil.resource:'JTRES_3a2fc781691739384afee22d074d5828' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A    
09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A    augustus: ERROR
09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A        Comparative gene prediction not possible with this compiled version. Please recompile with flag COMPGENEPRED set in common.mk.
09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A    
09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A    Traceback (most recent call last):
09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A      File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A      File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/job.py", line 1350, in _runner
09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A        returnValues = self._run(jobGraph, fileStore)
09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A      File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/job.py", line 1295, in _run
09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A        return self.run(fileStore)
09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A      File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/job.py", line 1479, in run
09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A        rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A      File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/cat/augustus_cgp.py", line 247, in cgp
09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A        tools.procOps.run_proc(cmd, stdout=stdout)
09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A      File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/procOps.py", line 60, in run_proc
09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A        pl.wait()
09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A      File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait
09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A        self.raiseIfExcept()
09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A      File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept
09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A        p.raiseIfExcept()
09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A      File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept
09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A        raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2]
09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A    ProcException: process exited 1: augustus --dbhints=1 --allow_hinted_splicesites=atac --extrinsicCfgFile=/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-26e535de-e224-4749-9f27-4454d05a21e8-e16d32a4-25b2-41b6-8d60-768dd5639220/tmpNcXn5G/42ece423-d4df-48fd-a6ad-f7e9ed42908c/tmp5eySz5.tmp --species=human --treefile=/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-26e535de-e224-4749-9f27-4454d05a21e8-e16d32a4-25b2-41b6-8d60-768dd5639220/tmpNcXn5G/42ece423-d4df-48fd-a6ad-f7e9ed42908c/tmpiXUg3a.tmp --alnfile=/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-26e535de-e224-4749-9f27-4454d05a21e8-e16d32a4-25b2-41b6-8d60-768dd5639220/tmpNcXn5G/42ece423-d4df-48fd-a6ad-f7e9ed42908c/tmpNOJc1c.tmp --dbaccess=/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-26e535de-e224-4749-9f27-4454d05a21e8-e16d32a4-25b2-41b6-8d60-768dd5639220/tmpNcXn5G/42ece423-d4df-48fd-a6ad-f7e9ed42908c/tmpJ_VdD5.tmp --speciesfilenames=/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-26e535de-e224-4749-9f27-4454d05a21e8-e16d32a4-25b2-41b6-8d60-768dd5639220/tmpNcXn5G/42ece423-d4df-48fd-a6ad-f7e9ed42908c/login-e-16.197570.3444860215.tmp --softmasking=1 --exoncands=1 --alternatives-from-evidence=0 --/CompPred/logreg=on --printOEs=1 --/CompPred/outdir=/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-26e535de-e224-4749-9f27-4454d05a21e8-e16d32a4-25b2-41b6-8d60-768dd5639220/tmpNcXn5G/42ece423-d4df-48fd-a6ad-f7e9ed42908c
09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A    ERROR:toil.worker:Exiting the worker because of a failed job on host login-e-16
09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'cgp' u/c/jobx9IR3A with ID u/c/jobx9IR3A to 0
09-21 13:31:44 toil.leader WARNING Job 'cgp' u/c/jobx9IR3A with ID u/c/jobx9IR3A is completely failed
09-21 13:31:44 cat   INFO  Evaluating transcript alignments for hg38.
09-21 13:31:44 cat   INFO  Loaded table: galGal4.mRNA_augTM_Metrics
09-21 13:31:44 cat   INFO  Loaded table: rheMac3.CDS_augTMR_Metrics
09-21 13:31:44 cat   INFO  Evaluating transcript alignments for monDom5.
09-21 13:31:44 cat   INFO  Launching homGeneMapping for transMap.
09-21 13:31:44 cat   INFO  Loaded table: rn6.CDS_augTM_Evaluation
09-21 13:31:44 cat   INFO  Loaded table: galGal4.CDS_transMap_Evaluation
09-21 13:31:45 luigi-interface INFO  [pid 196562] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done      Task: EvaluateDriverTask for rn6
09-21 13:31:45 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_70026afda7   has status   DONE
09-21 13:31:45 luigi-interface INFO  [pid 198025] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) running   ToilTask: AugustusPbDriverTask for hg38 using batchSystem singleMachine
09-21 13:31:45 cat   INFO  Loaded table: bosTau8.augTM_Hgm
09-21 13:31:45 cat   INFO  Loaded table: rheMac3.mRNA_augTMR_Evaluation
09-21 13:31:45 cat   INFO  Loaded table: hg38.mRNA_augTM_Evaluation
09-21 13:31:45 cat   INFO  Loaded table: galGal4.mRNA_transMap_Metrics
09-21 13:31:45 cat   INFO  Loaded table: bosTau8.mRNA_transMap_Metrics
09-21 13:31:45 cat   INFO  Loaded table: monDom5.mRNA_augTM_Evaluation
09-21 13:31:45 cat   INFO  Loaded table: galGal4.CDS_augTM_Evaluation
09-21 13:31:45 cat   INFO  Evaluating transcript alignments for canFam3.
09-21 13:31:45 cat   INFO  Loaded table: hg38.mRNA_augTMR_Metrics
09-21 13:31:46 cat   INFO  Loaded table: canFam3.augTM_Hgm
09-21 13:31:46 luigi-interface INFO  [pid 196573] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done      Task: EvaluateDriverTask for galGal4
09-21 13:31:46 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_9c9943f49b   has status   DONE
09-21 13:31:46 luigi-interface INFO  [pid 198080] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) running   Task: EvaluateTransMapDriverTask for bosTau8
09-21 13:31:46 cat   INFO  Loaded table: bosTau8.CDS_augTM_Evaluation
09-21 13:31:46 cat   INFO  Loaded table: rheMac3.mRNA_augTM_Metrics
09-21 13:31:46 cat   INFO  Loaded table: hg38.mRNA_transMap_Evaluation
09-21 13:31:46 cat   INFO  Loaded table: galGal4.augTM_Hgm
09-21 13:31:46 luigi-interface INFO  [pid 196580] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done      Task: EvaluateDriverTask for bosTau8
09-21 13:31:46 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_902022f711   has status   DONE
09-21 13:31:46 luigi-interface INFO  [pid 198149] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) running   Task: EvaluateTransMapDriverTask for canFam3
09-21 13:31:46 cat   INFO  Loaded table: monDom5.CDS_augTM_Metrics
09-21 13:31:46 luigi-interface INFO  [pid 196122] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done      Task: HgmDriverTask for augTM
09-21 13:31:46 luigi-interface INFO  Informed scheduler that task   HgmDriverTask_False_True_True_6a1c9d2ccf   has status   DONE
09-21 13:31:46 luigi-interface INFO  [pid 198180] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) running   Task: EvaluateTransMapDriverTask for hg38
09-21 13:31:47 cat   INFO  Loaded table: canFam3.mRNA_augTM_Evaluation
09-21 13:31:47 cat   INFO  Loaded table: hg38.CDS_transMap_Metrics
09-21 13:31:47 cat   INFO  Loaded table: rheMac3.CDS_transMap_Evaluation
09-21 13:31:47 cat   INFO  Loaded table: monDom5.mRNA_transMap_Evaluation
09-21 13:31:48 cat   INFO  Loaded table: hg38.CDS_augTM_Metrics
09-21 13:31:48 cat   INFO  Loaded table: canFam3.CDS_augTM_Metrics
09-21 13:31:48 cat   INFO  Loaded table: rheMac3.mRNA_transMap_Metrics
09-21 13:31:48 luigi-interface ERROR [pid 196561] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) failed    ToilTask: AugustusCgp using batchSystem singleMachine
Traceback (most recent call last):
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps
    task_gen = self.task.run()
  File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/cat/__init__.py", line 1446, in run
    augustus_cgp(cgp_args, toil_options)
  File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/cat/augustus_cgp.py", line 59, in augustus_cgp
    results, stdout_file_ids, param_file_id = t.restart()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 814, in restart
    return self._runMainLoop(rootJobGraph)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 242, in run
    raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
FailedJobsException
09-21 13:31:48 luigi-interface INFO  Informed scheduler that task   AugustusCgp_False_True_True_4bab51e20b   has status   FAILED
09-21 13:31:48 luigi-interface INFO  [pid 198305] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) running   Task: EvaluateTransMapDriverTask for rn6
09-21 13:31:48 cat   INFO  Loaded table: hg38.CDS_augTMR_Evaluation
09-21 13:31:48 cat   INFO  Loaded table: monDom5.CDS_transMap_Metrics
09-21 13:31:49 cat   INFO  Loaded table: canFam3.mRNA_transMap_Evaluation
09-21 13:31:49 cat   INFO  Loaded table: hg38.CDS_augTMR_Metrics
09-21 13:31:49 cat   INFO  Loaded table: rheMac3.CDS_augTM_Evaluation
09-21 13:31:49 cat   INFO  Loaded table: monDom5.mRNA_augTM_Metrics
09-21 13:31:49 luigi-interface INFO  [pid 196568] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done      Task: EvaluateDriverTask for rheMac3
09-21 13:31:49 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_8b824cb1d0   has status   DONE
09-21 13:31:50 luigi-interface INFO  [pid 198336] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) running   Task: EvaluateTransMapDriverTask for monDom5
09-21 13:31:50 cat   INFO  Loaded table: hg38.mRNA_augTMR_Evaluation
09-21 13:31:50 cat   INFO  Loaded table: canFam3.CDS_transMap_Metrics
09-21 13:31:50 luigi-interface ERROR [pid 198025] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) failed    ToilTask: AugustusPbDriverTask for hg38 using batchSystem singleMachine
Traceback (most recent call last):
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 179, in run
    missing = [dep.task_id for dep in self.task.deps() if not dep.complete()]
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/task.py", line 827, in complete
    return all(r.complete() for r in flatten(self.requires()))
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/task.py", line 827, in <genexpr>
    return all(r.complete() for r in flatten(self.requires()))
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/task.py", line 573, in complete
    return all(map(lambda output: output.exists(), outputs))
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/task.py", line 573, in <lambda>
    return all(map(lambda output: output.exists(), outputs))
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/contrib/sqla.py", line 236, in exists
    self.create_marker_table()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/contrib/sqla.py", line 257, in create_marker_table
    if not con.dialect.has_table(con, self.marker_table):
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/dialects/sqlite/base.py", line 1181, in has_table
    connection, "table_info", table_name, schema=schema)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/dialects/sqlite/base.py", line 1577, in _get_table_pragma
    cursor = connection.execute(statement)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 942, in execute
    return self._execute_text(object, multiparams, params)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1104, in _execute_text
    statement, parameters
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 509, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) database is locked [SQL: 'PRAGMA table_info("table_updates")'] (Background on this error at: http://sqlalche.me/e/e3q8)
09-21 13:31:50 luigi-interface INFO  Informed scheduler that task   AugustusPbDriverTask_False_True_True_a83bd75adf   has status   FAILED
09-21 13:31:50 luigi-interface INFO  [pid 198364] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) running   Task: EvaluateTransMapDriverTask for galGal4
09-21 13:31:50 cat   INFO  Loaded table: monDom5.CDS_transMap_Evaluation
09-21 13:31:51 cat   INFO  Loaded table: hg38.mRNA_augTM_Metrics
09-21 13:31:51 cat   INFO  Loaded table: canFam3.mRNA_augTM_Metrics
09-21 13:31:52 cat   INFO  Loaded table: hg38.CDS_transMap_Evaluation
09-21 13:31:52 cat   INFO  Loaded table: canFam3.CDS_transMap_Evaluation
09-21 13:31:52 cat   INFO  Loaded table: hg38.mRNA_transMap_Metrics
09-21 13:31:53 cat   INFO  Loaded table: monDom5.transMap_Hgm
09-21 13:31:53 cat   INFO  Loaded table: monDom5.mRNA_transMap_Metrics
09-21 13:31:53 cat   INFO  Loaded table: canFam3.mRNA_transMap_Metrics
09-21 13:31:53 cat   INFO  Loaded table: hg38.CDS_augTM_Evaluation
09-21 13:31:54 luigi-interface INFO  [pid 196585] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done      Task: EvaluateDriverTask for hg38
09-21 13:31:54 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_a83bd75adf   has status   DONE
09-21 13:31:54 luigi-interface INFO  [pid 198509] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) running   Task: EvaluateTransMapDriverTask for rheMac3
09-21 13:31:53 luigi-interface ERROR [pid 198305] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) failed    Task: EvaluateTransMapDriverTask for rn6
Traceback (most recent call last):
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 179, in run
    missing = [dep.task_id for dep in self.task.deps() if not dep.complete()]
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/task.py", line 827, in complete
    return all(r.complete() for r in flatten(self.requires()))
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/task.py", line 827, in <genexpr>
    return all(r.complete() for r in flatten(self.requires()))
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/task.py", line 573, in complete
    return all(map(lambda output: output.exists(), outputs))
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/task.py", line 573, in <lambda>
    return all(map(lambda output: output.exists(), outputs))
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/contrib/sqla.py", line 236, in exists
    self.create_marker_table()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/contrib/sqla.py", line 265, in create_marker_table
    metadata.reflect(only=[self.marker_table], bind=engine)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 3963, in reflect
    Table(name, self, **reflect_opts)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 457, in __new__
    metadata._remove_table(name, schema)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 452, in __new__
    table._init(name, metadata, *args, **kw)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 534, in _init
    include_columns, _extend_on=_extend_on)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 547, in _autoload
    _extend_on=_extend_on
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1545, in run_callable
    return callable_(self, *args, **kwargs)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 391, in reflecttable
    table, include_columns, exclude_columns, **opts)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/reflection.py", line 641, in reflecttable
    include_columns, exclude_columns, reflection_options)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/reflection.py", line 823, in _reflect_unique_constraints
    constraints = self.get_unique_constraints(table_name, schema)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/reflection.py", line 507, in get_unique_constraints
    self.bind, table_name, schema, info_cache=self.info_cache, **kw)
  File "<string>", line 2, in get_unique_constraints
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/reflection.py", line 54, in cache
    ret = fn(self, con, *args, **kw)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/dialects/sqlite/base.py", line 1449, in get_unique_constraints
    include_auto_indexes=True, **kw):
  File "<string>", line 2, in get_indexes
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/reflection.py", line 54, in cache
    ret = fn(self, con, *args, **kw)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/dialects/sqlite/base.py", line 1522, in get_indexes
    connection, "index_list", table_name, schema=schema)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/dialects/sqlite/base.py", line 1577, in _get_table_pragma
    cursor = connection.execute(statement)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 942, in execute
    return self._execute_text(object, multiparams, params)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1104, in _execute_text
    statement, parameters
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 509, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) database is locked [SQL: u'PRAGMA index_list("table_updates")'] (Background on this error at: http://sqlalche.me/e/e3q8)
09-21 13:31:54 luigi-interface INFO  Informed scheduler that task   EvaluateTransMapDriverTask_False_True_True_fc7d9960f5   has status   FAILED
09-21 13:31:54 cat   INFO  Loaded table: canFam3.CDS_augTM_Evaluation
09-21 13:31:54 cat   INFO  Evaluating transMap results for canFam3.
09-21 13:31:54 cat   INFO  Evaluating transMap results for monDom5.
09-21 13:31:54 luigi-interface INFO  [pid 196564] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done      Task: EvaluateDriverTask for canFam3
09-21 13:31:54 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_f4715e1d3c   has status   DONE
09-21 13:31:54 cat   INFO  Evaluating transMap results for bosTau8.
09-21 13:31:55 cat   INFO  Loaded table: hg38.transMap_Hgm
09-21 13:31:55 cat   INFO  Loaded table: bosTau8.TransMapEvaluation
09-21 13:31:55 cat   INFO  Loaded table: canFam3.TransMapEvaluation
09-21 13:31:56 luigi-interface INFO  [pid 198080] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done      Task: EvaluateTransMapDriverTask for bosTau8
09-21 13:31:56 luigi-interface INFO  Informed scheduler that task   EvaluateTransMapDriverTask_False_True_True_b217dbbb13   has status   DONE
09-21 13:31:56 cat   INFO  Loaded table: rheMac3.transMap_Hgm
09-21 13:31:56 cat   INFO  Loaded table: monDom5.TransMapEvaluation
09-21 13:31:56 luigi-interface INFO  [pid 198149] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done      Task: EvaluateTransMapDriverTask for canFam3
09-21 13:31:56 luigi-interface INFO  Informed scheduler that task   EvaluateTransMapDriverTask_False_True_True_b6b58a1a87   has status   DONE
09-21 13:31:56 luigi-interface INFO  [pid 198336] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done      Task: EvaluateTransMapDriverTask for monDom5
09-21 13:31:56 luigi-interface INFO  Informed scheduler that task   EvaluateTransMapDriverTask_False_True_True_0c8a652f49   has status   DONE
09-21 13:31:56 cat   INFO  Loaded table: monDom5.CDS_augTM_Evaluation
09-21 13:31:56 luigi-interface INFO  [pid 196594] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done      Task: EvaluateDriverTask for monDom5
09-21 13:31:56 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_41c87a08f0   has status   DONE
09-21 13:31:56 luigi-interface INFO  [pid 198743] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) running   Task: EvaluateTranscripts
09-21 13:31:57 cat   INFO  Loaded table: rn6.transMap_Hgm
09-21 13:31:57 cat   INFO  Loaded table: bosTau8.transMap_Hgm
09-21 13:31:59 cat   INFO  Loaded table: canFam3.transMap_Hgm
09-21 13:31:59 cat   INFO  Evaluating transMap results for rheMac3.
09-21 13:31:59 cat   INFO  Loaded table: galGal4.transMap_Hgm
09-21 13:31:59 cat   INFO  Evaluating transMap results for hg38.
09-21 13:31:59 cat   INFO  Evaluating transMap results for galGal4.
09-21 13:32:00 luigi-interface INFO  [pid 196774] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done      Task: HgmDriverTask for transMap
09-21 13:32:00 luigi-interface INFO  Informed scheduler that task   HgmDriverTask_False_True_True_66bc76859d   has status   DONE
09-21 13:32:00 luigi-interface INFO  [pid 198743] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done      Task: EvaluateTranscripts
09-21 13:32:00 luigi-interface INFO  Informed scheduler that task   EvaluateTranscripts_False_True_True_4bab51e20b   has status   DONE
09-21 13:32:00 cat   INFO  Loaded table: hg38.TransMapEvaluation
09-21 13:32:00 cat   INFO  Loaded table: galGal4.TransMapEvaluation
09-21 13:32:00 cat   INFO  Loaded table: rheMac3.TransMapEvaluation
09-21 13:32:00 luigi-interface INFO  [pid 198180] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done      Task: EvaluateTransMapDriverTask for hg38
09-21 13:32:00 luigi-interface INFO  Informed scheduler that task   EvaluateTransMapDriverTask_False_True_True_62f08feeb1   has status   DONE
09-21 13:32:01 luigi-interface INFO  [pid 198509] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done      Task: EvaluateTransMapDriverTask for rheMac3
09-21 13:32:01 luigi-interface INFO  Informed scheduler that task   EvaluateTransMapDriverTask_False_True_True_cd6004532f   has status   DONE
09-21 13:32:01 luigi-interface INFO  [pid 198364] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done      Task: EvaluateTransMapDriverTask for galGal4
09-21 13:32:01 luigi-interface INFO  Informed scheduler that task   EvaluateTransMapDriverTask_False_True_True_01193b493b   has status   DONE
09-21 13:32:01 luigi-interface INFO  Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) was stopped. Shutting down Keep-Alive thread
09-21 13:32:02 luigi-interface INFO  
===== Luigi Execution Summary =====

Scheduled 140 tasks of which:
* 25 complete ones were encountered:
    - 1 AlignTranscripts(...)
    - 1 Augustus(...)
    - 1 BuildDb(...)
    - 1 Chaining(...)
    - 1 FakePsl(...)
    ...
* 30 ran successfully:
    - 7 EvaluateDriverTask(...)
    - 6 EvaluateTransMapDriverTask(...)
    - 1 EvaluateTranscripts(...)
    - 7 FilterTransMap(...)
    - 1 Gff3ToAttrs(...)
    ...
* 3 failed:
    - 1 AugustusCgp(...)
    - 1 AugustusPbDriverTask(...)
    - 1 EvaluateTransMapDriverTask(...)
* 82 were left pending, among these:
    * 82 had failed dependencies:
        - 1 AssemblyHub(...)
        - 1 AugustusPb(...)
        - 3 AugustusTrack(...)
        - 8 BgpTrack(...)
        - 1 Consensus(...)
        ...

This progress looks :( because there were failed tasks

===== Luigi Execution Summary =====

ifiddes commented 6 years ago

You need to compile Augustus with COMPGENEPRED turned on in the common.mk file. The instructions for doing this is on the readme. On Fri, Sep 21, 2018 at 1:32 PM Francesco Cicconardi < notifications@github.com> wrote:

Ok, some progress. Now I have this:

[...] 09-21 13:31:40 cat INFO Loaded table: rn6.CDS_augTM_Metrics Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner self.run() File "/usr/lib64/python2.7/threading.py", line 765, in run self.target(*self.args, **self.__kwargs) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/fileStore.py", line 1471, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting

Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner self.run() File "/usr/lib64/python2.7/threading.py", line 765, in run self.target(*self.args, **self.__kwargs) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/fileStore.py", line 1471, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method CachingFileStore.del of <toil.fileStore.CachingFileStore object at 0x24b5e50>> ignored 09-21 13:31:40 cat INFO Loaded table: rheMac3.mRNA_augTMR_Metrics 09-21 13:31:40 toil.leader WARNING The job seems to have left a log file, indicating failure: 'cgp' u/c/jobx9IR3A 09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A INFO:toil.worker:---TOIL WORKER OUTPUT LOG--- 09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A INFO:toil:Running Toil version 3.17.0-585383ed5c4453b556269818571d3c7419c613b0. 09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A WARNING:toil.resource:'JTRES_3a2fc781691739384afee22d074d5828' may exist, but is not yet referenced by the worker (KeyError from os.environ[]). 09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A WARNING:toil.resource:'JTRES_3a2fc781691739384afee22d074d5828' may exist, but is not yet referenced by the worker (KeyError from os.environ[]). 09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A 09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A augustus: ERROR 09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A Comparative gene prediction not possible with this compiled version. Please recompile with flag COMPGENEPRED set in common.mk. 09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A 09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A Traceback (most recent call last): 09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript 09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore) 09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/job.py", line 1350, in _runner 09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A returnValues = self._run(jobGraph, fileStore) 09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/job.py", line 1295, in _run 09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A return self.run(fileStore) 09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/job.py", line 1479, in run 09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A rValue = userFunction(*((self,) + tuple(self._args)), *self._kwargs) 09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/cat/augustus_cgp.py", line 247, in cgp 09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A tools.procOps.run_proc(cmd, stdout=stdout) 09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/procOps.py", line 60, in run_proc 09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A pl.wait() 09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait 09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A self.raiseIfExcept() 09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept 09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A p.raiseIfExcept() 09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept 09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2] 09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A ProcException: process exited 1: augustus --dbhints=1 --allow_hintedsplicesites=atac --extrinsicCfgFile=/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-26e535de-e224-4749-9f27-4454d05a21e8-e16d32a4-25b2-41b6-8d60-768dd5639220/tmpGHsv8r/a31df534-97b2-4866-b429-ea3ea1ed96bd/tmpCDsQgR.tmp --species=human --treefile=/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-26e535de-e224-4749-9f27-4454d05a21e8-e16d32a4-25b2-41b6-8d60-768dd5639220/tmpGHsv8r/a31df534-97b2-4866-b429-ea3ea1ed96bd/tmpbKcJSM.tmp --alnfile=/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-26e535de-e224-4749-9f27-4454d05a21e8-e16d32a4-25b2-41b6-8d60-768dd5639220/tmpGHsv8r/a31df534-97b2-4866-b429-ea3ea1ed96bd/tmpRSh8tE.tmp --dbaccess=/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-26e535de-e224-4749-9f27-4454d05a21e8-e16d32a4-25b2-41b6-8d60-768dd5639220/tmpGHsv8r/a31df534-97b2-4866-b429-ea3ea1ed96bd/tmpjl3Ec.tmp --speciesfilenames=/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-26e535de-e224-4749-9f27-4454d05a21e8-e16d32a4-25b2-41b6-8d60-768dd5639220/tmpGHsv8r/a31df534-97b2-4866-b429-ea3ea1ed96bd/login-e-16.197146.0476613898.tmp --softmasking=1 --exoncands=1 --alternatives-from-evidence=0 --/CompPred/logreg=on --printOEs=1 --/CompPred/outdir=/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-26e535de-e224-4749-9f27-4454d05a21e8-e16d32a4-25b2-41b6-8d60-768dd5639220/tmpGHsv8r/a31df534-97b2-4866-b429-ea3ea1ed96bd 09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A ERROR:toil.worker:Exiting the worker because of a failed job on host login-e-16 09-21 13:31:40 toil.leader WARNING u/c/jobx9IR3A WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'cgp' u/c/jobx9IR3A with ID u/c/jobx9IR3A to 1 09-21 13:31:40 cat INFO Loaded table: galGal4.mRNA_augTMR_Metrics 09-21 13:31:40 cat INFO Loaded table: bosTau8.mRNA_transMap_Evaluation 09-21 13:31:40 cat INFO Loaded table: rn6.mRNA_transMap_Evaluation 09-21 13:31:41 cat INFO Loaded table: galGal4.mRNA_transMap_Evaluation 09-21 13:31:41 cat INFO Loaded table: rheMac3.mRNA_transMap_Evaluation 09-21 13:31:41 cat INFO Loaded table: bosTau8.CDS_transMap_Metrics 09-21 13:31:41 cat INFO Loaded table: galGal4.CDS_transMap_Metrics 09-21 13:31:41 cat INFO Loaded table: rn6.CDS_transMap_Metrics 09-21 13:31:42 cat INFO Loaded table: galGal4.CDS_augTM_Metrics 09-21 13:31:42 cat INFO Loaded table: rheMac3.CDS_transMap_Metrics 09-21 13:31:42 cat INFO Loaded table: galGal4.CDS_augTMR_Evaluation 09-21 13:31:42 cat INFO Loaded table: rn6.augTM_Hgm 09-21 13:31:42 cat INFO Loaded table: bosTau8.mRNA_augTM_Metrics 09-21 13:31:42 cat INFO Loaded table: rn6.mRNA_augTM_Metrics 09-21 13:31:43 cat INFO Loaded table: rheMac3.CDS_augTM_Metrics 09-21 13:31:43 cat INFO Loaded table: galGal4.CDS_augTMR_Metrics 09-21 13:31:43 cat INFO Loaded table: rn6.CDS_transMap_Evaluation 09-21 13:31:43 cat INFO Loaded table: bosTau8.CDS_transMap_Evaluation 09-21 13:31:43 cat INFO Loaded table: rheMac3.CDS_augTMR_Evaluation 09-21 13:31:43 cat INFO Loaded table: galGal4.mRNA_augTMR_Evaluation 09-21 13:31:44 cat INFO Loaded table: rn6.mRNA_transMap_Metrics Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner self.run() File "/usr/lib64/python2.7/threading.py", line 765, in run self.__target(self.args, **self.kwargs) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/fileStore.py", line 1471, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting

Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner self.run() File "/usr/lib64/python2.7/threading.py", line 765, in run self.target(*self.args, **self.__kwargs) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/fileStore.py", line 1471, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method CachingFileStore.del of <toil.fileStore.CachingFileStore object at 0x20b3e50>> ignored 09-21 13:31:44 toil.leader WARNING The job seems to have left a log file, indicating failure: 'cgp' u/c/jobx9IR3A 09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A INFO:toil.worker:---TOIL WORKER OUTPUT LOG--- 09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A INFO:toil:Running Toil version 3.17.0-585383ed5c4453b556269818571d3c7419c613b0. 09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A WARNING:toil.resource:'JTRES_3a2fc781691739384afee22d074d5828' may exist, but is not yet referenced by the worker (KeyError from os.environ[]). 09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A WARNING:toil.resource:'JTRES_3a2fc781691739384afee22d074d5828' may exist, but is not yet referenced by the worker (KeyError from os.environ[]). 09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A 09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A augustus: ERROR 09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A Comparative gene prediction not possible with this compiled version. Please recompile with flag COMPGENEPRED set in common.mk. 09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A 09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A Traceback (most recent call last): 09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript 09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore) 09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/job.py", line 1350, in _runner 09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A returnValues = self._run(jobGraph, fileStore) 09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/job.py", line 1295, in _run 09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A return self.run(fileStore) 09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/job.py", line 1479, in run 09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A rValue = userFunction(((self,) + tuple(self._args)), **self._kwargs) 09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/cat/augustus_cgp.py", line 247, in cgp 09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A tools.procOps.run_proc(cmd, stdout=stdout) 09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/procOps.py", line 60, in run_proc 09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A pl.wait() 09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait 09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A self.raiseIfExcept() 09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept 09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A p.raiseIfExcept() 09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept 09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2] 09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A ProcException: process exited 1: augustus --dbhints=1 --allow_hinted_splicesites=atac --extrinsicCfgFile=/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-26e535de-e224-4749-9f27-4454d05a21e8-e16d32a4-25b2-41b6-8d60-768dd5639220/tmpNcXn5G/42ece423-d4df-48fd-a6ad-f7e9ed42908c/tmp5eySz5.tmp --species=human --treefile=/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-26e535de-e224-4749-9f27-4454d05a21e8-e16d32a4-25b2-41b6-8d60-768dd5639220/tmpNcXn5G/42ece423-d4df-48fd-a6ad-f7e9ed42908c/tmpiXUg3a.tmp --alnfile=/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-26e535de-e224-4749-9f27-4454d05a21e8-e16d32a4-25b2-41b6-8d60-768dd5639220/tmpNcXn5G/42ece423-d4df-48fd-a6ad-f7e9ed42908c/tmpNOJc1c.tmp --dbaccess=/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-26e535de-e224-4749-9f27-4454d05a21e8-e16d32a4-25b2-41b6-8d60-768dd5639220/tmpNcXn5G/42ece423-d4df-48fd-a6ad-f7e9ed42908c/tmpJ_VdD5.tmp --speciesfilenames=/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-26e535de-e224-4749-9f27-4454d05a21e8-e16d32a4-25b2-41b6-8d60-768dd5639220/tmpNcXn5G/42ece423-d4df-48fd-a6ad-f7e9ed42908c/login-e-16.197570.3444860215.tmp --softmasking=1 --exoncands=1 --alternatives-from-evidence=0 --/CompPred/logreg=on --printOEs=1 --/CompPred/outdir=/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-26e535de-e224-4749-9f27-4454d05a21e8-e16d32a4-25b2-41b6-8d60-768dd5639220/tmpNcXn5G/42ece423-d4df-48fd-a6ad-f7e9ed42908c 09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A ERROR:toil.worker:Exiting the worker because of a failed job on host login-e-16 09-21 13:31:44 toil.leader WARNING u/c/jobx9IR3A WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'cgp' u/c/jobx9IR3A with ID u/c/jobx9IR3A to 0 09-21 13:31:44 toil.leader WARNING Job 'cgp' u/c/jobx9IR3A with ID u/c/jobx9IR3A is completely failed 09-21 13:31:44 cat INFO Evaluating transcript alignments for hg38. 09-21 13:31:44 cat INFO Loaded table: galGal4.mRNA_augTM_Metrics 09-21 13:31:44 cat INFO Loaded table: rheMac3.CDS_augTMR_Metrics 09-21 13:31:44 cat INFO Evaluating transcript alignments for monDom5. 09-21 13:31:44 cat INFO Launching homGeneMapping for transMap. 09-21 13:31:44 cat INFO Loaded table: rn6.CDS_augTM_Evaluation 09-21 13:31:44 cat INFO Loaded table: galGal4.CDS_transMap_Evaluation 09-21 13:31:45 luigi-interface INFO [pid 196562] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done Task: EvaluateDriverTask for rn6 09-21 13:31:45 luigi-interface INFO Informed scheduler that task EvaluateDriverTask_False_True_True_70026afda7 has status DONE 09-21 13:31:45 luigi-interface INFO [pid 198025] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) running ToilTask: AugustusPbDriverTask for hg38 using batchSystem singleMachine 09-21 13:31:45 cat INFO Loaded table: bosTau8.augTM_Hgm 09-21 13:31:45 cat INFO Loaded table: rheMac3.mRNA_augTMR_Evaluation 09-21 13:31:45 cat INFO Loaded table: hg38.mRNA_augTM_Evaluation 09-21 13:31:45 cat INFO Loaded table: galGal4.mRNA_transMap_Metrics 09-21 13:31:45 cat INFO Loaded table: bosTau8.mRNA_transMap_Metrics 09-21 13:31:45 cat INFO Loaded table: monDom5.mRNA_augTM_Evaluation 09-21 13:31:45 cat INFO Loaded table: galGal4.CDS_augTM_Evaluation 09-21 13:31:45 cat INFO Evaluating transcript alignments for canFam3. 09-21 13:31:45 cat INFO Loaded table: hg38.mRNA_augTMR_Metrics 09-21 13:31:46 cat INFO Loaded table: canFam3.augTM_Hgm 09-21 13:31:46 luigi-interface INFO [pid 196573] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done Task: EvaluateDriverTask for galGal4 09-21 13:31:46 luigi-interface INFO Informed scheduler that task EvaluateDriverTask_False_True_True_9c9943f49b has status DONE 09-21 13:31:46 luigi-interface INFO [pid 198080] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) running Task: EvaluateTransMapDriverTask for bosTau8 09-21 13:31:46 cat INFO Loaded table: bosTau8.CDS_augTM_Evaluation 09-21 13:31:46 cat INFO Loaded table: rheMac3.mRNA_augTM_Metrics 09-21 13:31:46 cat INFO Loaded table: hg38.mRNA_transMap_Evaluation 09-21 13:31:46 cat INFO Loaded table: galGal4.augTM_Hgm 09-21 13:31:46 luigi-interface INFO [pid 196580] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done Task: EvaluateDriverTask for bosTau8 09-21 13:31:46 luigi-interface INFO Informed scheduler that task EvaluateDriverTask_False_True_True_902022f711 has status DONE 09-21 13:31:46 luigi-interface INFO [pid 198149] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) running Task: EvaluateTransMapDriverTask for canFam3 09-21 13:31:46 cat INFO Loaded table: monDom5.CDS_augTM_Metrics 09-21 13:31:46 luigi-interface INFO [pid 196122] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done Task: HgmDriverTask for augTM 09-21 13:31:46 luigi-interface INFO Informed scheduler that task HgmDriverTask_False_True_True_6a1c9d2ccf has status DONE 09-21 13:31:46 luigi-interface INFO [pid 198180] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) running Task: EvaluateTransMapDriverTask for hg38 09-21 13:31:47 cat INFO Loaded table: canFam3.mRNA_augTM_Evaluation 09-21 13:31:47 cat INFO Loaded table: hg38.CDS_transMap_Metrics 09-21 13:31:47 cat INFO Loaded table: rheMac3.CDS_transMap_Evaluation 09-21 13:31:47 cat INFO Loaded table: monDom5.mRNA_transMap_Evaluation 09-21 13:31:48 cat INFO Loaded table: hg38.CDS_augTM_Metrics 09-21 13:31:48 cat INFO Loaded table: canFam3.CDS_augTM_Metrics 09-21 13:31:48 cat INFO Loaded table: rheMac3.mRNA_transMap_Metrics 09-21 13:31:48 luigi-interface ERROR [pid 196561] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) failed ToilTask: AugustusCgp using batchSystem singleMachine Traceback (most recent call last): File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run new_deps = self._run_get_new_deps() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps task_gen = self.task.run() File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/cat/init.py", line 1446, in run augustus_cgp(cgp_args, toil_options) File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/cat/augustus_cgp.py", line 59, in augustus_cgp results, stdout_file_ids, param_file_id = t.restart() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 814, in restart return self._runMainLoop(rootJobGraph) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop jobCache=self._jobCache).run() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 242, in run raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore) FailedJobsException 09-21 13:31:48 luigi-interface INFO Informed scheduler that task AugustusCgp_False_True_True_4bab51e20b has status FAILED 09-21 13:31:48 luigi-interface INFO [pid 198305] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) running Task: EvaluateTransMapDriverTask for rn6 09-21 13:31:48 cat INFO Loaded table: hg38.CDS_augTMR_Evaluation 09-21 13:31:48 cat INFO Loaded table: monDom5.CDS_transMap_Metrics 09-21 13:31:49 cat INFO Loaded table: canFam3.mRNA_transMap_Evaluation 09-21 13:31:49 cat INFO Loaded table: hg38.CDS_augTMR_Metrics 09-21 13:31:49 cat INFO Loaded table: rheMac3.CDS_augTM_Evaluation 09-21 13:31:49 cat INFO Loaded table: monDom5.mRNA_augTM_Metrics 09-21 13:31:49 luigi-interface INFO [pid 196568] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done Task: EvaluateDriverTask for rheMac3 09-21 13:31:49 luigi-interface INFO Informed scheduler that task EvaluateDriverTask_False_True_True_8b824cb1d0 has status DONE 09-21 13:31:50 luigi-interface INFO [pid 198336] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) running Task: EvaluateTransMapDriverTask for monDom5 09-21 13:31:50 cat INFO Loaded table: hg38.mRNA_augTMR_Evaluation 09-21 13:31:50 cat INFO Loaded table: canFam3.CDS_transMap_Metrics 09-21 13:31:50 luigi-interface ERROR [pid 198025] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) failed ToilTask: AugustusPbDriverTask for hg38 using batchSystem singleMachine Traceback (most recent call last): File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 179, in run missing = [dep.task_id for dep in self.task.deps() if not dep.complete()] File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/task.py", line 827, in complete return all(r.complete() for r in flatten(self.requires())) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/task.py", line 827, in return all(r.complete() for r in flatten(self.requires())) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/task.py", line 573, in complete return all(map(lambda output: output.exists(), outputs)) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/task.py", line 573, in return all(map(lambda output: output.exists(), outputs)) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/contrib/sqla.py", line 236, in exists self.create_marker_table() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/contrib/sqla.py", line 257, in create_marker_table if not con.dialect.has_table(con, self.marker_table): File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/dialects/sqlite/base.py", line 1181, in has_table connection, "table_info", table_name, schema=schema) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/dialects/sqlite/base.py", line 1577, in _get_table_pragma cursor = connection.execute(statement) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 942, in execute return self._execute_text(object, multiparams, params) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1104, in _execute_text statement, parameters File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context context) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception exc_info File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause reraise(type(exception), exception, tb=exc_tb, cause=cause) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context context) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 509, in do_execute cursor.execute(statement, parameters) OperationalError: (sqlite3.OperationalError) database is locked [SQL: 'PRAGMA table_info("table_updates")'] (Background on this error at: http://sqlalche.me/e/e3q8) 09-21 13:31:50 luigi-interface INFO Informed scheduler that task AugustusPbDriverTask_False_True_True_a83bd75adf has status FAILED 09-21 13:31:50 luigi-interface INFO [pid 198364] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) running Task: EvaluateTransMapDriverTask for galGal4 09-21 13:31:50 cat INFO Loaded table: monDom5.CDS_transMap_Evaluation 09-21 13:31:51 cat INFO Loaded table: hg38.mRNA_augTM_Metrics 09-21 13:31:51 cat INFO Loaded table: canFam3.mRNA_augTM_Metrics 09-21 13:31:52 cat INFO Loaded table: hg38.CDS_transMap_Evaluation 09-21 13:31:52 cat INFO Loaded table: canFam3.CDS_transMap_Evaluation 09-21 13:31:52 cat INFO Loaded table: hg38.mRNA_transMap_Metrics 09-21 13:31:53 cat INFO Loaded table: monDom5.transMap_Hgm 09-21 13:31:53 cat INFO Loaded table: monDom5.mRNA_transMap_Metrics 09-21 13:31:53 cat INFO Loaded table: canFam3.mRNA_transMap_Metrics 09-21 13:31:53 cat INFO Loaded table: hg38.CDS_augTM_Evaluation 09-21 13:31:54 luigi-interface INFO [pid 196585] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done Task: EvaluateDriverTask for hg38 09-21 13:31:54 luigi-interface INFO Informed scheduler that task EvaluateDriverTask_False_True_True_a83bd75adf has status DONE 09-21 13:31:54 luigi-interface INFO [pid 198509] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) running Task: EvaluateTransMapDriverTask for rheMac3 09-21 13:31:53 luigi-interface ERROR [pid 198305] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) failed Task: EvaluateTransMapDriverTask for rn6 Traceback (most recent call last): File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 179, in run missing = [dep.task_id for dep in self.task.deps() if not dep.complete()] File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/task.py", line 827, in complete return all(r.complete() for r in flatten(self.requires())) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/task.py", line 827, in return all(r.complete() for r in flatten(self.requires())) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/task.py", line 573, in complete return all(map(lambda output: output.exists(), outputs)) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/task.py", line 573, in return all(map(lambda output: output.exists(), outputs)) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/contrib/sqla.py", line 236, in exists self.create_marker_table() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/contrib/sqla.py", line 265, in create_marker_table metadata.reflect(only=[self.marker_table], bind=engine) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 3963, in reflect Table(name, self, **reflect_opts) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 457, in new metadata._remove_table(name, schema) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in exit compat.reraise(exc_type, exc_value, exc_tb) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 452, in new table._init(name, metadata, args, kw) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 534, in _init include_columns, _extend_on=_extend_on) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/sql/schema.py", line 547, in _autoload _extend_on=_extend_on File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1545, in runcallable return callable(self, *args, kwargs) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 391, in reflecttable table, include_columns, exclude_columns, opts) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/reflection.py", line 641, in reflecttable include_columns, exclude_columns, reflection_options) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/reflection.py", line 823, in _reflect_unique_constraints constraints = self.get_unique_constraints(table_name, schema) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/reflection.py", line 507, in get_unique_constraints self.bind, table_name, schema, info_cache=self.info_cache, *kw) File "", line 2, in get_unique_constraints File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/reflection.py", line 54, in cache ret = fn(self, con, args, kw) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/dialects/sqlite/base.py", line 1449, in get_unique_constraints include_auto_indexes=True, *kw): File "", line 2, in get_indexes File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/reflection.py", line 54, in cache ret = fn(self, con, args, **kw) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/dialects/sqlite/base.py", line 1522, in get_indexes connection, "index_list", table_name, schema=schema) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/dialects/sqlite/base.py", line 1577, in _get_table_pragma cursor = connection.execute(statement) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 942, in execute return self._execute_text(object, multiparams, params) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1104, in _execute_text statement, parameters File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context context) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception exc_info File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause reraise(type(exception), exception, tb=exc_tb, cause=cause) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context context) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 509, in do_execute cursor.execute(statement, parameters) OperationalError: (sqlite3.OperationalError) database is locked [SQL: u'PRAGMA index_list("table_updates")'] (Background on this error at: http://sqlalche.me/e/e3q8) 09-21 13:31:54 luigi-interface INFO Informed scheduler that task EvaluateTransMapDriverTask_False_True_True_fc7d9960f5 has status FAILED 09-21 13:31:54 cat INFO Loaded table: canFam3.CDS_augTM_Evaluation 09-21 13:31:54 cat INFO Evaluating transMap results for canFam3. 09-21 13:31:54 cat INFO Evaluating transMap results for monDom5. 09-21 13:31:54 luigi-interface INFO [pid 196564] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done Task: EvaluateDriverTask for canFam3 09-21 13:31:54 luigi-interface INFO Informed scheduler that task EvaluateDriverTask_False_True_True_f4715e1d3c has status DONE 09-21 13:31:54 cat INFO Evaluating transMap results for bosTau8. 09-21 13:31:55 cat INFO Loaded table: hg38.transMap_Hgm 09-21 13:31:55 cat INFO Loaded table: bosTau8.TransMapEvaluation 09-21 13:31:55 cat INFO Loaded table: canFam3.TransMapEvaluation 09-21 13:31:56 luigi-interface INFO [pid 198080] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done Task: EvaluateTransMapDriverTask for bosTau8 09-21 13:31:56 luigi-interface INFO Informed scheduler that task EvaluateTransMapDriverTask_False_True_True_b217dbbb13 has status DONE 09-21 13:31:56 cat INFO Loaded table: rheMac3.transMap_Hgm 09-21 13:31:56 cat INFO Loaded table: monDom5.TransMapEvaluation 09-21 13:31:56 luigi-interface INFO [pid 198149] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done Task: EvaluateTransMapDriverTask for canFam3 09-21 13:31:56 luigi-interface INFO Informed scheduler that task EvaluateTransMapDriverTask_False_True_True_b6b58a1a87 has status DONE 09-21 13:31:56 luigi-interface INFO [pid 198336] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done Task: EvaluateTransMapDriverTask for monDom5 09-21 13:31:56 luigi-interface INFO Informed scheduler that task EvaluateTransMapDriverTask_False_True_True_0c8a652f49 has status DONE 09-21 13:31:56 cat INFO Loaded table: monDom5.CDS_augTM_Evaluation 09-21 13:31:56 luigi-interface INFO [pid 196594] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done Task: EvaluateDriverTask for monDom5 09-21 13:31:56 luigi-interface INFO Informed scheduler that task EvaluateDriverTask_False_True_True_41c87a08f0 has status DONE 09-21 13:31:56 luigi-interface INFO [pid 198743] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) running Task: EvaluateTranscripts 09-21 13:31:57 cat INFO Loaded table: rn6.transMap_Hgm 09-21 13:31:57 cat INFO Loaded table: bosTau8.transMap_Hgm 09-21 13:31:59 cat INFO Loaded table: canFam3.transMap_Hgm 09-21 13:31:59 cat INFO Evaluating transMap results for rheMac3. 09-21 13:31:59 cat INFO Loaded table: galGal4.transMap_Hgm 09-21 13:31:59 cat INFO Evaluating transMap results for hg38. 09-21 13:31:59 cat INFO Evaluating transMap results for galGal4. 09-21 13:32:00 luigi-interface INFO [pid 196774] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done Task: HgmDriverTask for transMap 09-21 13:32:00 luigi-interface INFO Informed scheduler that task HgmDriverTask_False_True_True_66bc76859d has status DONE 09-21 13:32:00 luigi-interface INFO [pid 198743] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done Task: EvaluateTranscripts 09-21 13:32:00 luigi-interface INFO Informed scheduler that task EvaluateTranscripts_False_True_True_4bab51e20b has status DONE 09-21 13:32:00 cat INFO Loaded table: hg38.TransMapEvaluation 09-21 13:32:00 cat INFO Loaded table: galGal4.TransMapEvaluation 09-21 13:32:00 cat INFO Loaded table: rheMac3.TransMapEvaluation 09-21 13:32:00 luigi-interface INFO [pid 198180] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done Task: EvaluateTransMapDriverTask for hg38 09-21 13:32:00 luigi-interface INFO Informed scheduler that task EvaluateTransMapDriverTask_False_True_True_62f08feeb1 has status DONE 09-21 13:32:01 luigi-interface INFO [pid 198509] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done Task: EvaluateTransMapDriverTask for rheMac3 09-21 13:32:01 luigi-interface INFO Informed scheduler that task EvaluateTransMapDriverTask_False_True_True_cd6004532f has status DONE 09-21 13:32:01 luigi-interface INFO [pid 198364] Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) done Task: EvaluateTransMapDriverTask for galGal4 09-21 13:32:01 luigi-interface INFO Informed scheduler that task EvaluateTransMapDriverTask_False_True_True_01193b493b has status DONE 09-21 13:32:01 luigi-interface INFO Worker Worker(salt=412676406, workers=10, host=login-e-16, username=fc464, pid=195443) was stopped. Shutting down Keep-Alive thread 09-21 13:32:02 luigi-interface INFO ===== Luigi Execution Summary =====

Scheduled 140 tasks of which:

25 complete ones were encountered:

1 AlignTranscripts(...)

1 Augustus(...)

1 BuildDb(...)

1 Chaining(...)

1 FakePsl(...) ...

30 ran successfully:

7 EvaluateDriverTask(...)

6 EvaluateTransMapDriverTask(...)

1 EvaluateTranscripts(...)

7 FilterTransMap(...)

1 Gff3ToAttrs(...) ...

3 failed:

1 AugustusCgp(...)

1 AugustusPbDriverTask(...)

1 EvaluateTransMapDriverTask(...)

82 were left pending, among these:

82 had failed dependencies:

1 AssemblyHub(...)

1 AugustusPb(...)

3 AugustusTrack(...)

8 BgpTrack(...)

1 Consensus(...) ...

This progress looks :( because there were failed tasks

===== Luigi Execution Summary =====

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/ComparativeGenomicsToolkit/Comparative-Annotation-Toolkit/issues/111#issuecomment-423515551, or mute the thread https://github.com/notifications/unsubscribe-auth/AHdLXUBaDugnGxmWZLQn4tPsKJ17gYWHks5udNx4gaJpZM4WyEUo .

-- Ian Fiddes, PhD

francicco commented 6 years ago

Actually I don't find anything related to COMPGENEPRED in the README.md. But it looks like I need the lp_solve libraries...

lassancejm commented 6 years ago

Info are in README-cgp.txt (https://github.com/Gaius-Augustus/Augustus/blob/master/README-cgp.txt)

open the file common.mk with a text editor and uncomment the following line to enable comparative gene prediction

#COMGENEPRED = true

ifiddes commented 6 years ago

I also provide an overview of how to do this in the CAT readme. Let me know if there are any details missing. We also now package the required binaries in a Docker container if that works for you. On Fri, Sep 21, 2018 at 3:54 PM lassancejm notifications@github.com wrote:

Info are in README-cgp.txt ( https://github.com/Gaius-Augustus/Augustus/blob/master/README-cgp.txt)

open the file common.mk with a text editor and uncomment the following line to enable comparative gene prediction

COMGENEPRED = true
—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
<https://github.com/ComparativeGenomicsToolkit/Comparative-Annotation-Toolkit/issues/111#issuecomment-423560694>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AHdLXQ_NVGIChr90OgzoqxR1RoMNViOBks5udP2agaJpZM4WyEUo>
.

-- 
Ian Fiddes, PhD

francicco commented 6 years ago

I guess I fixed other dependencies. But still

===== Luigi Execution Summary =====

Scheduled 162 tasks of which:
* 7 complete ones were encountered:
    - 1 FakePsl(...)
    - 1 FlatTranscriptFasta(...)
    - 1 GenomeFiles(...)
    - 1 Gff3ToGenePred(...)
    - 1 TranscriptBed(...)
    ...
* 72 ran successfully:
    - 7 AlignTranscriptDriverTask(...)
    - 1 AlignTranscripts(...)
    - 1 Augustus(...)
    - 7 AugustusDriverTask(...)
    - 1 BuildDb(...)
    ...
* 2 failed:
    - 1 AugustusCgp(...)
    - 1 AugustusPbDriverTask(...)
* 81 were left pending, among these:
    * 81 had failed dependencies:
        - 1 AssemblyHub(...)
        - 1 AugustusPb(...)
        - 3 AugustusTrack(...)
        - 8 BgpTrack(...)
        - 1 Consensus(...)
        ...

This progress looks :( because there were failed tasks

===== Luigi Execution Summary =====

Cat.log

Help please!!!

ifiddes commented 6 years ago

What version of augustus are you using? In order for this to work, you need augustus v3.3.1 or later. I would recommend working from the latest source https://github.com/Gaius-Augustus/Augustus. It seems like CGP is crashing in the end of the training step, suggesting that your version of CGP is producing weird outputs. The output format structure changed around mid

And it seems like PB is failing in the joingenes step, which suggests that possibly your joingenes version is too old as well.

If Docker is an option for you, we have packaged versions of these tools that should work. Otherwise, I can try to add better error handling to these external program calls so that we can start to try and figure out what is broken.

On Sat, Sep 22, 2018 at 4:19 AM Francesco Cicconardi < notifications@github.com> wrote:

I guess I fixed other dependencies. But still

===== Luigi Execution Summary =====

Scheduled 162 tasks of which:

7 complete ones were encountered:

1 FakePsl(...)

1 FlatTranscriptFasta(...)

1 GenomeFiles(...)

1 Gff3ToGenePred(...)

1 TranscriptBed(...) ...

72 ran successfully:

7 AlignTranscriptDriverTask(...)

1 AlignTranscripts(...)

1 Augustus(...)

7 AugustusDriverTask(...)

1 BuildDb(...) ...

2 failed:

1 AugustusCgp(...)

1 AugustusPbDriverTask(...)

81 were left pending, among these:

81 had failed dependencies:

1 AssemblyHub(...)

1 AugustusPb(...)

3 AugustusTrack(...)

8 BgpTrack(...)

1 Consensus(...) ...

This progress looks :( because there were failed tasks

===== Luigi Execution Summary =====

Cat.log https://github.com/ComparativeGenomicsToolkit/Comparative-Annotation-Toolkit/files/2407723/Cat.log

Help please!!!

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/ComparativeGenomicsToolkit/Comparative-Annotation-Toolkit/issues/111#issuecomment-423737207, or mute the thread https://github.com/notifications/unsubscribe-auth/AHdLXe_MCLEO1Iith9xcgOlAMDOPfAqEks5udhzcgaJpZM4WyEUo .

-- Ian Fiddes, PhD

francicco commented 6 years ago

Thanks for the reply. Sadly Docker is not an option since it's not supported in the cluster, they preferred Singularity. Tomorrow I give another try with the version of Augustus you're suggesting. Thanks again for the support. I also hope that Cactus works, because so far I'm having a lot of problems with it too!!!

I'll let you know tomorrow, thanks again!!! F

ifiddes commented 6 years ago

No worries, we are well aware that these tools are quite a pain to set up and get running. That is why we put effort into the Docker containers and cloud integration, but I am also well aware that this is not an option for a lot of users. If using the repo version of augustus still fails, then we can try to dig in more and replicate the problem.

On Sun, Sep 23, 2018 at 6:51 AM Francesco Cicconardi < notifications@github.com> wrote:

Thanks for the reply. Sadly Docker is not an option since it's not supported in the cluster, they preferred Singularity. Tomorrow I give another try with the version of Augustus you're suggesting. Thanks again for the support. I also hope that Cactus works, because so far I'm having a lot of problems with it too!!!

I'll let you know tomorrow, thanks again!!! F

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/ComparativeGenomicsToolkit/Comparative-Annotation-Toolkit/issues/111#issuecomment-423818060, or mute the thread https://github.com/notifications/unsubscribe-auth/AHdLXcfv8Mrpc5aJNg954xNtCdFtCRgzks5ud5HcgaJpZM4WyEUo .

-- Ian Fiddes, PhD

francicco commented 6 years ago

Thank you very much!!!

francicco commented 6 years ago

Recompiled Augustus, and in doing that I fixed a dependence, but there still work to do: It looks like there's some problem with SQLite.

===== Luigi Execution Summary =====

Scheduled 139 tasks of which:
* 26 complete ones were encountered:
    - 1 AlignTranscripts(...)
    - 1 Augustus(...)
    - 1 AugustusPb(...)
    - 1 BuildDb(...)
    - 1 Chaining(...)
    ...
* 30 ran successfully:
    - 7 EvaluateDriverTask(...)
    - 4 EvaluateTransMapDriverTask(...)
    - 1 EvaluateTranscripts(...)
    - 7 FilterTransMap(...)
    - 1 FindDenovoParents(...)
    ...
* 4 failed:
    - 1 AugustusCgp(...)
    - 3 EvaluateTransMapDriverTask(...)
* 79 were left pending, among these:
    * 79 had failed dependencies:
        - 1 AssemblyHub(...)
        - 3 AugustusTrack(...)
        - 8 BgpTrack(...)
        - 1 Consensus(...)
        - 7 ConsensusDriverTask(...)
        ...

This progress looks :( because there were failed tasks

===== Luigi Execution Summary =====

Cat.Error.log

ifiddes-10x-zz commented 6 years ago

Interesting. There are two errors here -- the first is related to the SQL database that keeps track of run metrics. I have never seen that before. That shouldn't actually effect the results, and wouldn't happen on a real dataset (because concurrent attempts to get a lock on the DB would be rare).

The second is problems with CGP still. It seems that the cat step that combines the training values still fails. I have two things:

1) Can you run the pipeline with the flag --cgp-param=augustus_cfgs/log_reg_parameters_default.cfg? This will use the pre-trained default parameters. 2) I have made a new branch test-issue-111 that will check if the files exist. After you try #1, can you try the default test parameters when on this branch?

francicco commented 6 years ago

Hi @ifiddes-10x,

I tried using --cgp-param=augustus_cfgs/log_reg_parameters_default.cfg, but luigi: error: unrecognized arguments: --cgp-params=augustus_cfgs/log_reg_parameters_default.cfg

About the second point, how do I use the branch? Never done it before.

Thanks F

ifiddes commented 6 years ago

You need to use --cgp-param, not --cgp-params. I made a typo in the original post, sorry.

For the second point, you need to go to the repo and then do git pull && git checkout test-issue-111. Then try it again. You should do both of these on a clean run.

On Mon, Sep 24, 2018 at 1:32 PM Francesco Cicconardi < notifications@github.com> wrote:

Hi @ifiddes-10x https://github.com/ifiddes-10x,

I tried using --cgp-param=augustus_cfgs/log_reg_parameters_default.cfg, but luigi: error: unrecognized arguments: --cgp-params=augustus_cfgs/log_reg_parameters_default.cfg

About the second point, how do I use the branch? Never done it before.

Thanks F

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/ComparativeGenomicsToolkit/Comparative-Annotation-Toolkit/issues/111#issuecomment-424114646, or mute the thread https://github.com/notifications/unsubscribe-auth/AHdLXfCK6KO-eJ4En609D4_rjbC7BJdxks5ueUFHgaJpZM4WyEUo .

-- Ian Fiddes, PhD

francicco commented 6 years ago

I copied and pasted your script! :D F

francicco commented 6 years ago

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 765, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/fileStore.py", line 1471, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 765, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/fileStore.py", line 1471, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method CachingFileStore.__del__ of <toil.fileStore.CachingFileStore object at 0x2343e50>> ignored
09-24 21:47:28 toil.leader WARNING The job seems to have left a log file, indicating failure: 'cgp' U/f/jobD_xTPj
09-24 21:47:28 toil.leader WARNING U/f/jobD_xTPj    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
09-24 21:47:28 toil.leader WARNING U/f/jobD_xTPj    INFO:toil:Running Toil version 3.17.0-585383ed5c4453b556269818571d3c7419c613b0.
09-24 21:47:28 toil.leader WARNING U/f/jobD_xTPj    WARNING:toil.resource:'JTRES_3a2fc781691739384afee22d074d5828' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
09-24 21:47:28 toil.leader WARNING U/f/jobD_xTPj    WARNING:toil.resource:'JTRES_3a2fc781691739384afee22d074d5828' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
09-24 21:47:28 toil.leader WARNING U/f/jobD_xTPj    cat: /rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-a51496bc-94c5-4b2a-9350-ad6412a98fca-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd/tmpgGeYFY/606f18ea-b235-4a69-bec7-b1de4f99eee0/mm10.sampled_GFs.gff: No such file or directory
09-24 21:47:28 toil.leader WARNING U/f/jobD_xTPj    Traceback (most recent call last):
09-24 21:47:28 toil.leader WARNING U/f/jobD_xTPj      File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
09-24 21:47:28 toil.leader WARNING U/f/jobD_xTPj        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
09-24 21:47:28 toil.leader WARNING U/f/jobD_xTPj      File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/job.py", line 1350, in _runner
09-24 21:47:28 toil.leader WARNING U/f/jobD_xTPj        returnValues = self._run(jobGraph, fileStore)
09-24 21:47:28 toil.leader WARNING U/f/jobD_xTPj      File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/job.py", line 1295, in _run
09-24 21:47:28 toil.leader WARNING U/f/jobD_xTPj        return self.run(fileStore)
09-24 21:47:28 toil.leader WARNING U/f/jobD_xTPj      File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/job.py", line 1479, in run
09-24 21:47:28 toil.leader WARNING U/f/jobD_xTPj        rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
09-24 21:47:28 toil.leader WARNING U/f/jobD_xTPj      File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/cat/augustus_cgp.py", line 253, in cgp
09-24 21:47:28 toil.leader WARNING U/f/jobD_xTPj        tools.procOps.run_proc(cmd, stdout=combined_file)
09-24 21:47:28 toil.leader WARNING U/f/jobD_xTPj      File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/procOps.py", line 60, in run_proc
09-24 21:47:28 toil.leader WARNING U/f/jobD_xTPj        pl.wait()
09-24 21:47:28 toil.leader WARNING U/f/jobD_xTPj      File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait
09-24 21:47:28 toil.leader WARNING U/f/jobD_xTPj        self.raiseIfExcept()
09-24 21:47:28 toil.leader WARNING U/f/jobD_xTPj      File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept
09-24 21:47:28 toil.leader WARNING U/f/jobD_xTPj        p.raiseIfExcept()
09-24 21:47:28 toil.leader WARNING U/f/jobD_xTPj      File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept
09-24 21:47:28 toil.leader WARNING U/f/jobD_xTPj        raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2]
09-24 21:47:28 toil.leader WARNING U/f/jobD_xTPj    ProcException: process exited 1: cat /rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-a51496bc-94c5-4b2a-9350-ad6412a98fca-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd/tmpgGeYFY/606f18ea-b235-4a69-bec7-b1de4f99eee0/mm10.sampled_GFs.gff /rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-a51496bc-94c5-4b2a-9350-ad6412a98fca-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd/tmpgGeYFY/606f18ea-b235-4a69-bec7-b1de4f99eee0/exonCands.mm10.gff3 /rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-a51496bc-94c5-4b2a-9350-ad6412a98fca-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd/tmpgGeYFY/606f18ea-b235-4a69-bec7-b1de4f99eee0/orthoExons.mm10.gff3
09-24 21:47:28 toil.leader WARNING U/f/jobD_xTPj    ERROR:toil.worker:Exiting the worker because of a failed job on host login-e-14
09-24 21:47:28 toil.leader WARNING U/f/jobD_xTPj    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'cgp' U/f/jobD_xTPj with ID U/f/jobD_xTPj to 1

And it's still running F

francicco commented 6 years ago

===== Luigi Execution Summary =====

Scheduled 139 tasks of which:
* 26 complete ones were encountered:
    - 1 AlignTranscripts(...)
    - 1 Augustus(...)
    - 1 AugustusPb(...)
    - 1 BuildDb(...)
    - 1 Chaining(...)
    ...
* 33 ran successfully:
    - 7 EvaluateDriverTask(...)
    - 1 EvaluateTransMap(...)
    - 7 EvaluateTransMapDriverTask(...)
    - 1 EvaluateTranscripts(...)
    - 7 FilterTransMap(...)
    ...
* 2 failed:
    - 1 AugustusCgp(...)
    - 1 HgmDriverTask(...)
* 78 were left pending, among these:
    * 78 had failed dependencies:
        - 1 AssemblyHub(...)
        - 3 AugustusTrack(...)
        - 8 BgpTrack(...)
        - 1 Consensus(...)
        - 7 ConsensusDriverTask(...)
        ...

This progress looks :( because there were failed tasks

===== Luigi Execution Summary =====

This is the full log

Cat.log

ifiddes-10x-zz commented 6 years ago

The key line is 09-24 21:47:28 toil.leader WARNING U/f/jobD_xTPj cat: /rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-a51496bc-94c5-4b2a-9350-ad6412a98fca-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd/tmpgGeYFY/606f18ea-b235-4a69-bec7-b1de4f99eee0/mm10.sampled_GFs.gff: No such file or directory. Somehow the file mm10.sampled_GFs.gff isn't being generated, despite being in training mode. I don't know why this would be the case.

Is this on my branch? It should have crashed before that, since I was checking for these files existing. I have added a new commit to that branch. Try it again please.

However, this is all still suggesting that something is wrong with your augustus. If you are now working with the absolute latest on their github, it may be that the commands changed again. Looking at the work @cnk113 did, it seems like the Docker version points to commit e2dff65. You can copy the code that Docker is using to compile augustus:

https://github.com/ComparativeGenomicsToolkit/Comparative-Annotation-Toolkit/blob/master/Dockerfile#L24-L30

francicco commented 6 years ago

Sorry I didn't get it. Do I have to re compile Augustus? F

ifiddes commented 6 years ago

Yes, I think your best bet is to recompile augustus on that commit. Clone a clean augustus repository and copy the commands in that link:

git clone https://github.com/Gaius-Augustus/Augustus augustus cd augustus && git reset --hard e2dff65 echo 'COMPGENEPRED = true' >> augustus/common.mk echo 'SQLITE = true' >> augustus/common.mk cd augustus/auxprogs/homGeneMapping/src && sed 's/# BOOST = true/BOOST = true/g' -i Makefile && sed 's/# SQLITE = true/SQLITE = true/g' -i Makefile cd augustus && make

Then also make sure that your newly compiled augustus binary is on your path. the command which augustus should point to the new one.

On Mon, Sep 24, 2018 at 2:04 PM Francesco Cicconardi < notifications@github.com> wrote:

Sorry I didn't get it. Do I have to re compile Augustus? F

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/ComparativeGenomicsToolkit/Comparative-Annotation-Toolkit/issues/111#issuecomment-424124333, or mute the thread https://github.com/notifications/unsubscribe-auth/AHdLXfhbvgaJ6HMhvegnKr9LcORTGq9Yks5ueUjVgaJpZM4WyEUo .

-- Ian Fiddes, PhD

francicco commented 6 years ago

So, I'm recompilling Augustus. The only thing I'm doing is modify the paths to needed libraries. Namely lp_solve, COLAMD. Are specific versions I need to use? F

ifiddes commented 6 years ago

No, those shouldn't matter. The only thing that matters is the exact augustus commands. The error you are seeing is suggesting that augustus is not producing the training files properly. I am not sure if they changed the way it works or what, I need to talk to the augustus developers and see if I can reproduce this myself.

On Mon, Sep 24, 2018 at 2:14 PM Francesco Cicconardi < notifications@github.com> wrote:

So, I'm recompilling Augustus. The only thing I'm doing is modify the paths to needed libraries. Namely lp_solve, COLAMD. Are specific versions I need to use? F

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/ComparativeGenomicsToolkit/Comparative-Annotation-Toolkit/issues/111#issuecomment-424127294, or mute the thread https://github.com/notifications/unsubscribe-auth/AHdLXenfBskC2QO-dryDWK7Q_r7uHXDFks5ueUsegaJpZM4WyEUo .

-- Ian Fiddes, PhD

francicco commented 6 years ago

I'm re running the test....

francicco commented 6 years ago

Btw, this whole CAT looks extremely complicated. You should organise workshops to teach people how to use it and its potentials, Cactus included!!! F

(still running with no errors, so far)

francicco commented 6 years ago

Same error:

09-24 22:26:44 toil.leader WARNING U/f/jobD_xTPj    ProcException: process exited 1: cat /rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-a51496bc-94c5-4b2a-9350-ad6412a98fca-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd/tmp62VmGR/15435c0d-6a44-4abd-8425-9f4cdcb70602/mm10.sampled_GFs.gff /rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-a51496bc-94c5-4b2a-9350-ad6412a98fca-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd/tmp62VmGR/15435c0d-6a44-4abd-8425-9f4cdcb70602/exonCands.mm10.gff3 /rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-a51496bc-94c5-4b2a-9350-ad6412a98fca-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd/tmp62VmGR/15435c0d-6a44-4abd-8425-9f4cdcb70602/orthoExons.mm10.gff3
09-24 22:26:44 toil.leader WARNING U/f/jobD_xTPj    ERROR:toil.worker:Exiting the worker because of a failed job on host login-e-14

ifiddes commented 6 years ago

Hmm, this is weird. Ok, can you try setting --cleanWorkDir=never, and then once it runs, go look in the directory? You will need to look at the log to find it, because the path changes each time. For example, in the log you just sent, the directory is /rds/project/shm37/rds-shm37- helixmbodyw/Comparative-Annotation-Toolkit/toil-a51496bc-94c5-4b2a-9350- ad6412a98fca-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd/ tmp62VmGR/15435c0d-6a44-4abd-8425-9f4cdcb70602/.

Go into the new version of that directory after the run, and take a look. What do the files look like?

On Mon, Sep 24, 2018 at 2:29 PM Francesco Cicconardi < notifications@github.com> wrote:

Same error:

09-24 22:26:44 toil.leader WARNING U/f/jobD_xTPj ProcException: process exited 1: cat /rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-a51496bc-94c5-4b2a-9350-ad6412a98fca-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd/tmp62VmGR/15435c0d-6a44-4abd-8425-9f4cdcb70602/mm10.sampled_GFs.gff /rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-a51496bc-94c5-4b2a-9350-ad6412a98fca-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd/tmp62VmGR/15435c0d-6a44-4abd-8425-9f4cdcb70602/exonCands.mm10.gff3 /rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-a51496bc-94c5-4b2a-9350-ad6412a98fca-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd/tmp62VmGR/15435c0d-6a44-4abd-8425-9f4cdcb70602/orthoExons.mm10.gff3 09-24 22:26:44 toil.leader WARNING U/f/jobD_xTPj ERROR:toil.worker:Exiting the worker because of a failed job on host login-e-14

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/ComparativeGenomicsToolkit/Comparative-Annotation-Toolkit/issues/111#issuecomment-424131940, or mute the thread https://github.com/notifications/unsubscribe-auth/AHdLXRfez1tCjaNdSMm_kYZXXgVILWtEks5ueU6ygaJpZM4WyEUo .

-- Ian Fiddes, PhD

francicco commented 6 years ago

Still using --cgp-param=augustus_cfgs/log_reg_parameters_default.cfg?

ifiddes commented 6 years ago

Hmm I forgot about that test. Does it still crash when you run it with that on? Either way, I would remove that flag and then check the work directory as I said above. Practically, you are going to want to run training anyways.

One other, practical question -- for your purposes, are you going to even use CGP? Since that is the only thing that isn't working, if you aren't going to actually use it on your data then you could just run the test without it and I can figure out what is wrong later.

On Mon, Sep 24, 2018 at 2:33 PM Francesco Cicconardi < notifications@github.com> wrote:

Still using --cgp-param=augustus_cfgs/log_reg_parameters_default.cfg?

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/ComparativeGenomicsToolkit/Comparative-Annotation-Toolkit/issues/111#issuecomment-424133261, or mute the thread https://github.com/notifications/unsubscribe-auth/AHdLXe2yqsy-OqMiuLPtQu8K2Dg3jx2bks5ueU-lgaJpZM4WyEUo .

-- Ian Fiddes, PhD

francicco commented 6 years ago

These errors are with that flag F

francicco commented 6 years ago

===== Luigi Execution Summary =====

Scheduled 134 tasks of which:
* 26 complete ones were encountered:
    - 1 AlignTranscripts(...)
    - 1 Augustus(...)
    - 1 AugustusPb(...)
    - 1 BuildDb(...)
    - 1 Chaining(...)
    ...
* 30 ran successfully:
    - 7 EvaluateDriverTask(...)
    - 1 EvaluateTransMap(...)
    - 7 EvaluateTransMapDriverTask(...)
    - 1 EvaluateTranscripts(...)
    - 7 FilterTransMap(...)
    ...
* 1 failed:
    - 1 AugustusCgp(...)
* 1 failed scheduling:
    - 1 Hgm(...)
* 76 were left pending, among these:
    * 76 had failed dependencies:
        - 1 AssemblyHub(...)
        - 3 AugustusTrack(...)
        - 8 BgpTrack(...)
        - 1 Consensus(...)
        - 7 ConsensusDriverTask(...)
        ...
    * 55 had dependencies whose scheduling failed:
        - 3 AugustusTrack(...)
        - 8 BgpTrack(...)
        - 7 ConsensusDriverTask(...)
        - 7 ConsensusTrack(...)
        - 1 CreateDirectoryStructure(...)
        ...

This progress looks :( because there were failed tasks and tasks whose scheduling failed

===== Luigi Execution Summary =====

drwxr-xr-x 3 fc464 4.0K Sep 24 22:37 tmpIxIfQE
drwxr-xr-x 3 fc464 4.0K Sep 24 22:45 tmpNjNtPp
[fc464@login-e-12 toil-a51496bc-94c5-4b2a-9350-ad6412a98fca-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd]$ cd 
[fc464@login-e-12 ~]$ 
[fc464@login-e-12 ~]$ cd -
/home/fc464/rds/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-a51496bc-94c5-4b2a-9350-ad6412a98fca-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd
[fc464@login-e-12 toil-a51496bc-94c5-4b2a-9350-ad6412a98fca-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd]$ ll tmp*
tmpIxIfQE:
total 8.0K
drwxrwxrwx 2 fc464 4.0K Sep 24 22:45 245229b4-988e-4430-833c-736e21e44921
-rwxrwxr-x 1 fc464 3.0K Sep 24 22:45 worker_log.txt

tmpNjNtPp:
total 8.0K
drwxrwxrwx 2 fc464 4.0K Sep 24 22:53 7c0dc1e5-9331-4195-91ba-68c8b44839d6
-rwxrwxr-x 1 fc464 3.0K Sep 24 22:53 worker_log.txt
[fc464@login-e-12 toil-a51496bc-94c5-4b2a-9350-ad6412a98fca-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd]$ ll tmp*/*
-rwxrwxr-x 1 fc464 3.0K Sep 24 22:45 tmpIxIfQE/worker_log.txt
-rwxrwxr-x 1 fc464 3.0K Sep 24 22:53 tmpNjNtPp/worker_log.txt

tmpIxIfQE/245229b4-988e-4430-833c-736e21e44921:
total 37M
-rw-rw-r-- 1 fc464  32K Sep 24 22:45 bosTau8.cgp.gff
-rw-rw-r-- 1 fc464  30K Sep 24 22:45 canFam3.cgp.gff
-rw-rw-r-- 1 fc464 2.6M Sep 24 22:45 exonCands.bosTau8.gff3
-rw-rw-r-- 1 fc464 2.7M Sep 24 22:45 exonCands.canFam3.gff3
-rw-rw-r-- 1 fc464 2.7M Sep 24 22:45 exonCands.galGal4.gff3
-rw-rw-r-- 1 fc464 3.7M Sep 24 22:45 exonCands.hg38.gff3
-rw-rw-r-- 1 fc464 2.9M Sep 24 22:45 exonCands.mm10.gff3
-rw-rw-r-- 1 fc464 7.2M Sep 24 22:45 exonCands.monDom5.gff3
-rw-rw-r-- 1 fc464 3.5M Sep 24 22:45 exonCands.rheMac3.gff3
-rw-rw-r-- 1 fc464 1.4M Sep 24 22:45 exonCands.rn6.gff3
-rw-rw-r-- 1 fc464  27K Sep 24 22:45 galGal4.cgp.gff
-rw-rw-r-- 1 fc464  31K Sep 24 22:45 hg38.cgp.gff
-rw-rw-r-- 1 fc464 3.9M Sep 24 22:45 login-e-14.313012.5031576748.tmp
-rw-rw-r-- 1 fc464 880K Sep 24 22:45 login-e-14.313012.6189419096.tmp
-rw-rw-r-- 1 fc464 1.8K Sep 24 22:37 login-e-14.313012.6609061795.tmp
-rw-rw-r-- 1 fc464  25K Sep 24 22:45 mm10.cgp.gff
-rw-rw-r-- 1 fc464  29K Sep 24 22:45 monDom5.cgp.gff
-rw-rw-r-- 1 fc464 441K Sep 24 22:45 orthoExons.bosTau8.gff3
-rw-rw-r-- 1 fc464 426K Sep 24 22:45 orthoExons.canFam3.gff3
-rw-rw-r-- 1 fc464 172K Sep 24 22:45 orthoExons.galGal4.gff3
-rw-rw-r-- 1 fc464 993K Sep 24 22:45 orthoExons.hg38.gff3
-rw-rw-r-- 1 fc464 979K Sep 24 22:45 orthoExons.mm10.gff3
-rw-rw-r-- 1 fc464 294K Sep 24 22:45 orthoExons.monDom5.gff3
-rw-rw-r-- 1 fc464 979K Sep 24 22:45 orthoExons.rheMac3.gff3
-rw-rw-r-- 1 fc464 816K Sep 24 22:45 orthoExons.rn6.gff3
-rw-rw-r-- 1 fc464  29K Sep 24 22:45 rheMac3.cgp.gff
-rw-rw-r-- 1 fc464  11K Sep 24 22:45 rn6.cgp.gff

tmpNjNtPp/7c0dc1e5-9331-4195-91ba-68c8b44839d6:
total 37M
-rw-rw-r-- 1 fc464  32K Sep 24 22:53 bosTau8.cgp.gff
-rw-rw-r-- 1 fc464  30K Sep 24 22:53 canFam3.cgp.gff
-rw-rw-r-- 1 fc464 2.6M Sep 24 22:53 exonCands.bosTau8.gff3
-rw-rw-r-- 1 fc464 2.7M Sep 24 22:53 exonCands.canFam3.gff3
-rw-rw-r-- 1 fc464 2.7M Sep 24 22:53 exonCands.galGal4.gff3
-rw-rw-r-- 1 fc464 3.7M Sep 24 22:53 exonCands.hg38.gff3
-rw-rw-r-- 1 fc464 2.9M Sep 24 22:53 exonCands.mm10.gff3
-rw-rw-r-- 1 fc464 7.2M Sep 24 22:53 exonCands.monDom5.gff3
-rw-rw-r-- 1 fc464 3.5M Sep 24 22:53 exonCands.rheMac3.gff3
-rw-rw-r-- 1 fc464 1.4M Sep 24 22:53 exonCands.rn6.gff3
-rw-rw-r-- 1 fc464  27K Sep 24 22:53 galGal4.cgp.gff
-rw-rw-r-- 1 fc464  31K Sep 24 22:53 hg38.cgp.gff
-rw-rw-r-- 1 fc464 3.9M Sep 24 22:53 login-e-14.321804.2898436705.tmp
-rw-rw-r-- 1 fc464 880K Sep 24 22:53 login-e-14.321804.4381590525.tmp
-rw-rw-r-- 1 fc464 1.8K Sep 24 22:45 login-e-14.321804.9471213309.tmp
-rw-rw-r-- 1 fc464  25K Sep 24 22:53 mm10.cgp.gff
-rw-rw-r-- 1 fc464  29K Sep 24 22:53 monDom5.cgp.gff
-rw-rw-r-- 1 fc464 441K Sep 24 22:53 orthoExons.bosTau8.gff3
-rw-rw-r-- 1 fc464 426K Sep 24 22:53 orthoExons.canFam3.gff3
-rw-rw-r-- 1 fc464 172K Sep 24 22:53 orthoExons.galGal4.gff3
-rw-rw-r-- 1 fc464 993K Sep 24 22:53 orthoExons.hg38.gff3
-rw-rw-r-- 1 fc464 979K Sep 24 22:53 orthoExons.mm10.gff3
-rw-rw-r-- 1 fc464 294K Sep 24 22:53 orthoExons.monDom5.gff3
-rw-rw-r-- 1 fc464 979K Sep 24 22:53 orthoExons.rheMac3.gff3
-rw-rw-r-- 1 fc464 816K Sep 24 22:53 orthoExons.rn6.gff3
-rw-rw-r-- 1 fc464  29K Sep 24 22:53 rheMac3.cgp.gff
-rw-rw-r-- 1 fc464  11K Sep 24 22:53 rn6.cgp.gff

ifiddes commented 6 years ago

You are missing the file 'mm10.sampled_GFs.gff'. I have no idea why that is the case.

On Mon, Sep 24, 2018 at 2:55 PM Francesco Cicconardi < notifications@github.com> wrote:

===== Luigi Execution Summary =====

Scheduled 134 tasks of which:

26 complete ones were encountered:

1 AlignTranscripts(...)

1 Augustus(...)

1 AugustusPb(...)

1 BuildDb(...)

1 Chaining(...) ...

30 ran successfully:

7 EvaluateDriverTask(...)

1 EvaluateTransMap(...)

7 EvaluateTransMapDriverTask(...)

1 EvaluateTranscripts(...)

7 FilterTransMap(...) ...

1 failed:

1 AugustusCgp(...)

1 failed scheduling:

1 Hgm(...)

76 were left pending, among these:

76 had failed dependencies:

1 AssemblyHub(...)

3 AugustusTrack(...)

8 BgpTrack(...)

1 Consensus(...)

7 ConsensusDriverTask(...) ...

55 had dependencies whose scheduling failed:

3 AugustusTrack(...)

8 BgpTrack(...)

7 ConsensusDriverTask(...)

7 ConsensusTrack(...)

1 CreateDirectoryStructure(...) ...

This progress looks :( because there were failed tasks and tasks whose scheduling failed

===== Luigi Execution Summary =====

drwxr-xr-x 3 fc464 4.0K Sep 24 22:37 tmpIxIfQE drwxr-xr-x 3 fc464 4.0K Sep 24 22:45 tmpNjNtPp [fc464@login-e-12 toil-a51496bc-94c5-4b2a-9350-ad6412a98fca-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd]$ cd [fc464@login-e-12 ~]$ [fc464@login-e-12 ~]$ cd - /home/fc464/rds/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-a51496bc-94c5-4b2a-9350-ad6412a98fca-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd [fc464@login-e-12 toil-a51496bc-94c5-4b2a-9350-ad6412a98fca-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd]$ ll tmp* tmpIxIfQE: total 8.0K drwxrwxrwx 2 fc464 4.0K Sep 24 22:45 245229b4-988e-4430-833c-736e21e44921 -rwxrwxr-x 1 fc464 3.0K Sep 24 22:45 worker_log.txt

tmpNjNtPp: total 8.0K drwxrwxrwx 2 fc464 4.0K Sep 24 22:53 7c0dc1e5-9331-4195-91ba-68c8b44839d6 -rwxrwxr-x 1 fc464 3.0K Sep 24 22:53 worker_log.txt [fc464@login-e-12 toil-a51496bc-94c5-4b2a-9350-ad6412a98fca-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd]$ ll tmp/ -rwxrwxr-x 1 fc464 3.0K Sep 24 22:45 tmpIxIfQE/worker_log.txt -rwxrwxr-x 1 fc464 3.0K Sep 24 22:53 tmpNjNtPp/worker_log.txt

tmpIxIfQE/245229b4-988e-4430-833c-736e21e44921: total 37M -rw-rw-r-- 1 fc464 32K Sep 24 22:45 bosTau8.cgp.gff -rw-rw-r-- 1 fc464 30K Sep 24 22:45 canFam3.cgp.gff -rw-rw-r-- 1 fc464 2.6M Sep 24 22:45 exonCands.bosTau8.gff3 -rw-rw-r-- 1 fc464 2.7M Sep 24 22:45 exonCands.canFam3.gff3 -rw-rw-r-- 1 fc464 2.7M Sep 24 22:45 exonCands.galGal4.gff3 -rw-rw-r-- 1 fc464 3.7M Sep 24 22:45 exonCands.hg38.gff3 -rw-rw-r-- 1 fc464 2.9M Sep 24 22:45 exonCands.mm10.gff3 -rw-rw-r-- 1 fc464 7.2M Sep 24 22:45 exonCands.monDom5.gff3 -rw-rw-r-- 1 fc464 3.5M Sep 24 22:45 exonCands.rheMac3.gff3 -rw-rw-r-- 1 fc464 1.4M Sep 24 22:45 exonCands.rn6.gff3 -rw-rw-r-- 1 fc464 27K Sep 24 22:45 galGal4.cgp.gff -rw-rw-r-- 1 fc464 31K Sep 24 22:45 hg38.cgp.gff -rw-rw-r-- 1 fc464 3.9M Sep 24 22:45 login-e-14.313012.5031576748.tmp -rw-rw-r-- 1 fc464 880K Sep 24 22:45 login-e-14.313012.6189419096.tmp -rw-rw-r-- 1 fc464 1.8K Sep 24 22:37 login-e-14.313012.6609061795.tmp -rw-rw-r-- 1 fc464 25K Sep 24 22:45 mm10.cgp.gff -rw-rw-r-- 1 fc464 29K Sep 24 22:45 monDom5.cgp.gff -rw-rw-r-- 1 fc464 441K Sep 24 22:45 orthoExons.bosTau8.gff3 -rw-rw-r-- 1 fc464 426K Sep 24 22:45 orthoExons.canFam3.gff3 -rw-rw-r-- 1 fc464 172K Sep 24 22:45 orthoExons.galGal4.gff3 -rw-rw-r-- 1 fc464 993K Sep 24 22:45 orthoExons.hg38.gff3 -rw-rw-r-- 1 fc464 979K Sep 24 22:45 orthoExons.mm10.gff3 -rw-rw-r-- 1 fc464 294K Sep 24 22:45 orthoExons.monDom5.gff3 -rw-rw-r-- 1 fc464 979K Sep 24 22:45 orthoExons.rheMac3.gff3 -rw-rw-r-- 1 fc464 816K Sep 24 22:45 orthoExons.rn6.gff3 -rw-rw-r-- 1 fc464 29K Sep 24 22:45 rheMac3.cgp.gff -rw-rw-r-- 1 fc464 11K Sep 24 22:45 rn6.cgp.gff

tmpNjNtPp/7c0dc1e5-9331-4195-91ba-68c8b44839d6: total 37M -rw-rw-r-- 1 fc464 32K Sep 24 22:53 bosTau8.cgp.gff -rw-rw-r-- 1 fc464 30K Sep 24 22:53 canFam3.cgp.gff -rw-rw-r-- 1 fc464 2.6M Sep 24 22:53 exonCands.bosTau8.gff3 -rw-rw-r-- 1 fc464 2.7M Sep 24 22:53 exonCands.canFam3.gff3 -rw-rw-r-- 1 fc464 2.7M Sep 24 22:53 exonCands.galGal4.gff3 -rw-rw-r-- 1 fc464 3.7M Sep 24 22:53 exonCands.hg38.gff3 -rw-rw-r-- 1 fc464 2.9M Sep 24 22:53 exonCands.mm10.gff3 -rw-rw-r-- 1 fc464 7.2M Sep 24 22:53 exonCands.monDom5.gff3 -rw-rw-r-- 1 fc464 3.5M Sep 24 22:53 exonCands.rheMac3.gff3 -rw-rw-r-- 1 fc464 1.4M Sep 24 22:53 exonCands.rn6.gff3 -rw-rw-r-- 1 fc464 27K Sep 24 22:53 galGal4.cgp.gff -rw-rw-r-- 1 fc464 31K Sep 24 22:53 hg38.cgp.gff -rw-rw-r-- 1 fc464 3.9M Sep 24 22:53 login-e-14.321804.2898436705.tmp -rw-rw-r-- 1 fc464 880K Sep 24 22:53 login-e-14.321804.4381590525.tmp -rw-rw-r-- 1 fc464 1.8K Sep 24 22:45 login-e-14.321804.9471213309.tmp -rw-rw-r-- 1 fc464 25K Sep 24 22:53 mm10.cgp.gff -rw-rw-r-- 1 fc464 29K Sep 24 22:53 monDom5.cgp.gff -rw-rw-r-- 1 fc464 441K Sep 24 22:53 orthoExons.bosTau8.gff3 -rw-rw-r-- 1 fc464 426K Sep 24 22:53 orthoExons.canFam3.gff3 -rw-rw-r-- 1 fc464 172K Sep 24 22:53 orthoExons.galGal4.gff3 -rw-rw-r-- 1 fc464 993K Sep 24 22:53 orthoExons.hg38.gff3 -rw-rw-r-- 1 fc464 979K Sep 24 22:53 orthoExons.mm10.gff3 -rw-rw-r-- 1 fc464 294K Sep 24 22:53 orthoExons.monDom5.gff3 -rw-rw-r-- 1 fc464 979K Sep 24 22:53 orthoExons.rheMac3.gff3 -rw-rw-r-- 1 fc464 816K Sep 24 22:53 orthoExons.rn6.gff3 -rw-rw-r-- 1 fc464 29K Sep 24 22:53 rheMac3.cgp.gff -rw-rw-r-- 1 fc464 11K Sep 24 22:53 rn6.cgp.gff

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/ComparativeGenomicsToolkit/Comparative-Annotation-Toolkit/issues/111#issuecomment-424138900, or mute the thread https://github.com/notifications/unsubscribe-auth/AHdLXdZgIqvIawwLLcZbuOP1zTADGwazks5ueVTsgaJpZM4WyEUo .

-- Ian Fiddes, PhD

ifiddes commented 6 years ago

Are you sure that the recompiled version of augustus is on your path? (which augustus). I cannot figure out how this could happen otherwise, because the test dataset works on travis using the exact commit that you just recompiled with...

On Mon, Sep 24, 2018 at 3:12 PM Ian Fiddes ian.t.fiddes@gmail.com wrote:

You are missing the file 'mm10.sampled_GFs.gff'. I have no idea why that is the case.

On Mon, Sep 24, 2018 at 2:55 PM Francesco Cicconardi < notifications@github.com> wrote:

===== Luigi Execution Summary =====

Scheduled 134 tasks of which:

26 complete ones were encountered:

1 AlignTranscripts(...)

1 Augustus(...)

1 AugustusPb(...)

1 BuildDb(...)

1 Chaining(...) ...

30 ran successfully:

7 EvaluateDriverTask(...)

1 EvaluateTransMap(...)

7 EvaluateTransMapDriverTask(...)

1 EvaluateTranscripts(...)

7 FilterTransMap(...) ...

1 failed:

1 AugustusCgp(...)

1 failed scheduling:

1 Hgm(...)

76 were left pending, among these:

76 had failed dependencies:

1 AssemblyHub(...)

3 AugustusTrack(...)

8 BgpTrack(...)

1 Consensus(...)

7 ConsensusDriverTask(...) ...

55 had dependencies whose scheduling failed:

3 AugustusTrack(...)

8 BgpTrack(...)

7 ConsensusDriverTask(...)

7 ConsensusTrack(...)

1 CreateDirectoryStructure(...) ...

This progress looks :( because there were failed tasks and tasks whose scheduling failed

===== Luigi Execution Summary =====

drwxr-xr-x 3 fc464 4.0K Sep 24 22:37 tmpIxIfQE drwxr-xr-x 3 fc464 4.0K Sep 24 22:45 tmpNjNtPp [fc464@login-e-12 toil-a51496bc-94c5-4b2a-9350-ad6412a98fca-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd]$ cd [fc464@login-e-12 ~]$ [fc464@login-e-12 ~]$ cd - /home/fc464/rds/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-a51496bc-94c5-4b2a-9350-ad6412a98fca-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd [fc464@login-e-12 toil-a51496bc-94c5-4b2a-9350-ad6412a98fca-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd]$ ll tmp* tmpIxIfQE: total 8.0K drwxrwxrwx 2 fc464 4.0K Sep 24 22:45 245229b4-988e-4430-833c-736e21e44921 -rwxrwxr-x 1 fc464 3.0K Sep 24 22:45 worker_log.txt

tmpNjNtPp: total 8.0K drwxrwxrwx 2 fc464 4.0K Sep 24 22:53 7c0dc1e5-9331-4195-91ba-68c8b44839d6 -rwxrwxr-x 1 fc464 3.0K Sep 24 22:53 worker_log.txt [fc464@login-e-12 toil-a51496bc-94c5-4b2a-9350-ad6412a98fca-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd]$ ll tmp/ -rwxrwxr-x 1 fc464 3.0K Sep 24 22:45 tmpIxIfQE/worker_log.txt -rwxrwxr-x 1 fc464 3.0K Sep 24 22:53 tmpNjNtPp/worker_log.txt

tmpIxIfQE/245229b4-988e-4430-833c-736e21e44921: total 37M -rw-rw-r-- 1 fc464 32K Sep 24 22:45 bosTau8.cgp.gff -rw-rw-r-- 1 fc464 30K Sep 24 22:45 canFam3.cgp.gff -rw-rw-r-- 1 fc464 2.6M Sep 24 22:45 exonCands.bosTau8.gff3 -rw-rw-r-- 1 fc464 2.7M Sep 24 22:45 exonCands.canFam3.gff3 -rw-rw-r-- 1 fc464 2.7M Sep 24 22:45 exonCands.galGal4.gff3 -rw-rw-r-- 1 fc464 3.7M Sep 24 22:45 exonCands.hg38.gff3 -rw-rw-r-- 1 fc464 2.9M Sep 24 22:45 exonCands.mm10.gff3 -rw-rw-r-- 1 fc464 7.2M Sep 24 22:45 exonCands.monDom5.gff3 -rw-rw-r-- 1 fc464 3.5M Sep 24 22:45 exonCands.rheMac3.gff3 -rw-rw-r-- 1 fc464 1.4M Sep 24 22:45 exonCands.rn6.gff3 -rw-rw-r-- 1 fc464 27K Sep 24 22:45 galGal4.cgp.gff -rw-rw-r-- 1 fc464 31K Sep 24 22:45 hg38.cgp.gff -rw-rw-r-- 1 fc464 3.9M Sep 24 22:45 login-e-14.313012.5031576748.tmp -rw-rw-r-- 1 fc464 880K Sep 24 22:45 login-e-14.313012.6189419096.tmp -rw-rw-r-- 1 fc464 1.8K Sep 24 22:37 login-e-14.313012.6609061795.tmp -rw-rw-r-- 1 fc464 25K Sep 24 22:45 mm10.cgp.gff -rw-rw-r-- 1 fc464 29K Sep 24 22:45 monDom5.cgp.gff -rw-rw-r-- 1 fc464 441K Sep 24 22:45 orthoExons.bosTau8.gff3 -rw-rw-r-- 1 fc464 426K Sep 24 22:45 orthoExons.canFam3.gff3 -rw-rw-r-- 1 fc464 172K Sep 24 22:45 orthoExons.galGal4.gff3 -rw-rw-r-- 1 fc464 993K Sep 24 22:45 orthoExons.hg38.gff3 -rw-rw-r-- 1 fc464 979K Sep 24 22:45 orthoExons.mm10.gff3 -rw-rw-r-- 1 fc464 294K Sep 24 22:45 orthoExons.monDom5.gff3 -rw-rw-r-- 1 fc464 979K Sep 24 22:45 orthoExons.rheMac3.gff3 -rw-rw-r-- 1 fc464 816K Sep 24 22:45 orthoExons.rn6.gff3 -rw-rw-r-- 1 fc464 29K Sep 24 22:45 rheMac3.cgp.gff -rw-rw-r-- 1 fc464 11K Sep 24 22:45 rn6.cgp.gff

tmpNjNtPp/7c0dc1e5-9331-4195-91ba-68c8b44839d6: total 37M -rw-rw-r-- 1 fc464 32K Sep 24 22:53 bosTau8.cgp.gff -rw-rw-r-- 1 fc464 30K Sep 24 22:53 canFam3.cgp.gff -rw-rw-r-- 1 fc464 2.6M Sep 24 22:53 exonCands.bosTau8.gff3 -rw-rw-r-- 1 fc464 2.7M Sep 24 22:53 exonCands.canFam3.gff3 -rw-rw-r-- 1 fc464 2.7M Sep 24 22:53 exonCands.galGal4.gff3 -rw-rw-r-- 1 fc464 3.7M Sep 24 22:53 exonCands.hg38.gff3 -rw-rw-r-- 1 fc464 2.9M Sep 24 22:53 exonCands.mm10.gff3 -rw-rw-r-- 1 fc464 7.2M Sep 24 22:53 exonCands.monDom5.gff3 -rw-rw-r-- 1 fc464 3.5M Sep 24 22:53 exonCands.rheMac3.gff3 -rw-rw-r-- 1 fc464 1.4M Sep 24 22:53 exonCands.rn6.gff3 -rw-rw-r-- 1 fc464 27K Sep 24 22:53 galGal4.cgp.gff -rw-rw-r-- 1 fc464 31K Sep 24 22:53 hg38.cgp.gff -rw-rw-r-- 1 fc464 3.9M Sep 24 22:53 login-e-14.321804.2898436705.tmp -rw-rw-r-- 1 fc464 880K Sep 24 22:53 login-e-14.321804.4381590525.tmp -rw-rw-r-- 1 fc464 1.8K Sep 24 22:45 login-e-14.321804.9471213309.tmp -rw-rw-r-- 1 fc464 25K Sep 24 22:53 mm10.cgp.gff -rw-rw-r-- 1 fc464 29K Sep 24 22:53 monDom5.cgp.gff -rw-rw-r-- 1 fc464 441K Sep 24 22:53 orthoExons.bosTau8.gff3 -rw-rw-r-- 1 fc464 426K Sep 24 22:53 orthoExons.canFam3.gff3 -rw-rw-r-- 1 fc464 172K Sep 24 22:53 orthoExons.galGal4.gff3 -rw-rw-r-- 1 fc464 993K Sep 24 22:53 orthoExons.hg38.gff3 -rw-rw-r-- 1 fc464 979K Sep 24 22:53 orthoExons.mm10.gff3 -rw-rw-r-- 1 fc464 294K Sep 24 22:53 orthoExons.monDom5.gff3 -rw-rw-r-- 1 fc464 979K Sep 24 22:53 orthoExons.rheMac3.gff3 -rw-rw-r-- 1 fc464 816K Sep 24 22:53 orthoExons.rn6.gff3 -rw-rw-r-- 1 fc464 29K Sep 24 22:53 rheMac3.cgp.gff -rw-rw-r-- 1 fc464 11K Sep 24 22:53 rn6.cgp.gff

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/ComparativeGenomicsToolkit/Comparative-Annotation-Toolkit/issues/111#issuecomment-424138900, or mute the thread https://github.com/notifications/unsubscribe-auth/AHdLXdZgIqvIawwLLcZbuOP1zTADGwazks5ueVTsgaJpZM4WyEUo .

-- Ian Fiddes, PhD

-- Ian Fiddes, PhD

francicco commented 6 years ago

Yes I am sure.

francicco commented 6 years ago

can we isolate the command that should generate that file and execute it? F

ifiddes commented 6 years ago

Yes, that is doable. I have just pushed a change to your branch that will write the command to the log. Run that with --cleanWorkDir=never and then you should be able to re-run the command. Make sure to do git pull and ensure you are on branch test-issue-111. You should see an AssertionError in the log, followed by the command that would be used. Try running that command by hand and see what happens.

francicco commented 6 years ago

I don't know how to pull the test-issue-111... sorry

ifiddes commented 6 years ago

git checkout test-issue-111 && git pull origin test-issue-111

francicco commented 6 years ago

I'm sorry, where do I have to run this? in the Comparative-Annotation-Toolkit directory? F

ifiddes commented 6 years ago

Yes, in the CAT directory.

francicco commented 6 years ago

[fc464@login-e-12 Comparative-Annotation-Toolkit]$ git checkout test-issue-111 && git pull origin test-issue-111
Branch test-issue-111 set up to track remote branch test-issue-111 from origin.
Switched to a new branch 'test-issue-111'
From https://github.com/ComparativeGenomicsToolkit/Comparative-Annotation-Toolkit
 * branch            test-issue-111 -> FETCH_HEAD
Already up-to-date.

ifiddes commented 6 years ago

Alright, and so if you look on line 247 of cat/augustus_cgp.py you should see the command assert False, cmd. That will cause the program to crash before it runs the command, and print it to the log.

francicco commented 6 years ago

Ok, the cmd is there, now I run the script

francicco commented 6 years ago

[...]
09-24 23:54:13 cat   INFO  Loaded table: hg38.mRNA_augTM_Metrics
09-24 23:54:13 luigi-interface INFO  [pid 418304] Worker Worker(salt=017534574, workers=10, host=login-e-14, username=fc464, pid=401574) done      Task: EvaluateDriverTask for rn6
09-24 23:54:13 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_70026afda7   has status   DONE
09-24 23:54:13 cat   INFO  Loaded table: monDom5.mRNA_transMap_Metrics
09-24 23:54:13 cat   INFO  Loaded table: galGal4.mRNA_augTMR_Evaluation
09-24 23:54:13 cat   INFO  Loaded table: canFam3.CDS_augTM_Evaluation
09-24 23:54:13 cat   INFO  Loaded table: rheMac3.CDS_transMap_Evaluation
09-24 23:54:13 luigi-interface INFO  [pid 418305] Worker Worker(salt=017534574, workers=10, host=login-e-14, username=fc464, pid=401574) done      Task: EvaluateDriverTask for canFam3
09-24 23:54:13 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_f4715e1d3c   has status   DONE
09-24 23:54:14 cat   INFO  Loaded table: bosTau8.CDS_augTM_Evaluation
09-24 23:54:14 cat   INFO  Loaded table: hg38.CDS_transMap_Evaluation
09-24 23:54:14 luigi-interface INFO  [pid 418316] Worker Worker(salt=017534574, workers=10, host=login-e-14, username=fc464, pid=401574) done      Task: EvaluateDriverTask for bosTau8
09-24 23:54:14 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_902022f711   has status   DONE
09-24 23:54:14 cat   INFO  Loaded table: galGal4.mRNA_augTM_Metrics
09-24 23:54:14 cat   INFO  Loaded table: rheMac3.mRNA_transMap_Metrics
09-24 23:54:14 cat   INFO  Loaded table: monDom5.CDS_augTM_Evaluation
09-24 23:54:14 cat   INFO  Loaded table: hg38.mRNA_transMap_Metrics
09-24 23:54:14 luigi-interface INFO  [pid 418330] Worker Worker(salt=017534574, workers=10, host=login-e-14, username=fc464, pid=401574) done      Task: EvaluateDriverTask for monDom5
09-24 23:54:14 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_41c87a08f0   has status   DONE
09-24 23:54:14 cat   INFO  Loaded table: rheMac3.CDS_augTM_Evaluation
09-24 23:54:15 luigi-interface INFO  [pid 418307] Worker Worker(salt=017534574, workers=10, host=login-e-14, username=fc464, pid=401574) done      Task: EvaluateDriverTask for rheMac3
09-24 23:54:15 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_8b824cb1d0   has status   DONE
09-24 23:54:15 cat   INFO  Loaded table: galGal4.CDS_transMap_Evaluation
09-24 23:54:15 cat   INFO  Loaded table: hg38.CDS_augTM_Evaluation
09-24 23:54:15 luigi-interface INFO  [pid 418320] Worker Worker(salt=017534574, workers=10, host=login-e-14, username=fc464, pid=401574) done      Task: EvaluateDriverTask for hg38
09-24 23:54:15 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_a83bd75adf   has status   DONE
09-24 23:54:16 cat   INFO  Loaded table: galGal4.mRNA_transMap_Metrics
09-24 23:54:17 cat   INFO  Loaded table: galGal4.CDS_augTM_Evaluation
09-24 23:54:17 luigi-interface INFO  [pid 418312] Worker Worker(salt=017534574, workers=10, host=login-e-14, username=fc464, pid=401574) done      Task: EvaluateDriverTask for galGal4
09-24 23:54:17 luigi-interface INFO  Informed scheduler that task   EvaluateDriverTask_False_True_True_9c9943f49b   has status   DONE
09-24 23:54:17 luigi-interface INFO  [pid 419359] Worker Worker(salt=017534574, workers=10, host=login-e-14, username=fc464, pid=401574) running   Task: EvaluateTranscripts
09-24 23:54:19 luigi-interface INFO  [pid 419359] Worker Worker(salt=017534574, workers=10, host=login-e-14, username=fc464, pid=401574) done      Task: EvaluateTranscripts
09-24 23:54:19 luigi-interface INFO  Informed scheduler that task   EvaluateTranscripts_False_True_True_4bab51e20b   has status   DONE
Exception in thread Thread-8:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 765, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/fileStore.py", line 1471, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-7:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 765, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/fileStore.py", line 1471, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method CachingFileStore.__del__ of <toil.fileStore.CachingFileStore object at 0x16a6490>> ignored
09-24 23:57:20 toil.leader WARNING The job seems to have left a log file, indicating failure: 'cgp' M/H/jobBzGDVG
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG    INFO:toil:Running Toil version 3.17.0-585383ed5c4453b556269818571d3c7419c613b0.
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG    WARNING:toil.resource:'JTRES_3a2fc781691739384afee22d074d5828' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG    WARNING:toil.resource:'JTRES_3a2fc781691739384afee22d074d5828' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG    WARNING:toil.resource:Can't globalize module ModuleDescriptor(dirPath='/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages', name='toil.job', fromVirtualEnv=True).
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG    WARNING:toil.resource:Can't globalize module ModuleDescriptor(dirPath='/home/fc464/software/luigi/Comparative-Annotation-Toolkit', name='cat.augustus_cgp', fromVirtualEnv=False).
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG    WARNING:toil.resource:Can't globalize module ModuleDescriptor(dirPath='/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages', name='toil.job', fromVirtualEnv=True).
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG    WARNING:toil.resource:Can't globalize module ModuleDescriptor(dirPath='/home/fc464/software/luigi/Comparative-Annotation-Toolkit', name='cat.augustus_cgp', fromVirtualEnv=False).
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG    WARNING:toil.resource:Can't globalize module ModuleDescriptor(dirPath='/home/fc464/software/luigi/Comparative-Annotation-Toolkit', name='cat.augustus_cgp', fromVirtualEnv=False).
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG    WARNING:toil.resource:Can't globalize module ModuleDescriptor(dirPath='/home/fc464/software/luigi/Comparative-Annotation-Toolkit', name='cat.augustus_cgp', fromVirtualEnv=False).
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG    WARNING:toil.resource:'JTRES_3a2fc781691739384afee22d074d5828' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG    WARNING:toil.resource:'JTRES_3a2fc781691739384afee22d074d5828' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG    WARNING:toil.resource:'JTRES_3a2fc781691739384afee22d074d5828' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG    cat: /rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-ac83685f-cec9-4400-b68a-a3104d9d8acf-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd/tmp9oMnUh/b2437231-917c-4bf1-8411-34d2bc945f57/mm10.sampled_GFs.gff: No such file or directory
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG    Traceback (most recent call last):
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG      File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG      File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/job.py", line 1350, in _runner
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG        returnValues = self._run(jobGraph, fileStore)
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG      File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/job.py", line 1295, in _run
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG        return self.run(fileStore)
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG      File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/job.py", line 1479, in run
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG        rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG      File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/cat/augustus_cgp.py", line 253, in cgp
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG        tools.procOps.run_proc(cmd, stdout=combined_file)
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG      File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/procOps.py", line 60, in run_proc
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG        pl.wait()
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG      File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG        self.raiseIfExcept()
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG      File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG        p.raiseIfExcept()
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG      File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG        raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2]
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG    ProcException: process exited 1: cat /rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-ac83685f-cec9-4400-b68a-a3104d9d8acf-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd/tmp9oMnUh/b2437231-917c-4bf1-8411-34d2bc945f57/mm10.sampled_GFs.gff /rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-ac83685f-cec9-4400-b68a-a3104d9d8acf-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd/tmp9oMnUh/b2437231-917c-4bf1-8411-34d2bc945f57/exonCands.mm10.gff3 /rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-ac83685f-cec9-4400-b68a-a3104d9d8acf-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd/tmp9oMnUh/b2437231-917c-4bf1-8411-34d2bc945f57/orthoExons.mm10.gff3
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG    ERROR:toil.worker:Exiting the worker because of a failed job on host login-e-14
09-24 23:57:20 toil.leader WARNING M/H/jobBzGDVG    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'cgp' M/H/jobBzGDVG with ID M/H/jobBzGDVG to 1

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 765, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/fileStore.py", line 1471, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 765, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/fileStore.py", line 1471, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method CachingFileStore.__del__ of <toil.fileStore.CachingFileStore object at 0x398ae50>> ignored
09-25 00:04:57 toil.leader WARNING The job seems to have left a log file, indicating failure: 'cgp' M/H/jobBzGDVG
09-25 00:04:57 toil.leader WARNING M/H/jobBzGDVG    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
09-25 00:04:57 toil.leader WARNING M/H/jobBzGDVG    INFO:toil:Running Toil version 3.17.0-585383ed5c4453b556269818571d3c7419c613b0.
09-25 00:04:57 toil.leader WARNING M/H/jobBzGDVG    WARNING:toil.resource:'JTRES_3a2fc781691739384afee22d074d5828' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
09-25 00:04:57 toil.leader WARNING M/H/jobBzGDVG    WARNING:toil.resource:'JTRES_3a2fc781691739384afee22d074d5828' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
09-25 00:04:57 toil.leader WARNING M/H/jobBzGDVG    cat: /rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-ac83685f-cec9-4400-b68a-a3104d9d8acf-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd/tmpOKlK0R/7f7f754b-8091-40cd-b73e-123c56a28dbd/mm10.sampled_GFs.gff: No such file or directory
09-25 00:04:57 toil.leader WARNING M/H/jobBzGDVG    Traceback (most recent call last):
09-25 00:04:57 toil.leader WARNING M/H/jobBzGDVG      File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
09-25 00:04:57 toil.leader WARNING M/H/jobBzGDVG        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
09-25 00:04:57 toil.leader WARNING M/H/jobBzGDVG      File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/job.py", line 1350, in _runner
09-25 00:04:57 toil.leader WARNING M/H/jobBzGDVG        returnValues = self._run(jobGraph, fileStore)
09-25 00:04:57 toil.leader WARNING M/H/jobBzGDVG      File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/job.py", line 1295, in _run
09-25 00:04:57 toil.leader WARNING M/H/jobBzGDVG        return self.run(fileStore)
09-25 00:04:57 toil.leader WARNING M/H/jobBzGDVG      File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/job.py", line 1479, in run
09-25 00:04:57 toil.leader WARNING M/H/jobBzGDVG        rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
09-25 00:04:57 toil.leader WARNING M/H/jobBzGDVG      File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/cat/augustus_cgp.py", line 253, in cgp
09-25 00:04:57 toil.leader WARNING M/H/jobBzGDVG        tools.procOps.run_proc(cmd, stdout=combined_file)
09-25 00:04:57 toil.leader WARNING M/H/jobBzGDVG      File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/procOps.py", line 60, in run_proc
09-25 00:04:57 toil.leader WARNING M/H/jobBzGDVG        pl.wait()
09-25 00:04:57 toil.leader WARNING M/H/jobBzGDVG      File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1127, in wait
09-25 00:04:57 toil.leader WARNING M/H/jobBzGDVG        self.raiseIfExcept()
09-25 00:04:57 toil.leader WARNING M/H/jobBzGDVG      File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/pipeline.py", line 1085, in raiseIfExcept
09-25 00:04:57 toil.leader WARNING M/H/jobBzGDVG        p.raiseIfExcept()
09-25 00:04:57 toil.leader WARNING M/H/jobBzGDVG      File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/tools/pipeline.py", line 749, in raiseIfExcept
09-25 00:04:57 toil.leader WARNING M/H/jobBzGDVG        raise self.exceptInfo[0], self.exceptInfo[1], self.exceptInfo[2]
09-25 00:04:57 toil.leader WARNING M/H/jobBzGDVG    ProcException: process exited 1: cat /rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-ac83685f-cec9-4400-b68a-a3104d9d8acf-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd/tmpOKlK0R/7f7f754b-8091-40cd-b73e-123c56a28dbd/mm10.sampled_GFs.gff /rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-ac83685f-cec9-4400-b68a-a3104d9d8acf-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd/tmpOKlK0R/7f7f754b-8091-40cd-b73e-123c56a28dbd/exonCands.mm10.gff3 /rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/toil-ac83685f-cec9-4400-b68a-a3104d9d8acf-06a4796e-1e25-4c7a-bb62-eeb5d330f8dd/tmpOKlK0R/7f7f754b-8091-40cd-b73e-123c56a28dbd/orthoExons.mm10.gff3
09-25 00:04:57 toil.leader WARNING M/H/jobBzGDVG    ERROR:toil.worker:Exiting the worker because of a failed job on host login-e-14
09-25 00:04:57 toil.leader WARNING M/H/jobBzGDVG    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'cgp' M/H/jobBzGDVG with ID M/H/jobBzGDVG to 0
09-25 00:04:57 toil.leader WARNING Job 'cgp' M/H/jobBzGDVG with ID M/H/jobBzGDVG is completely failed
09-25 00:05:01 luigi-interface ERROR [pid 406513] Worker Worker(salt=017534574, workers=10, host=login-e-14, username=fc464, pid=401574) failed    ToilTask: AugustusCgp using batchSystem singleMachine
Traceback (most recent call last):
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps
    task_gen = self.task.run()
  File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/cat/__init__.py", line 1446, in run
    augustus_cgp(cgp_args, toil_options)
  File "/home/fc464/software/luigi/Comparative-Annotation-Toolkit/cat/augustus_cgp.py", line 57, in augustus_cgp
    results, stdout_file_ids, param_file_id = t.start(job)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 783, in start
    return self._runMainLoop(rootJobGraph)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 242, in run
    raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
FailedJobsException
09-25 00:05:01 luigi-interface INFO  Informed scheduler that task   AugustusCgp_False_True_True_4bab51e20b   has status   FAILED
09-25 00:05:01 luigi-interface INFO  Worker Worker(salt=017534574, workers=10, host=login-e-14, username=fc464, pid=401574) was stopped. Shutting down Keep-Alive thread
09-25 00:05:03 luigi-interface INFO  
===== Luigi Execution Summary =====

Scheduled 162 tasks of which:
* 8 complete ones were encountered:
    - 1 FakePsl(...)
    - 1 FlatTranscriptFasta(...)
    - 1 GenerateHints(...)
    - 1 GenomeFiles(...)
    - 1 Gff3ToGenePred(...)
    ...
* 75 ran successfully:
    - 7 AlignTranscriptDriverTask(...)
    - 1 AlignTranscripts(...)
    - 1 Augustus(...)
    - 7 AugustusDriverTask(...)
    - 1 AugustusPb(...)
    ...
* 1 failed:
    - 1 AugustusCgp(...)
* 78 were left pending, among these:
    * 78 had failed dependencies:
        - 1 AssemblyHub(...)
        - 3 AugustusTrack(...)
        - 8 BgpTrack(...)
        - 1 Consensus(...)
        - 7 ConsensusDriverTask(...)
        ...

This progress looks :( because there were failed tasks

===== Luigi Execution Summary =====

ComparativeGenomicsToolkit / Comparative-Annotation-Toolkit

Test #111

COMGENEPRED = true