elixir-cloud-aai / cwl-WES

Trigger CWL workflows via GA4GH WES and TES
Apache License 2.0
16 stars 18 forks source link

Error when submitting a workflow #140

Closed lvarin closed 4 years ago

lvarin commented 4 years ago

Describe the bug

We have been trying the deployment of WES as TESK in Openshift here in CSC in Finland. There seems to be a problem while trying to create a lock file (see the logs below).

To Reproduce Steps to reproduce the behavior:

Go to https://wes-csc-wes.rahtiapp.fi/ga4gh/wes/v1/ui/#/ and run a workflow. @shukapoo has the details on which workflow to run. The workflow ends in error.

Expected behavior

The workflow should finish successfully.

Software Add versions of relevant software, e.g.

Additional context

This is the log of the error:

Here is the log:

/usr/local/bin/cwl-tes 0.2.0 with cwltool 1.0.20181114114354
Resolved "/data/output/VLE1V9/workflow_files/repo/hashsplitter-workflow.cwl" to "file:///data/output/VLE1V9/workflow_files/repo/hashsplitter-workflow.cwl"
[workflow ] initialized from file:///data/output/VLE1V9/workflow_files/repo/hashsplitter-workflow.cwl
[workflow ] start
[workflow ] {
    "input": {
        "class": "File",
        "location": "https://github.com/uniqueg/cwltool/blob/master/requirements.txt",
        "basename": "requirements.txt",
        "nameroot": "requirements",
        "nameext": ".txt"
    }
}
[workflow ] starting step md5
[job step md5] job input {
    "file:///data/output/VLE1V9/workflow_files/repo/hashsplitter-workflow.cwl#md5/input": {
        "class": "File",
        "location": "https://github.com/uniqueg/cwltool/blob/master/requirements.txt",
        "basename": "requirements.txt",
        "nameroot": "requirements",
        "nameext": ".txt"
    }
}
[job step md5] evaluated job input to {
    "file:///data/output/VLE1V9/workflow_files/repo/hashsplitter-workflow.cwl#md5/input": {
        "class": "File",
        "location": "https://github.com/uniqueg/cwltool/blob/master/requirements.txt",
        "basename": "requirements.txt",
        "nameroot": "requirements",
        "nameext": ".txt"
    }
}
[step md5] start
[job md5] initializing from file:///data/output/VLE1V9/workflow_files/repo/hashsplitter-md5.cwl.yml as part of step md5
[job md5] {
    "input": {
        "class": "File",
        "location": "https://github.com/uniqueg/cwltool/blob/master/requirements.txt",
        "basename": "requirements.txt",
        "nameroot": "requirements",
        "nameext": ".txt"
    }
}
Unexpected exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/lockfile/linklockfile.py", line 19, in acquire
    open(self.unique_name, "wb").close()
FileNotFoundError: [Errno 2] No such file or directory: "/.cache/cwltool/2/4/d/2/5/celery-worker-6679c5868f-fstcx-27575700.122614218408434280774"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/src/cwl-tes/cwl_tes/tes.py", line 126, in visit
    deref = downloadHttpFile(path)
  File "/usr/local/lib/python3.6/site-packages/cwltool-1.0.20181114114354-py3.6.egg/cwltool/pathmapper.py", line 157, in downloadHttpFile
    for chunk in r.iter_content(chunk_size=16384):
  File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 750, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 461, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 637, in read_chunked
    self._update_chunk_length()
  File "/usr/local/lib/python3.6/site-packages/cachecontrol/adapter.py", line 106, in _update_chunk_length
    self._fp._close()
  File "/usr/local/lib/python3.6/site-packages/cachecontrol/filewrapper.py", line 50, in _close
    self.__callback(self.__buf.getvalue())
  File "/usr/local/lib/python3.6/site-packages/cachecontrol/controller.py", line 275, in cache_response
    self.serializer.dumps(request, response, body=body),
  File "/usr/local/lib/python3.6/site-packages/cachecontrol/caches/file_cache.py", line 99, in set
    with self.lock_class(name) as lock:
  File "/usr/local/lib/python3.6/site-packages/lockfile/__init__.py", line 197, in __enter__
    self.acquire()
  File "/usr/local/lib/python3.6/site-packages/lockfile/linklockfile.py", line 21, in acquire
    raise LockFailed("failed to create %s" % self.unique_name)
lockfile.LockFailed: failed to create /.cache/cwltool/2/4/d/2/5/celery-worker-6679c5868f-fstcx-27575700.122614218408434280774

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/cwltool-1.0.20181114114354-py3.6.egg/cwltool/workflow.py", line 789, in job
    runtimeContext):
  File "/usr/local/lib/python3.6/site-packages/cwltool-1.0.20181114114354-py3.6.egg/cwltool/command_line_tool.py", line 409, in job
    reffiles, builder.stagedir, runtimeContext, True)
  File "/app/src/cwl-tes/cwl_tes/tes.py", line 61, in make_path_mapper
    runtimeContext.make_fs_access(self.remote_storage_url or ""))
  File "/app/src/cwl-tes/cwl_tes/tes.py", line 83, in __init__
    separateDirs)
  File "/usr/local/lib/python3.6/site-packages/cwltool-1.0.20181114114354-py3.6.egg/cwltool/pathmapper.py", line 241, in __init__
    self.setup(dedup(referenced_files), basedir)
  File "/usr/local/lib/python3.6/site-packages/cwltool-1.0.20181114114354-py3.6.egg/cwltool/pathmapper.py", line 298, in setup
    self.visit(fob, stagedir, basedir, copy=fob.get("writable"), staged=True)
  File "/app/src/cwl-tes/cwl_tes/tes.py", line 143, in visit
    copy=copy, staged=staged)
  File "/usr/local/lib/python3.6/site-packages/schema_salad/sourceline.py", line 166, in __exit__
    raise self.makeError("
".join(traceback.format_exception(exc_type, exc_value, tb)))
schema_salad.validate.ValidationException: ../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3: Traceback (most recent call last):
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/lockfile/linklockfile.py", line 19, in acquire
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     open(self.unique_name, "wb").close()
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3: FileNotFoundError: [Errno 2] No such file or directory: "/.cache/cwltool/2/4/d/2/5/celery-worker-6679c5868f-fstcx-27575700.122614218408434280774"
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3: During handling of the above exception, another exception occurred:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3: Traceback (most recent call last):
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/app/src/cwl-tes/cwl_tes/tes.py", line 126, in visit
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     deref = downloadHttpFile(path)
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/cwltool-1.0.20181114114354-py3.6.egg/cwltool/pathmapper.py", line 157, in downloadHttpFile
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     for chunk in r.iter_content(chunk_size=16384):
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 750, in generate
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     for chunk in self.raw.stream(chunk_size, decode_content=True):
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 461, in stream
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     for line in self.read_chunked(amt, decode_content=decode_content):
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 637, in read_chunked
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     self._update_chunk_length()
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/cachecontrol/adapter.py", line 106, in _update_chunk_length
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     self._fp._close()
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/cachecontrol/filewrapper.py", line 50, in _close
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     self.__callback(self.__buf.getvalue())
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/cachecontrol/controller.py", line 275, in cache_response
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     self.serializer.dumps(request, response, body=body),
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/cachecontrol/caches/file_cache.py", line 99, in set
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     with self.lock_class(name) as lock:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/lockfile/__init__.py", line 197, in __enter__
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     self.acquire()
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/lockfile/linklockfile.py", line 21, in acquire
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     raise LockFailed("failed to create %s" % self.unique_name)
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3: lockfile.LockFailed: failed to create /.cache/cwltool/2/4/d/2/5/celery-worker-6679c5868f-fstcx-27575700.122614218408434280774
[step md5] Cannot make job: ../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3: Traceback (most recent call last):
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/lockfile/linklockfile.py", line 19, in acquire
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     open(self.unique_name, "wb").close()
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3: FileNotFoundError: [Errno 2] No such file or directory: "/.cache/cwltool/2/4/d/2/5/celery-worker-6679c5868f-fstcx-27575700.122614218408434280774"
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3: During handling of the above exception, another exception occurred:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3: Traceback (most recent call last):
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/app/src/cwl-tes/cwl_tes/tes.py", line 126, in visit
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     deref = downloadHttpFile(path)
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/cwltool-1.0.20181114114354-py3.6.egg/cwltool/pathmapper.py", line 157, in downloadHttpFile
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     for chunk in r.iter_content(chunk_size=16384):
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 750, in generate
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     for chunk in self.raw.stream(chunk_size, decode_content=True):
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 461, in stream
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     for line in self.read_chunked(amt, decode_content=decode_content):
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 637, in read_chunked
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     self._update_chunk_length()
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/cachecontrol/adapter.py", line 106, in _update_chunk_length
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     self._fp._close()
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/cachecontrol/filewrapper.py", line 50, in _close
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     self.__callback(self.__buf.getvalue())
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/cachecontrol/controller.py", line 275, in cache_response
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     self.serializer.dumps(request, response, body=body),
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/cachecontrol/caches/file_cache.py", line 99, in set
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     with self.lock_class(name) as lock:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/lockfile/__init__.py", line 197, in __enter__
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     self.acquire()
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/lockfile/linklockfile.py", line 21, in acquire
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     raise LockFailed("failed to create %s" % self.unique_name)
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3: lockfile.LockFailed: failed to create /.cache/cwltool/2/4/d/2/5/celery-worker-6679c5868f-fstcx-27575700.122614218408434280774

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/lockfile/linklockfile.py", line 19, in acquire
    open(self.unique_name, "wb").close()
FileNotFoundError: [Errno 2] No such file or directory: "/.cache/cwltool/2/4/d/2/5/celery-worker-6679c5868f-fstcx-27575700.122614218408434280774"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/src/cwl-tes/cwl_tes/tes.py", line 126, in visit
    deref = downloadHttpFile(path)
  File "/usr/local/lib/python3.6/site-packages/cwltool-1.0.20181114114354-py3.6.egg/cwltool/pathmapper.py", line 157, in downloadHttpFile
    for chunk in r.iter_content(chunk_size=16384):
  File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 750, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 461, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 637, in read_chunked
    self._update_chunk_length()
  File "/usr/local/lib/python3.6/site-packages/cachecontrol/adapter.py", line 106, in _update_chunk_length
    self._fp._close()
  File "/usr/local/lib/python3.6/site-packages/cachecontrol/filewrapper.py", line 50, in _close
    self.__callback(self.__buf.getvalue())
  File "/usr/local/lib/python3.6/site-packages/cachecontrol/controller.py", line 275, in cache_response
    self.serializer.dumps(request, response, body=body),
  File "/usr/local/lib/python3.6/site-packages/cachecontrol/caches/file_cache.py", line 99, in set
    with self.lock_class(name) as lock:
  File "/usr/local/lib/python3.6/site-packages/lockfile/__init__.py", line 197, in __enter__
    self.acquire()
  File "/usr/local/lib/python3.6/site-packages/lockfile/linklockfile.py", line 21, in acquire
    raise LockFailed("failed to create %s" % self.unique_name)
lockfile.LockFailed: failed to create /.cache/cwltool/2/4/d/2/5/celery-worker-6679c5868f-fstcx-27575700.122614218408434280774

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/cwltool-1.0.20181114114354-py3.6.egg/cwltool/workflow.py", line 789, in job
    runtimeContext):
  File "/usr/local/lib/python3.6/site-packages/cwltool-1.0.20181114114354-py3.6.egg/cwltool/command_line_tool.py", line 409, in job
    reffiles, builder.stagedir, runtimeContext, True)
  File "/app/src/cwl-tes/cwl_tes/tes.py", line 61, in make_path_mapper
    runtimeContext.make_fs_access(self.remote_storage_url or ""))
  File "/app/src/cwl-tes/cwl_tes/tes.py", line 83, in __init__
    separateDirs)
  File "/usr/local/lib/python3.6/site-packages/cwltool-1.0.20181114114354-py3.6.egg/cwltool/pathmapper.py", line 241, in __init__
    self.setup(dedup(referenced_files), basedir)
  File "/usr/local/lib/python3.6/site-packages/cwltool-1.0.20181114114354-py3.6.egg/cwltool/pathmapper.py", line 298, in setup
    self.visit(fob, stagedir, basedir, copy=fob.get("writable"), staged=True)
  File "/app/src/cwl-tes/cwl_tes/tes.py", line 143, in visit
    copy=copy, staged=staged)
  File "/usr/local/lib/python3.6/site-packages/schema_salad/sourceline.py", line 166, in __exit__
    raise self.makeError("
".join(traceback.format_exception(exc_type, exc_value, tb)))
schema_salad.validate.ValidationException: ../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3: Traceback (most recent call last):
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/lockfile/linklockfile.py", line 19, in acquire
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     open(self.unique_name, "wb").close()
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3: FileNotFoundError: [Errno 2] No such file or directory: "/.cache/cwltool/2/4/d/2/5/celery-worker-6679c5868f-fstcx-27575700.122614218408434280774"
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3: During handling of the above exception, another exception occurred:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3: Traceback (most recent call last):
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/app/src/cwl-tes/cwl_tes/tes.py", line 126, in visit
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     deref = downloadHttpFile(path)
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/cwltool-1.0.20181114114354-py3.6.egg/cwltool/pathmapper.py", line 157, in downloadHttpFile
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     for chunk in r.iter_content(chunk_size=16384):
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 750, in generate
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     for chunk in self.raw.stream(chunk_size, decode_content=True):
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 461, in stream
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     for line in self.read_chunked(amt, decode_content=decode_content):
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 637, in read_chunked
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     self._update_chunk_length()
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/cachecontrol/adapter.py", line 106, in _update_chunk_length
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     self._fp._close()
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/cachecontrol/filewrapper.py", line 50, in _close
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     self.__callback(self.__buf.getvalue())
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/cachecontrol/controller.py", line 275, in cache_response
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     self.serializer.dumps(request, response, body=body),
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/cachecontrol/caches/file_cache.py", line 99, in set
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     with self.lock_class(name) as lock:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/lockfile/__init__.py", line 197, in __enter__
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     self.acquire()
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/lockfile/linklockfile.py", line 21, in acquire
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     raise LockFailed("failed to create %s" % self.unique_name)
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3: lockfile.LockFailed: failed to create /.cache/cwltool/2/4/d/2/5/celery-worker-6679c5868f-fstcx-27575700.122614218408434280774

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/cwltool-1.0.20181114114354-py3.6.egg/cwltool/workflow.py", line 469, in job
    for newjob in step.iterable:
  File "/usr/local/lib/python3.6/site-packages/cwltool-1.0.20181114114354-py3.6.egg/cwltool/workflow.py", line 401, in try_make_job
    for j in jobs:
  File "/usr/local/lib/python3.6/site-packages/cwltool-1.0.20181114114354-py3.6.egg/cwltool/workflow.py", line 204, in job
    for j in self.step.job(joborder, output_callback, runtimeContext):
  File "/usr/local/lib/python3.6/site-packages/cwltool-1.0.20181114114354-py3.6.egg/cwltool/workflow.py", line 796, in job
    raise WorkflowException(Text(exc))
cwltool.errors.WorkflowException: ../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3: Traceback (most recent call last):
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/lockfile/linklockfile.py", line 19, in acquire
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     open(self.unique_name, "wb").close()
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3: FileNotFoundError: [Errno 2] No such file or directory: "/.cache/cwltool/2/4/d/2/5/celery-worker-6679c5868f-fstcx-27575700.122614218408434280774"
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3: During handling of the above exception, another exception occurred:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3: Traceback (most recent call last):
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/app/src/cwl-tes/cwl_tes/tes.py", line 126, in visit
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     deref = downloadHttpFile(path)
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/cwltool-1.0.20181114114354-py3.6.egg/cwltool/pathmapper.py", line 157, in downloadHttpFile
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     for chunk in r.iter_content(chunk_size=16384):
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 750, in generate
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     for chunk in self.raw.stream(chunk_size, decode_content=True):
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 461, in stream
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     for line in self.read_chunked(amt, decode_content=decode_content):
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/urllib3/response.py", line 637, in read_chunked
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     self._update_chunk_length()
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/cachecontrol/adapter.py", line 106, in _update_chunk_length
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     self._fp._close()
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/cachecontrol/filewrapper.py", line 50, in _close
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     self.__callback(self.__buf.getvalue())
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/cachecontrol/controller.py", line 275, in cache_response
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     self.serializer.dumps(request, response, body=body),
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/cachecontrol/caches/file_cache.py", line 99, in set
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     with self.lock_class(name) as lock:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/lockfile/__init__.py", line 197, in __enter__
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     self.acquire()
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:   File "/usr/local/lib/python3.6/site-packages/lockfile/linklockfile.py", line 21, in acquire
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:     raise LockFailed("failed to create %s" % self.unique_name)
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3:
../../output/VLE1V9/workflow_files/hashsplitter-workflow.yml:2:3: lockfile.LockFailed: failed to create /.cache/cwltool/2/4/d/2/5/celery-worker-6679c5868f-fstcx-27575700.122614218408434280774
[workflow ] completed permanentFail
[workflow ] {
    "output": null
}
{
    "output": null
}
Final process status is permanentFail
uniqueg commented 4 years ago

Couldn't be reproduced on c03 cluster with the latest version (changes between #143 and #148), so will close. Re-open if issue still persists or re-occurs

uniqueg commented 4 years ago

Still shows up on c03 when workflow, with auth, is submitted via CLA

lvarin commented 4 years ago

I think the problem is that the $HOME directory of the user in Openshift is /. Should be easily solvable (famous last words).

lvarin commented 4 years ago

After the home directory is created (and using no auth), the error seems to be gone:

$ curl -X POST --header 'Content-Type: multipart/form-data' --header 'Accept: application/json' -F workflow_params='{"input":{"class":"File","path":"ftp://ftp-private.ebi.ac.uk/upload/foivos/test.txt"}}' -F workflow_type=CWL -F workflow_type_version=v1.0 -F workflow_url='https://github.com/uniqueg/cwl-example-workflows/blob/master/hashsplitter-workflow.cwl' 'https://cwlwes.csc-wes.rahtiapp.fi/ga4gh/wes/v1/runs'
{                                                     
  "run_id": "SKP32H"
}

Then,it fails somewhere else:

 curl -X GET --header 'Accept: application/json' 'https://cwlwes.csc-wes.rahtiapp.fi/ga4gh/wes/v1/runs/SKP32H'
{
  "outputs": {},
  "request": {
    "workflow_params": {
      "input": {
        "class": "File",
        "path": "ftp://ftp-private.ebi.ac.uk/upload/foivos/test.txt"
      }
    },
    "workflow_type": "CWL",
    "workflow_type_version": "v1.0",
    "workflow_url": "https://github.com/uniqueg/cwl-example-workflows/blob/master/hashsplitter-workflow.cwl"
  },
  "run_log": {
    "command": "cwl-tes --debug --leave-outputs --remote-storage-url ftp://ftp-private.ebi.ac.uk/upload/foivos --tes https://csc-tesk.c03.k8s-popup.csc.fi/ /data/output/SKP32H/workflow_files/repo/hashsplitter-workflow.cwl /data/output/SKP32H/workflow_files/hashsplitter-workflow.yml",
    "exception": "OSError(107, 'Transport endpoint is not connected')",
    "expires": null,
    "max_retries": 0,
    "task_finished": "2020-04-30 05:40:13.031092",
    "task_received": "2020-04-30 05:40:13.024255",
    "task_started": "2020-04-30 05:40:13.025183",
    "time_execution": 0.006,
    "time_queue": 0.001,
    "time_total": 0.007,
    "utc_offset": 0
  },
  "state": "SYSTEM_ERROR",
  "task_logs": []
}
uniqueg commented 4 years ago

Thanks @lvarin! I have never seen this error and unfortunately it's not very informative. Is this on Rahti? What do the Flask and Celery container logs say? Let's take it to Slack

lvarin commented 4 years ago

So the image lvarin/cwl-wes:fake-home should have the changes that fix this ticket. I will activate the auth, solve problems is any and then do a PR :)