search

broadinstitute / cromwell

Scientific workflow engine designed for simplicity & scalability. Trivially transition between one off use cases to massive scale production environments
http://cromwell.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
996 stars 361 forks source link

How does cromwell choose the backend #3533

Open EvanTheB opened 6 years ago

EvanTheB commented 6 years ago

I tried to ask this on the broad forums but got an error."You need the Garden.Community.Manage permission to do that.", "Class": "Gdn_UserException"

I have been using a

backend.default = SGE

Line in my config file. This has been working until now, with my jobs submitted using my defined SGE backend. Today I changed some other parts of the server, unrelated and numerous. Now SOME jobs are being submitted on the SGE backend, and some on the Local backend. (I did not configure a local backend, but I assume one is built in to cromwell.

How is the backend to run on chosen by cromwell? (I know of backend.default, and I beleive there is a wdl.task.runtime.backend parameter also (undocumented from what I can tell)).

How can I prevent any task ever running on the local backend? I want this to be a hard error, and not overload my sge login node.

Thanks

tmm211 commented 6 years ago

4/19 PM Hi Evan, a patch went out to fix this at 10 AM this morning. Can you confirm that you no longer see this?

4/19 AM Hi Evan - were you signed into the forum when you got this error? Can you send me the url of the page you were on?

ruchim commented 6 years ago

Hello @EvanTheB,

The order of operations to choose a backend are:

  1. Check if a default backend is declared within the workflow options.
  2. Check if the task definition includes the runtime attribute key backend.
  3. Check if a default backend is declared.

Based on your issue, it sounds like you're not submitting a workflow options file, or even using the runtime attribute backend in your tasks. As a way to debug, would you mind posting the output from running a GET of /backends endpoint to ensure your default is set properly.

Additionally, just a clarification -- when you say "SOME jobs are being submitted on the SGE backend, and some on the Local backend", do you some jobs within the same workflow are going to both backends?

EvanTheB commented 6 years ago

I have not reproduced the issue, I will try when I get a chance here. At the time I was modifying my backend's 'runtime-attributes'. I made all those attributes optional. I also removed all the 'runtime' stanzas in the wdl file I was testing. I believe there was an error in the 'submit' syntax of my backend config.

Is it possible a config file that fails to parse will cause backends to default to the Local backend? I know parts of the config are not parsed until a job is submitted. I will try to isolate the problem.

Thanks