search

common-workflow-language / cwlviewer

A web application to view and share Common Workflow Language workflows
https://view.commonwl.org/
Apache License 2.0
44 stars 28 forks source link

valid workflow fails "An internal server error occurred" #419

Closed mr-c closed 2 years ago

mr-c commented 2 years ago

Description

https://github.com/athenarc/whole-exome-seq/blob/main/workflows/Whole_Exome-Seq.cwl fails to visualize

Expected Behavior

Whole_Exome-Seq

Actual Behavior

Whoops - Something Went Wrong! Error: An internal server error occurred

Steps to Reproduce

Local rendering with cwltool works: cwltool: cwltool --print-dot https://github.com/athenarc/whole-exome-seq/raw/main/workflows/Whole_Exome-Seq.cwl

kinow commented 2 years ago

I used the main branch, where I found a problem with the entrypoint of the updated Dockerfile, but I think that hasn't been deployed to production yet.

Updating the entrypoint command, and using the URL https://github.com/athenarc/whole-exome-seq/blob/main/workflows/Whole_Exome-Seq.cwl, it is imported successfully in my local environment.

image

image

image

Going to use the 1.4.3 tag now.

kinow commented 2 years ago

Exception in 1.4.3 Docker compose logs:

postgres_1_e5bdf7fd818a | 2022-06-08 11:00:17.832 UTC [57] LOG:  execute S_2: COMMIT
spring_1_2d9b71585be6 | 2022-06-08 11:00:18,021 ERROR [task-1] org.commonwl.view.cwl.CWLToolRunner: ERROR I'm sorry, I couldn't load this CWL file, try again with --debug for more information.
spring_1_2d9b71585be6 | The error was: No plugin registered for (json-ld, <class 'rdflib.parser.Parser'>)
spring_1_2d9b71585be6 | 
spring_1_2d9b71585be6 | org.commonwl.view.cwl.CWLValidationException: ERROR I'm sorry, I couldn't load this CWL file, try again with --debug for more information.
spring_1_2d9b71585be6 | The error was: No plugin registered for (json-ld, <class 'rdflib.parser.Parser'>)
spring_1_2d9b71585be6 | 
spring_1_2d9b71585be6 |     at org.commonwl.view.cwl.CWLTool.runCwltoolOnWorkflow(CWLTool.java:121)
spring_1_2d9b71585be6 |     at org.commonwl.view.cwl.CWLTool.getRDF(CWLTool.java:50)
spring_1_2d9b71585be6 |     at org.commonwl.view.cwl.CWLService.parseWorkflowWithCwltool(CWLService.java:293)
spring_1_2d9b71585be6 |     at org.commonwl.view.cwl.CWLToolRunner.createWorkflowFromQueued(CWLToolRunner.java:86)
spring_1_2d9b71585be6 |     at org.commonwl.view.cwl.CWLToolRunner$$FastClassBySpringCGLIB$$cd465805.invoke(<generated>)
spring_1_2d9b71585be6 |     at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
spring_1_2d9b71585be6 |     at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783)
spring_1_2d9b71585be6 |     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
spring_1_2d9b71585be6 |     at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
spring_1_2d9b71585be6 |     at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
spring_1_2d9b71585be6 |     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
spring_1_2d9b71585be6 |     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
spring_1_2d9b71585be6 |     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
spring_1_2d9b71585be6 |     at java.lang.Thread.run(Thread.java:748)
bash-4.4# cwltool --version
/usr/bin/cwltool 3.1.20220502060230
kinow commented 2 years ago

Installed cwltool with pip install cwltool==3.1.20220502060230. Then cwltool --non-strict --quiet --skip-schemas --pack https://raw.githubusercontent.com/athenarc/whole-exome-seq/main/workflows/Whole_Exome-Seq.cwl. Executed successfully.

kinow commented 2 years ago

Tried the other command executed in CWLViewer's CWLTool, cwltool --non-strict --quiet --skip-schemas --print-rdf https://raw.githubusercontent.com/athenarc/whole-exome-seq/main/workflows/Whole_Exome-Seq.cwl.

Also executed successfully :exploding_head: (i.e. no errors in the terminal, echo $? also returning 0)

kinow commented 2 years ago

Ah, I was using the raw URL. Reproduced the error in cwltool on my local environment (Python 3.9, @mr-c ):

(venv) kinow@ranma:/tmp$ cwltool --non-strict --quiet --skip-schemas --print-rdf https://github.com/athenarc/whole-exome-seq/blob/main/workflows/Whole_Exome-Seq.cwl
While fetching https://github.com/athenarc/whole-exome-seq/blob/main/workflows/Whole_Exome-Seq.cwl, got content-type of 'text/html'. Expected one of ['text/plain', 'application/json', 'text/vnd.yaml', 'text/yaml', 'text/x-yaml', 'application/x-yaml', 'application/octet-stream'].
ERROR Tool definition failed validation:
https://github.com/athenarc/whole-exome-seq/blob/main/workflows/Whole_Exome-Seq.cwl:28:66: mapping values are
                                                                                           not allowed here
(venv) kinow@ranma:/tmp$ cwltool --version
/tmp/venv/bin/cwltool 3.1.20220502060230

(venv) kinow@ranma:/tmp$ python
Python 3.9.12 | packaged by conda-forge | (main, Mar 24 2022, 23:25:59) 
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

(not sure if this is an issue in cwltool, or just CWLViewer forgetting to use git to checkout the workflow… will probably need to restart working on it in the morning after :coffee: )

kinow commented 2 years ago

Huh, scary, now I realized the whole CWLViewer app is stuck. The server appears to be processing the steps in that workflow, even after it failed to execute cwltool. Either a loop, or something that is taking a really long time to execute.

mr-c commented 2 years ago

Exception in 1.4.3 Docker compose logs:

postgres_1_e5bdf7fd818a | 2022-06-08 11:00:17.832 UTC [57] LOG:  execute S_2: COMMIT
spring_1_2d9b71585be6 | 2022-06-08 11:00:18,021 ERROR [task-1] org.commonwl.view.cwl.CWLToolRunner: ERROR I'm sorry, I couldn't load this CWL file, try again with --debug for more information.
spring_1_2d9b71585be6 | The error was: No plugin registered for (json-ld, <class 'rdflib.parser.Parser'>)
spring_1_2d9b71585be6 | 
spring_1_2d9b71585be6 | org.commonwl.view.cwl.CWLValidationException: ERROR I'm sorry, I couldn't load this CWL file, try again with --debug for more information.
spring_1_2d9b71585be6 | The error was: No plugin registered for (json-ld, <class 'rdflib.parser.Parser'>)
spring_1_2d9b71585be6 | 
spring_1_2d9b71585be6 |   at org.commonwl.view.cwl.CWLTool.runCwltoolOnWorkflow(CWLTool.java:121)
spring_1_2d9b71585be6 |   at org.commonwl.view.cwl.CWLTool.getRDF(CWLTool.java:50)
spring_1_2d9b71585be6 |   at org.commonwl.view.cwl.CWLService.parseWorkflowWithCwltool(CWLService.java:293)
spring_1_2d9b71585be6 |   at org.commonwl.view.cwl.CWLToolRunner.createWorkflowFromQueued(CWLToolRunner.java:86)
spring_1_2d9b71585be6 |   at org.commonwl.view.cwl.CWLToolRunner$$FastClassBySpringCGLIB$$cd465805.invoke(<generated>)
spring_1_2d9b71585be6 |   at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
spring_1_2d9b71585be6 |   at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783)
spring_1_2d9b71585be6 |   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
spring_1_2d9b71585be6 |   at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
spring_1_2d9b71585be6 |   at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
spring_1_2d9b71585be6 |   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
spring_1_2d9b71585be6 |   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
spring_1_2d9b71585be6 |   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
spring_1_2d9b71585be6 |   at java.lang.Thread.run(Thread.java:748)
bash-4.4# cwltool --version
/usr/bin/cwltool 3.1.20220502060230

Thanks for reproducing this! Odd, that cwltool version matches the latest on quay.io, which is Python 3.10 based and shouldn't have that error: https://quay.io/repository/commonwl/cwltool/manifest/sha256:339d5687c91608e75016d65fcdb7dec8d5d24b4ac9e3547766a1db2faa5806d4

Ah, quay.io/commonwl/cwltool is not the container being used; it is https://github.com/common-workflow-language/cwlviewer/blob/d1646502a3a38eae0435bb2c4efe9662a4640154/Dockerfile#L1 → https://quay.io/repository/commonwl/cwlviewer/manifest/sha256:f44d3bb01a3f1b2818ac5b15dfaab4d6f4718dbc0b0191340f7a3ab825d8260d

Ah, I was using the raw URL. Reproduced the error in cwltool on my local environment (Python 3.9, @mr-c ):


(venv) kinow@ranma:/tmp$ cwltool --non-strict --quiet --skip-schemas --print-rdf https://github.com/athenarc/whole-exome-seq/blob/main/workflows/Whole_Exome-Seq.cwl
While fetching https://github.com/athenarc/whole-exome-seq/blob/main/workflows/Whole_Exome-Seq.cwl, got content-type of 'text/html'. Expected one of ['text/plain', 'application/json', 'text/vnd.yaml', 'text/yaml', 'text/x-yaml', 'application/x-yaml', 'application/octet-stream'].
ERROR Tool definition failed validation:
https://github.com/athenarc/whole-exome-seq/blob/main/workflows/Whole_Exome-Seq.cwl:28:66: mapping values are
                                                                                           not allowed here

Yes, that should fail. You asked cwltool to parse a webpage; which is why I was using the raw URL.

mr-c commented 2 years ago

Yep, the container's base image has an old Python version (3.6.9)

$ docker run  quay.io/commonwl/cwlviewer python3 -V
Python 3.6.9
mr-c commented 2 years ago

I'm rebuilding the container locally; it now pulls in a newer Python (3.9.7); I'm also making a new cwltool release that should fix the issue for Python 3.6

mr-c commented 2 years ago

New cwltool release with fixed Python 3.6 support: https://github.com/common-workflow-language/cwltool/releases/tag/3.1.20220607081835 & https://pypi.org/project/cwltool/3.1.20220607081835/#files

The new cwlviewer version 1.4.4 container is available via docker.io/commonworkflowlanguage/cwlviewer:v1.4.4 & quay.io/commonwl/cwlviewer:v1.4.4

It contains the new cwltool release and it uses Python 3.10; so once view.commonwl.org is refreshed, this issue can be closed

mr-c commented 2 years ago

Success! https://view.commonwl.org/workflows/github.com/athenarc/whole-exome-seq/blob/main/workflows/Whole_Exome-Seq.cwl

Thank you @kinow and @cure !