Problems when deploying web service (NLprot)

martavillegas commented 6 years ago

We have deployed a WS as wrapper for a command line tool (NLProt) that reads PDF and returns a JSON (we understood that XMI was recommended but not compulsory).

you can run it: curl -F cas=@input-test.pdf inb.bsc.es/service/openminted/process

input-test.pdf

When running the app in test.openminted we get the following message:

Failed Unable to locate named workflow

gkirtzou commented 6 years ago

@martavillegas the error you have mentioned is a know issue in the test.openminted.eu for the software that is registered as application (and is reported to the responsible tech team in the redmine, see issue 840)

Independent of that, we need first to insure that the software you provide is OMTD compliant to the specifications and guidelines for web services. The OMTD specifications and guidelines for a web services require that both the input and output are corpora of UIMA CAS XMI files. Furthermore, in order to deserialize correct the output of the web service it requires a typesystem, which is available in the maven central repository.

Given the metadata you have provided us, in order for your software to run within OMTD platform, you would need to make it OMTD compliant. In other words,

you need to generate UIMA CAS XMI files as output, not JSON
you need to define what typesystem the XMI CAS file use so as to be deserialized correctly and
make that typesystem available via the maven central repository.

For more details see here: 1) Guidelines : https://guidelines.openminted.eu/guidelines_for_providers_of_sw_resources/sharing-web-services.html 2) Specifications : https://openminted.github.io/releases/processing-web-services/1.0.0/specification

martavillegas commented 6 years ago

Thanks! just two questions:

we understood that apps differ from components in that they did not to consume/return XMI. Is this right? (that's why we changed to app)
can we consume PDF files directly?

gkirtzou commented 6 years ago

@martavillegas To answer your first question what is the difference between a component and an application and what an application or a component can consume/produce, the OMTD interoperability specification say that :

components that support (i.e. produce & consume) data in the XML Metadata Interchange (XMI) format; in the case of components that support other formats, appropriate readers and writers for converting XMI messages from and to the components’ format must also be provided

applications that support as input file formats that are used for publications (e.g. PDF, PubMed XML, plain text etc.) or XMI.

In general, both components and applications can consume/produce whatever they want. For the components it is advisable to offer other components converters (readers/writers from/to XMI) for interoperability reasons. ie to allow a TDM expert to combine components from different providers/architectures.

Note that an application can be create by two different ways: 1) Via the Workflow editor, where a TDM expert combines interoperable components or 2) By registering a single component as an application.

In this second case, where a single component is registered as application, the way the OMTD workflow service handles them makes the assumption that the application consumes XMI, and thus always executes before the component-application the following two components: 1) the omtd importer so as to fetch the data from the OMTD store to OMTD workflow executor 2) and the PDF reader component to transform PDF files to XMI.

As for the second question, since you have decided to offer your software as a web service, you need to make your component compliant not to the generic specifications, but to the web service ones. Which means that you need to: 1) ensure that it uses UIMA CAS XMI messaging system as specified at https://openminted.github.io/releases/processing-web-services/1.0.0/specification and 2) upload the typesystem that the web service deploys in Maven.

So to answer your second question, you cannot read PDF files directly if you are offering your software as web service. If you want to read PDF files directly, you could register your software as component of docker image and create an application using the workflow editor so that you can execute your software. The instructions how you can build a workflow can be found here: https://openminted.github.io/releases/workflow-editor/1.0.0/workflow

martavillegas commented 6 years ago

Hi Katerina, we are back....

Our NLProt service reads and delivers MXI with DKPro Core type system. in this case do we need to uoload the typesystem (we understand it is already uploaded).

Thanks!

gkirtzou commented 6 years ago

@martavillegas

Our NLProt service reads and delivers MXI with DKPro Core type system. in this case do we need to uoload the typesystem (we understand it is already uploaded).

That's great! In order to be compliant you need to update the metadata of the application/component by adding the typesystem details, as the guidelines mention:

Please, note that in the metadata record, you must include (1) the URL where the web service can be executed in the element distributionLocation, e.g.
<distributionLocation>http://nactem.ac.uk/api/openminted/chebi</distributionLocation> (2) the typesystem details in the the module componentDependencies/typesystem and more specifically in the elements resourceName and resourceIdentifier (Maven coordinates), e.g.
<resourceName>Chebi Curation Type System</resourceName> and <resourceIdentifier resourceIdentifierSchemeName="maven">mvn:uk.ac.nactem.uima:ChebiCurationTypeSystem:0.1</resourceIdentifier>

Then you could try again to call you web service and see whether it works as expected.

martavillegas commented 6 years ago

Sorry Katerina, but in case we use default type system, namely:

xmlns:type="http:///de/tudarmstadt/ukp/dkpro/core/api/metadata/type.ecore" xmlns:type2="http:///de/tudarmstadt/ukp/dkpro/core/api/segmentation/type.ecore"

do we need to upload it in Maven?

reckart commented 6 years ago

The DKPro Core type system is split over various "API" artifacts on Maven: http://search.maven.org/#search%7Cga%7C1%7Cde.tudarmstadt.ukp.dkpro.core.api

galanisd commented 6 years ago

@reckart just said the one thing that I wanted to say

The DKPro Core type system is split over various "API" artifacts on Maven: http://search.maven.org/#search%7Cga%7C1%7Cde.tudarmstadt.ukp.dkpro.core.api

The typesystem that you use is already uploaded to maven central. So you have just specify in the metadata its coordinates. E.g. like

a. chebi web service

https://services.openminted.eu/landingPage/application/11dc2e5e-1cd8-40bf-a46f-df17664b3c77 -> mvn:uk.ac.nactem.uima:ChebiCurationTypeSystem:0.1

https://mvnrepository.com/artifact/uk.ac.nactem.uima/ChebiCurationTypeSystem/0.1

b. the OGER web service https://services.openminted.eu/landingPage/application/71345d18-297f-4ac5-b4de-38ef3cacbe75 -> mvn:de.tudarmstadt.ukp.dkpro.core:de.tudarmstadt.ukp.dkpro.core.api.ner-asl:1.9.1

https://mvnrepository.com/artifact/de.tudarmstadt.ukp.dkpro.core/de.tudarmstadt.ukp.dkpro.core.api.ner-asl/1.9.1

martavillegas commented 6 years ago

Thanks guys with the types!!

I updated the metadata with types (two coordinated from maven) but now when I try to run the application I get the following error message:

System error executing service (Server responded: #: 2 schema violations found)

gkirtzou commented 6 years ago

@martavillegas Would be possible to send the xml file of the metadata with us, to check them?

martavillegas commented 6 years ago

Sure! 2c448b7c-6fe2-451f-b9d3-1426330251ef (2).zip

galanisd commented 6 years ago

We have checked the workflow engine (Galaxy); it seems that the Galaxy wrapper that is required has been generated as it should. The wrapper contains the command that will be executed from the generic web service executor that we have designed. The command seems correct and the executor was designed to support more that one typesystems. I remember also that I have tested it in such a case and it worked.

Also we checked Galaxy (with @gkirtzou) and it seems that nothing has been executed the last few days. So I assume (not sure 100%) that the error in the Registry side, in particular in the point where the workflow execution is initiated. I think that we should ask @antleb.

Please could you test the same app also in services.openminted.eu? test.openminted usually is not as stable as services.

Dimitris

martavillegas commented 6 years ago

I registered the tool in services. When I run it I get:

There was a problem running the application. Try again in a while. (Unable to locate named workflow)

gkirtzou commented 6 years ago

@martavillegas This is an know problem that used to be in test platform, maybe has been moved in the services as well (see here : http://redmine.openminted.eu/issues/840)

You can avoid this, if you register your software as a component and create a simple workflow using the workflow editor by adding the omtd_importer and the pdf reader (with parameter pattern="*/.pdf"). In other words, by creating the parts that the platform add by itself if you register something as application. For more details how you can create a workflow please check here: https://openminted.github.io/releases/workflow-editor/1.0.0/workflow

martavillegas commented 6 years ago

Ok, I registered the service as a component. When I go to galaxy editor I have one problem: When I search for NLProt I get one result ("NLProt by kat") but if I browse I find three NLProts, the 'one by kat' and two identical NLProts (mine I guess)

martavillegas commented 6 years ago

I created a workflow, with PDFreader and NLProt. When I go to apps to run it I find two NLProt applications....

I tried to run the first one and got the message: There was a problem running the application. Try again in a while. (Unable to locate named workflow) I received an email message with a link. When I follow the link I get a white screen with this message: {"url":"http://services.openminted.eu/omtd-registry/mySpace/operations","error":"No handler found for GET /omtd-registry/mySpace/operations"}

then I tried to run the second one and it is still running (no error messages but running for a long time. I used the 2 pdf files corpus used for testing with Chebi web service

gkirtzou commented 6 years ago

@martavillegas I have delete my private NLprot component ("NLProt by kat") so that you don't get confused.

I tried to run the first one and got the message: There was a problem running the application. Try again in a while. (Unable to locate named workflow) I received an email message with a link. When I follow the link I get a white screen with this message: {"url":"http://services.openminted.eu/omtd-registry/mySpace/operations","error":"No handler found for GET /omtd-registry/mySpace/operations"}

I don't know what is causing this error. I would suggest to report it to the redmine, so that someone could have a look at it

then I tried to run the second one and it is still running (no error messages but running for a long time. I used the 2 pdf files corpus used for testing with Chebi web service

I found a workflow with nlprot in the webservice engine, but it hasn't load any documents. Are you sure you followed the guidelines I sent you? Meaning is the first step of the workflow omtdImporter, the second step is Pdf Reader with parameter patterns="**/.pdf" and then NLprot????

martavillegas commented 6 years ago

Hi,

sorry I'm afraid the initial workflow was wrong... I edited it again (following your instructions) and I get this message, thanks:

There was a problem running the application. Try again in a while. (a workflow state is in error: b445e1e784215eb6/a4128fc2307c4024/f44bd52ac458d92a/5c4779f75d735313)

gkirtzou commented 6 years ago

@martavillegas did you initialize the parameter patterns with */.pdf in the PDFreader? Because I found the workflow, but the PDFReader produced zero xmi files.

martavillegas commented 6 years ago

I get the same error.....

gkirtzou commented 6 years ago

The log from pdf reader shows that the pattern is patterns="/.pdf", not /*.pdf. Thus zero files matched the pattern and the workflow ended up with an error. Maybe you added it, but sometimes, galaxy editor/registry don't save it correctly. Could you verify by opening the workflow that the pattern is set correctly? And try again?

martavillegas commented 6 years ago

I cannot edit the parameters. When I try to edit the workflow, galaxy does not display the parameters on the right panel. What I do is to delete the pdfreader and add another one, then I can edit. The screenshot shows the empty pannel....

I'm gona delete this workflow and create a new one ...

martavillegas commented 6 years ago

I sent you the wrong schreenshot... the thing is that when I try to edit an existing workflow I get nothing on the right panel....

martavillegas commented 6 years ago

Sorry, it failed again with the newly create workflow. I got this message:

There was a problem running the application. Try again in a while. (a workflow state is in error: 5b590cfdc83940a2/c5e4e2683b6e9022/d6f6ae8e19773cf9/349ddf9556a98b88)

gkirtzou commented 6 years ago

PDF reader worked this time, so it produced XMI files from the PDFs. Below you would find the log from the omtd-webservice client. It seems that there is a problem with the webservice URI. It is correct?

Executing..
========================
className:null
inputDir:tmp
outputDir:/srv/galaxy/database/jobs_directory/006/6080/working/out/
**parameters
wsurl:http://inb.bsc.es/service/openminted
typesystems:de.tudarmstadt.ukp.dkpro.core_de.tudarmstadt.ukp.dkpro.core.api.metadata-asl_1.9.2
========================
========================

2018-07-13 14:04:34.927  INFO 53 --- [           main] d.t.ukp.dkpro.core.io.xmi.XmiReader      : Scanning [file:/opt/omtd-component-executor/scripts/working/tmp/]
2018-07-13 14:04:34.932  INFO 53 --- [           main] d.t.ukp.dkpro.core.io.xmi.XmiReader      : Found [2] resources to be read
2018-07-13 14:04:36.142 ERROR 53 --- [           main] o.a.u.a.i.PrimitiveAnalysisEngine_impl   : Exception occurred

org.apache.uima.analysis_engine.AnalysisEngineProcessException: Annotator processing failed.    
    at org.apache.uima.analysis_engine.impl.PrimitiveAnalysisEngine_impl.callAnalysisComponentProcess(PrimitiveAnalysisEngine_impl.java:416) [uimaj-core-2.10.2.jar!/:2.10.2]
    at org.apache.uima.analysis_engine.impl.PrimitiveAnalysisEngine_impl.processAndOutputNewCASes(PrimitiveAnalysisEngine_impl.java:318) [uimaj-core-2.10.2.jar!/:2.10.2]
    at org.apache.uima.analysis_engine.impl.AnalysisEngineImplBase.process(AnalysisEngineImplBase.java:269) [uimaj-core-2.10.2.jar!/:2.10.2]
    at org.apache.uima.fit.pipeline.SimplePipeline.runPipeline(SimplePipeline.java:342) [uimafit-core-2.4.0.jar!/:2.4.0]
    at org.apache.uima.fit.pipeline.SimplePipeline.runPipeline(SimplePipeline.java:226) [uimafit-core-2.4.0.jar!/:2.4.0]
    at eu.openminted.component.webservice.WebServiceClient.run(WebServiceClient.java:51) [classes!/:0.0.1-SNAPSHOT]
    at eu.openminted.component.webservice.executor.WSExecutor.run(WSExecutor.java:45) [classes!/:0.0.1-SNAPSHOT]
    at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:800) [spring-boot-1.4.2.RELEASE.jar!/:1.4.2.RELEASE]
    at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:784) [spring-boot-1.4.2.RELEASE.jar!/:1.4.2.RELEASE]
    at org.springframework.boot.SpringApplication.afterRefresh(SpringApplication.java:771) [spring-boot-1.4.2.RELEASE.jar!/:1.4.2.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:316) [spring-boot-1.4.2.RELEASE.jar!/:1.4.2.RELEASE]
    at eu.openminted.component.webservice.executor.WSExecutor.main(WSExecutor.java:30) [classes!/:0.0.1-SNAPSHOT]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_161]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_161]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_161]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161]
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [omtd-component-webservice-0.0.1-SNAPSHOT-exec.jar:0.0.1-SNAPSHOT]
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [omtd-component-webservice-0.0.1-SNAPSHOT-exec.jar:0.0.1-SNAPSHOT]
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) [omtd-component-webservice-0.0.1-SNAPSHOT-exec.jar:0.0.1-SNAPSHOT]
    at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:521) [omtd-component-webservice-0.0.1-SNAPSHOT-exec.jar:0.0.1-SNAPSHOT]
Caused by: java.lang.IllegalArgumentException: URI is not absolute
    at java.net.URI.toURL(URI.java:1088) ~[na:1.8.0_161]
    at org.springframework.http.client.SimpleClientHttpRequestFactory.createRequest(SimpleClientHttpRequestFactory.java:137) ~[spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
    at org.springframework.http.client.support.HttpAccessor.createRequest(HttpAccessor.java:77) ~[spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:615) ~[spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:580) ~[spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
    at org.springframework.web.client.RestTemplate.getForObject(RestTemplate.java:287) ~[spring-web-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
    at eu.openminted.remoteexecution.client.RemoteComponent.waitForResult(RemoteComponent.java:112) ~[omtd-remote-execution-client-0.0.1-SNAPSHOT.jar!/:na]
    at eu.openminted.remoteexecution.client.RemoteComponent.process(RemoteComponent.java:66) ~[omtd-remote-execution-client-0.0.1-SNAPSHOT.jar!/:na]
    at org.apache.uima.analysis_component.JCasAnnotator_ImplBase.process(JCasAnnotator_ImplBase.java:48) ~[uimaj-core-2.10.2.jar!/:2.10.2]
    at org.apache.uima.analysis_engine.impl.PrimitiveAnalysisEngine_impl.callAnalysisComponentProcess(PrimitiveAnalysisEngine_impl.java:401) [uimaj-core-2.10.2.jar!/:2.10.2]
    ... 19 common frames omitted

martavillegas commented 6 years ago

will check with Miguel.... thanks!

martavillegas commented 6 years ago

Hi, I'm here again .... Miguel, fixed the "not absolute URIs" problem but it seems something is wrong. I just run the application and I get an error:

There was a problem running the application. Try again in a while. (a workflow state is in error: 5b590cfdc83940a2/358acff37bc4d34d/6ef404cf6a7da0be/8b6129c80e356c68)

martavillegas commented 6 years ago

@gkirtzou could you check our previous post? thanks!

gkirtzou commented 6 years ago

@martavillegas Sorry I didn't see your comment on time. In the workflow service engine, I found that you tried to run your workflow 2 days again and the workflow seems to have stacked at the NLProt component. Unfortunately, I couldn't find any log in the workflow engine slaves, that the docker (OMTD web service client) actually ran. Could you please give it another try to today and let me know, so I could check?

Also is the declared URL in the metadata the correct one? Or does it need to update?

martavillegas commented 6 years ago

Hi @gkirtzou thanks. I changed the URL and deleted the workflow just in case. When the workflow editor is back I'll build a new workflow and try again n(. Let you know... Thanks

martavillegas commented 6 years ago

Hi @gkirtzou ,still getting the "Sorry, our editor is down at the moment, please try again later." message when trying to build a workflow. Did I miss anything?

gkirtzou commented 6 years ago

@martavillegas I confirm that the workflow editor is down. I have no idea why it is not working. Please contact @antleb who is responsible for setting up the production site.

martavillegas commented 6 years ago

@antleb it seems the workflow editor is down (has been down since last Friday). Thanks

antleb commented 6 years ago

We are looking into it. I’ll let you know when it’s back online.

On 25 Jul 2018, at 10:34, martavillegas notifications@github.com wrote:

@antleb https://github.com/antleb it seems the workflow editor is down (has been down since last Friday). Thanks

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openminted/Open-Call-Discussions/issues/44#issuecomment-407662441, or mute the thread https://github.com/notifications/unsubscribe-auth/AALr2wgP3LCzCsbLqfKpbdUwQix04oGWks5uKB-JgaJpZM4UvVSV.

galanisd commented 6 years ago

The Galaxy editor instance is up. I think it has to do with the communication between Registry & Editor instance.

martavillegas commented 6 years ago

@galanisd @antleb I still get the same error message (at 09:53)

Sorry, our editor is down at the moment, please try again later.

greenwoodma commented 6 years ago

Could it be that someone has started implementing the proxy rules to block access to the rest of the Galaxy instance and the editor has been accidentally included? This would explain why it's up but not embedded in the UI.

martavillegas commented 6 years ago

Hi, I still get the same error message when trying to use the galaxy editor: "Sorry, our editor is down at the moment, please try again later.".

openminted / Open-Call-Discussions

Problems when deploying web service (NLprot) #44