search

SSHOC / sshoc-marketplace-backend

Code for the backend
Apache License 2.0
2 stars 0 forks source link

Response time to get workflows #432

Open KlausIllmayer opened 7 months ago

KlausIllmayer commented 7 months ago

Currently, there are 45 workflows on marketplace, each having some steps. When calling them on frontend (https://marketplace.sshopencloud.eu/search?categories=workflow), the response time is okay, because it takes in the overview the result set from the Solr. But when trying to get a direct list of all workflows with the API endpoint GET /api/workflows?perpage=100 it takes very long and currently is on the edge to 60 seconds response time. If it goes over 60 seconds response time, an unknown response code error "Failure when receiving data from the peer" is returned. In the log files I see that a broken pipe is the cause:

WARN  o.s.w.s.m.m.a.ExceptionHandlerExceptionResolver.doResolveHandlerMethodException - Failure in @ExceptionHandler eu.sshopencloud.marketplace.controllers.MarketplaceExceptionHandler#handleServerException(Exception, WebRequest)
org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
    at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:351)
...

@tparkola Could it be, that after 60 seconds something happens in the current setup of the backend? If I call the same endpoint with perpage=10 - which takes less then 60 seconds - it works, but when going over 60 seconds, I run into this error.

Next to this, the response time is far from being acceptable for this few items. I guess we will need some optimiziaton for the workflows/steps API endpoint. I was told, that when saving workflows, this can also take a long time, which does give editors not a good feeling.

Thanks to @cesareconcordia for discovinger this issue

tparkola commented 6 months ago

I think the problem is due to long response time - it causes a timeout and hence error is returned. I suggest to have a look at the backend routines and seek for optimisation. @KlausIllmayer @cesareconcordia does the problem relate to https://marketplace.sshopencloud.eu/search?categories=workflow only? I'm asking to confirm where to investigate the code. For the time being please use smaller page size, so that there is no chance for going over 60 seconds.

cesareconcordia commented 6 months ago

hi, I've only experienced the problem with workfows, other item categories seems ok. I'm currently using 20 as page size and there are no problems.

KlausIllmayer commented 6 months ago

I also see the problem only for workflows. It could be connected to the handling of steps for workflows, as there are quite many steps involved and all of them are loaded when getting the workflows.

tparkola commented 6 months ago

Indeed, the problem is most probably related to the amount of data loaded when providing relevant information about workflows. There is even a todo in the source core related to this. This is a medium-size task in my opinion.