search

proycon / clam

Quickly turn command-line applications into RESTful webservices with a web-application front-end. You provide a specification of your command line application, its input, output and parameters, and CLAM wraps around your application to form a fully fledged RESTful webservice.
https://proycon.github.io/clam
GNU General Public License v3.0
129 stars 17 forks source link

Handle very large projects (thousands of files) #79

Closed proycon closed 4 years ago

proycon commented 5 years ago

Timeouts occur when clam attempts to handle large projects (thousands of files).

proycon commented 4 years ago

For result pages that take too long too load due to having many output files (or may not load at all and throw a HTTP 500 due to server-side timeouts), there is now an advanced trick that you can append ?quick=1 to the URL, which will skip some of the more expensive processing and present the result page quicker. When enabled, the result page will lack certain information (such as associated viewers, formats, templates) but it will at least load. (@VincentCCL This should address your problem with visualisation of some existing projects)

Also, I'm implementing measures so that future projects no longer generate excessive provenance data that unnecessarily slows down the loading and presentation of results, which will hopefully have a high enough impact on the loading times so that the ?quick=1 workaround is not necessary.

VincentCCL commented 4 years ago

Hi Maarten,

When I try this:

https://webservices-lst.science.ru.nl/frog/karrewiet/?quick=1

I still get a 500 error

v.

On 2020-04-01 17:40, Maarten van Gompel wrote:

For result pages that take too long too load due to having many output files (or may not load at all and throw a HTTP 500 due to server-side timeouts), there is now an advanced trick that you can append ?quick=1 to the URL, which will skip some of the more expensive processing and present the result page quicker. When enabled, the result page will lack certain information (such as associated viewers, formats, templates) but it will at least load. (@VincentCCL [1] This should address your problem with visualisation of some existing projects)

Also, I'm implementing measures so that future projects do longer generate excessive provenance data that unnecessarily slows down the loading and presentation of results, which will hopefully have a high enough impact on the loading times so that the ?quick=1 workaround is not necessary.

-- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub [2], or unsubscribe [3]. [ { "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "https://github.com/proycon/clam/issues/79#issuecomment-607325029", "url": "https://github.com/proycon/clam/issues/79#issuecomment-607325029", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

Links:

[1] https://github.com/VincentCCL [2] https://github.com/proycon/clam/issues/79#issuecomment-607325029 [3] https://github.com/notifications/unsubscribe-auth/AIEEDJJBKUGLPRKI6TZ33MDRKNN55ANCNFSM4HY2HQUQ

proycon commented 4 years ago

@VincentCCL Oh yes, sorry for the confusion, it's implemented but not released and deployed yet. I'm still working on it (along with some other issues) and will do a release and deployment then. (it'll be in CLAM v3.0.12)

VincentCCL commented 4 years ago

ok bedankt,

v.

On 2020-04-03 13:23, Maarten van Gompel wrote:

@VincentCCL [1] Oh yes, sorry for the confusion, it's implemented but not released and deployed yet. I'm still working on it (along with some other issues) and will do a release and deployment then. (it'll be in CLAM v3.0.12)

-- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub [2], or unsubscribe [3]. [ { "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "https://github.com/proycon/clam/issues/79#issuecomment-608380227", "url": "https://github.com/proycon/clam/issues/79#issuecomment-608380227", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

Links:

[1] https://github.com/VincentCCL [2] https://github.com/proycon/clam/issues/79#issuecomment-608380227 [3] https://github.com/notifications/unsubscribe-auth/AIEEDJNPTYVJ2RHMSWTDJ6DRKXBK7ANCNFSM4HY2HQUQ

proycon commented 4 years ago

The implementation is finished and released, the "quick mode" will be enabled automatically if loading the result index exceeds a certain time out (90s by default).

Deployment in Nijmegen will follow later tonight.