Open haesleinhuepf opened 11 months ago
Interestingly https://biii.eu/search?search_api_fulltext=%22Fiji+tracking%22 is working, as well as https://biii.eu/search?search_api_fulltext=%22Fiji++Tracking%22 and https://biii.eu/search?search_api_fulltext=Fiji%2Btracking (when Fiji+tracking was entered as search in the text https://biii.eu/search?search_api_fulltext=microtubule+tracking, to be investigated further
The following sounds as a very similar issue , now reported by Hervé Ménager Hi BIII friends,
It looks like there is a problem on BIII. When trying to retrieve the list of software from your server (URL http://biii.eu/soft?_format=json), the reply I get is a 503. Detailed message is:
Error 503 Backend fetch failed Backend fetch failed
Guru Meditation: XID: 975783143
Varnish cache server
I am checking the host server database health and then sending an issue to our hosting server support
I'm keeping you updated here, below is the ticket. From You Hosting - Error on website
Hosting concerned web server
Domain concerned biii.eu
Is this error present on some pages or on the whole site? search pages e.g. http://biii.eu/soft?_format=json but working well on http://biii.eu/fiji?_format=json or https://biii.eu/search?search_api_fulltext=%22fiji+tracking%22 or
Did this error appear after a particular change? no
Error message Error 503 Backend fetch failed Backend fetch failed
Guru Meditation: XID: 975783143
Varnish cache server
Additional information Nothing changed we think from the client side Date received 10/11/2023 15:58
Thanks @PerrineGilloteaux , just re-tested the page http://biii.eu/soft?_format=json, still getting a 503:
Thanks @PerrineGilloteaux , just re-tested the page http://biii.eu/soft?_format=json, still getting a 503:
Yes, I ve just reported so far, so I am not expecting an improvement right now :-) , but I confirm I am getting also the error constantly on the pages I've mentionned (including the one you've reported), while some other json export (including the onbe used by comulis) are working well. Google told me it could be from the server side, so I ve contacted our host server support.
More info: accessing from GitHub Actions fails with the same error. Alban's test: https://github.com/research-software-ecosystem/content/actions/runs/6955587741/job/18924654703
Same error happened with https://biii.eu/search?search_api_fulltext=segmentation or https://biii.eu/search?search_api_fulltext=image
But this works: https://biii.eu/search?search_api_fulltext=fiji
Could you please look into this?
Hi @oeway
thanks for reporting the problem. My guess now is that the server running biii has low performance and the response to searches sometimes becomes too slow causing the 503 error. Do you maybe have any suggestions for testing if such a problem is happening? This does not appear in the Drupal log - I looked for any error in responding but this is not recorded. You have been running ImageJ.JS and I believe that you have good knowledge in this direction and would like to reply to you for any tips.
If this is an issue that can be solved simply by upgrading the server, we are lucky... but for this, it's better to have some clear indication that this is due to the low-performance server.
Kota
Hi Kota,
Thanks for looking into this! I can reliably reproduce the error with the above URL, so i think it's not a performance issue. Giving the 503 error, my guess is that the server (most likely the varnish cache server) entered a bad state.
I am not an expert in Varnish, but maybe you should check the log via "varnishlog".
Update: It looks like the server is getting back, I assume you did some refreshing on the caching sever since I see a very slow response on new queries but becomes rapid for a second search and onwards. Thank you!
FYI: Still failing for https://biii.eu/search?search_api_fulltext=image+analysis, same 503 error.
Hi, it works for me??
Could it be that you are searching very generic terms here so a lot of output and then time out by your own internet provider ?
Try with something like https://biii.eu/search?search_api_fulltext=%22c-elegans+analysis%22 Meaning c-elegans AND analysis (if you don't put the " " it will search for c-elegans OR analysis https://biii.eu/search?search_api_fulltext=c-elegans+analysis less relevant
It fails from time to time l, it I do see a pattern with more general terms fails more.
A bit of context, we are building chatbot which interface with biii, the user might trigger very general search.
I still can reproduce the issue now
Hello,
we are using a CDN provided by the host , I do not think we use varnish? At least it not installed as a module ? @kota did you install it (not configure ongithub at least) ? We should not have it we already have a CDN? Or if you think it is better I should desactivate the CDN. https://help.ovhcloud.com/csm/en-gb-web-paas-bestpractices-http-caching?id=kb_article_view&sysparm_article=KB0053654
For now I've just purged the CDN (which is actually something to be done from time to time and I don't think I never did), let me know if better.
Here are some time pattern in errors (red curves), but they are mostly 400s http errors (I guess bots trying non existing pages ) , very few 500s. And it does not seems correlated with the number of access (blue curve)
Please report if the 503 is still hapening from your side? (after emptying your own browser cache maybe?)
Thanks for looking into this! I can still reproduce it with the link I pasted above.
Hello (adding @albangaignard @hmenager ), I am waiting for @miura Kota's confirmation that we did not install Varnish (That I can not see from the server side or hosting service but maybe very well hidden: we use another method = CDN so indeed we should not using Vanirsh ). Then I will contact again the hosting service to check with them claimin that we did not install OVH.
Interestingly there is some suggestions here that it can come from the browser or router of the people accessing the page (that we should solve in any case) https://www.partitionwizard.com/partitionmagic/error-503-backend-fetch-failed.html .
I've just tested (with @albangaignard as a witness)
Thanks @PerrineGilloteaux . If this is client side, then the situation is pretty bad for us, because this means it would stand somewhere in in the github actions stack. still doesn't work there for us anyway.
Interestingly I get this when using chrome or safari from desktop, but it works with iOS Safari. :shrug:
Hi all,
when searching for "fiji tracking" on biii.eu leads to this error at the moment:
https://biii.eu/search?search_api_fulltext=fiji+tracking