search

openstreetmap / operations

OSMF Operations Working Group issue tracking
https://operations.osmfoundation.org/
98 stars 13 forks source link

503 on help.openstreetmap.org #298

Closed TZornG closed 5 years ago

TZornG commented 5 years ago

For a couple of days I have been experiencing frequent 503s. Reloading the page once usually successfully returns the proper page.

Service Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
Apache/2.4.29 (Ubuntu) Server at help.openstreetmap.org Port 443
tomhughes commented 5 years ago

That usually means that somebody is scraping trac and tieing up all the resources - there's not a huge amount we can do about it sadly.

tomhughes commented 5 years ago

It also doesn't help that OSQA is completely dead upstream and we certainly don't have the resources to maintain it ourselves - really help.osm.org is on life support these days. If it works then fine but that's about the best we can offer.

tomhughes commented 5 years ago

That said I don't see any 503s logged for help.osm.org in the last few days - are you sure the error isn't coming from some of proxy between you and us? They often issue 503 errors.

tomhughes commented 5 years ago

I see no evidence in our logs that the errors are coming from us - the only 503s on that machine appear to be against the default vhost from somebody probing for PHP vulnerabilities.

TZornG commented 5 years ago

I'm not sure it does not come from somewhere else, but this is the only site I experience such error with. How would I find out?

tomhughes commented 5 years ago

It's true that as it's SSL it shouldn't be but if it doesn't appear in our logs then I don't see how it can come from us either, and I certainly don't know what I can do about something I can't even see happening.

The only thing I can think is that it's really your browser faking it when a connection times out?

About the only thing you can try and do is to look at the request in your browsers debug console to see if that gives any clues.

TZornG commented 5 years ago

Really strange. I tried with Firefox as well as Opera and I experience the same issue. One out of 20 or 30 page loads results in the 503. But only on this site, not a single problem with any other page.

For what it's worth here is what the network inspector of Firefox gives me (very similar in Opera):

Request URL:https://help.openstreetmap.org/questions/
Request method:GET
Remote address:[2001:41c9:1:400::32]:443
Version:HTTP/2.0

request headers:

Host: help.openstreetmap.org
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: de-DE,de;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate, br
Referer: https://help.openstreetmap.org/questions/69029/map-data-on-mobile-version
Connection: keep-alive
Cookie: <deleted>
Upgrade-Insecure-Requests: 1
TE: Trailers

response headers:

HTTP/2.0 503 Service Unavailable
date: Mon, 06 May 2019 12:19:20 GMT
server: Apache/2.4.29 (Ubuntu)
strict-transport-security: max-age=31536000; includeSubDomains; preload
expect-ct: max-age=0, report-uri="https://openstreetmap.report-uri.com/r/d/ct/reportOnly"
content-length: 388
content-type: text/html; charset=iso-8859-1
X-Firefox-Spdy: h2

In at least one instance I got 'undefined undefined undefined' as first line instead.

response

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>503 Service Unavailable</title>
</head><body>
<h1>Service Unavailable</h1>
<p>The server is temporarily unable to service your
request due to maintenance downtime or capacity
problems. Please try again later.</p>
<hr>
<address>Apache/2.4.29 (Ubuntu) Server at help.openstreetmap.org Port 443</address>
</body></html>

The inspector confirms a correctly set up secure connection.

Two other time stamps are: date: Mon, 06 May 2019 11:42:08 GMT date: Mon, 06 May 2019 19:25:36 GMT

Well, if you can't find anything (and I don't on my side) I live with reloading the page every now and then. Thanks for having a look into this anyway.

Tigerfell commented 5 years ago

Just for the record: I managed to get some 503s, but at a far lower rate of about 1 in 10, but no new insides unfortunately.

TZornG commented 5 years ago

Was anyone able to look into this more? I still regularly get the 503s (just a couple of seconds ago again).

tomhughes commented 5 years ago

What am I supposed to look at? I've already said there is no evidence of them on the server, and I have never seen it myself.

TZornG commented 5 years ago

I'm not an expert on this. But since others seem to experience the same and @gravitystorm reopened the issue I found it worthwhile asking.

gravitystorm commented 5 years ago

@gravitystorm reopened the issue I found it worthwhile asking.

I reopened it because the response headers had accurate timestamps and seem pretty unambiguous to me. If there's no corresponding errors on the server, then it makes me wonder if the server is giving 503 responses but not logging them to disk.

mmd-osm commented 5 years ago

IIRC, I started seeing those errors first time after the Ubuntu upgrade back in January (which might be just a coincidence), still it’s fairly infrequent.

TZornG commented 5 years ago

Just want to let you know that I haven't seen any more 503s in the past four weeks or so.

Tigerfell commented 5 years ago

@TZornG I do not get 503s when reading or refreshing pages anymore either.

mmd-osm commented 5 years ago

@TZornG : if the issue doesn't appear anymore, I think sysadmin would be happy if you could close this issue then.

tomhughes commented 5 years ago

I thought it was closed...

Firefishy commented 5 years ago

Minor follow up, I've made some changes to the process model, which seems to be helping: https://github.com/openstreetmap/chef/commit/ea7cec92bc26d3a920da06070036ea62953ff03c

TZornG commented 4 years ago

Since yesterday it has become extreme again. I estimate 4 out of 5 requests get a 503 response. And judging by the low volume of contributions to the site in the past two days I'd assume others have problems, too.

TZornG commented 3 years ago

For now three or four days, we have seen regular 503s again on help.openstreetmap.org. Other users have the issue, too.

Can you have another look?