surge.sh seems to be down?

[prevail90]

is anyone else seeing this as well?

[Tennisatw]

Me too. I always get a 503 error even on the help page.

[joanniclaborde]

Hi @sintaxi. Should we consider surge.sh to be abandoned? It's an amazing service but it's been very unreliable lately, here's the last 30 days:

Thanks!

[sintaxi]

@joanniclaborde Every year or two surge hits a new scaling issue that has to be worked through. The downtime above is for our Toronto datacenter only. For some reason (that is not yet understood) in that datacenter we are having a cache server that is crashing when it attempts to snapshot its data. It can be valuable to observe the problem and come up with solutions that make the system more fault tolerant. Perhaps I haven't address this issue with enough urgency and my poor communication doesn't help either. For that Im sorry.

[joanniclaborde]

@sintaxi Thanks for the quick update, it's appreciated! Do you have an approximate date when this issue should be fixed?

[sintaxi]

spinning up new edgenodes now. should be fixed within the day.

[joanniclaborde]

That's amazing, thanks! I'll keep you posted if I notice any more downtime.

[emenel]

Hi all. Still seeing a 503 when I visit surge.sh unfortunately. Any updates on when this will be more completely resolved?

[sintaxi]

Fixes still in the works.

[aogaili]

All my apps are down :( 503 Service Unavailable

[aogaili]

Seems to be back up! I hope there is some protection against this kind of disturbance, it rarely happens but it hits hard when it does.

[sintaxi]

UPDATE: New edgenodes have been stood up in the Toronto location. Doing some QA before re-routing traffic.

[aogaili]

Thank you so much @sintaxi , honestly, surge has superior UX, the simplicity and functionality are just too good, and it has been my go-to.

[sintaxi]

New Toronto Edgenode has been running flawlessly for 72h. Also spun up a new edgenode in London which has been running for 48h.

Going to close this ticket. Thank you all for the help and support.

[aogaili]

Wonderful, much appreciated 🙏

[kevinta893]

Hi there, I am still getting 502 Bad gateway errors for my sites as well. Any updates?

[aogaili]

Could be specific to your deployment or region, don't see the issue here.

On Sun, Oct 15, 2023, 11:14 a.m. kevinta893 @.***> wrote:

Hi there, I am still getting 502 Bad gateway errors for my sites as well. Any updates?

— Reply to this email directly, view it on GitHub https://github.com/sintaxi/surge/issues/513#issuecomment-1763420441, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMZAK5PJJUY2PX4BCX3JTLX7P4UJANCNFSM6AAAAAA5ETT2JQ . You are receiving this because you commented.Message ID: @.***>

[joanniclaborde]

Still happening here too, here's the last 24h:

And the last 7 days:

[kevinta893]

I have no issues during deployment, but visiting my websites is hit or miss. Sometimes my browser will grab a few files and 50X error. Sometimes I'm lucky and the whole site loads in time before another error.

I confirmed with various users of my sites across the globe that they too are having the same issues.

[sintaxi]

I can confirm there has been issues with a couple of our North America locations. Changes have been made and I think we are through the worst of it. Please keep me posted with your findings for the coming week.

thanks, Brock

[sintaxi]

Im reopening this ticket until stability is confirmed by you all.

[KeyboardSounds]

Im reopening this ticket until stability is confirmed by you all.

I was seeing 502 errors across all of my deployments for a few weeks (I'm based in Australia, if that makes a difference), but they're all fixed now and everything is working as expected. Thanks for your hard work!

[joanniclaborde]

The last incident was on 2023-10-20 for me.

[kevinta893]

Seems stable for me for now.

[ultracodez]

All subdomains having issues right now for me. Updates @sintaxi ?

[ronskie69]

Is the server down?

[ronskie69]

it's now working...

[ultracodez]

Now getting a 504 Gateway Time-out: "The server didn't respond in time."

[wkande]

Same here and it was down yesterday for me as well. Had sent an email to support but no response. I wish they would at least post outages somewhere.

[ultracodez]

Seems resolved now...

[wkande]

Yes but sadly Surge is not saying much.

UPDATE: not five minutes after posting this comment I got an update from Surge:

Apologies for not getting back to you sooner. We have been fighting off a wave of phishing campaigns on our platform and it has lead to our SFO datacenter being temporarily out of service. We are routing the traffic to our Toronto datacenter until we get the issue resolved.

The 40 second delay is caused from the deployment server trying to verify the deployment to the SFO location and timing out. We will look into improving that experience.

So I spoke too soon.

[techterbium]

@sintaxi - I am seeing this error since today. Is there a way to get around this? Thanks in advance.

Error: aborted
    at connResetException (node:internal/errors:721:14)
    at TLSSocket.socketCloseListener (node:_http_client:455:19)
    at TLSSocket.emit (node:events:526:35)
    at node:net:337:12
    at TCP.done (node:_tls_wrap:657:7) {
  code: 'ECONNRESET'
}

[briantist]

I'm also unable to publish. Site is up but can't push to it

Error - Deployment did not succeed.

That's all the information I get from the process.

[aogaili]

I'm getting the same issue.

[sintaxi]

Hey everyone. This was an issue with the deployment servers and has now been resolved. This did not affect uptime of projects. Please let me know if you still see issues on your end.

[briantist]

Looks good to me, thanks @sintaxi

[seidlere]

this morning i'm unexpectedly getting 503s on my subdomains that have been working fine for the last few days. no problems with the deploys, though. anyone else seeing this? @sintaxi

[wkande]

Same here but it seems to vary across the globe. I have some co-workers in other countries that work and some do not.

[alexgleason]

Looks like there's downtime this morning. Getting complaints about it.

[joanniclaborde]

Still unstable here...

[JhonTreft]

Hello, could someone tell me what is happening with arise? hehe it's down

[joanniclaborde]

Hi @sintaxi! Do you think we can expect Surge to be back online soon?

[hoylemd]

I'm having trouble accessing an api docs page for my company. I'm in Toronto, but a colleague near the east coast of canada can access it without trouble. Is the Toronto data center having a bad day maybe?

edit: details:

I'm getting 503 errors when I hit the URL:

curl -i crowdmark-api-docs.surge.sh

HTTP/1.0 503 Service Unavailable
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>

[schandler88]

Same issue here

[hoylemd]

@schandler88 I just added some more detail - is that also what you're seeing?

I also had a colleague based in Toronto say they could access the site too, confusingly

[hoylemd]

it's working again for me!

[mitchellonii]

Based in Australia, getting 504 for new deployments and no changes for updated deployments. Seems to be working fine with a US VPN on...

[christianleong]

Based in Australia, is anyone else getting a 502 Bad Gateway error?

[qnfzks55]

Based in Australia, is anyone else getting a 502 Bad Gateway error?

The same 502 Bad Gateway error is occurring in Republic Of Korea.

[sintaxi]

Not seeing any issues on our end and Im having a hard time figuring out what edgenode you are being directed to.

Can you run ping <domain> and let me know the results?

[qnfzks55]

Not seeing any issues on our end and Im having a hard time figuring out what edgenode you are being directed to.

Can you run ping <domain> and let me know the results?

Returns Request Timed Out result value.