blah.supabase.co/rest/v1 request failing

[jaredramirez]

Bug report

Describe the bug

Both on the Supabase dashboard (app.supabase.com) and our apps, requests to instance-id.supabase.co/rest/v1 are failing.

This is happening on both Supabase organizations I am a part of.

To Reproduce

Attempt to send any request to `instance-id.supabase.co/rest/v1.

Expected behavior

I expect requests to the supabase postgrest instance to be successful.

Screenshots

If applicable, add screenshots to help explain your problem.

System information

• OS: MacOS
• Browser (if applies) Firefox

Additional context

[w3b6x9]

@jaredramirez can you please open up a support ticket and send us your projects' refs (instance-ids)? We'll investigate!

[ajoslin103]

I am having the same problem the dashboard loads but trying to list the database tables fails with:

GET https://scbqrtiaoyjyyqxuxzjl.supabase.co/rest/v1/ net::ERR_SSL_PROTOCOL_ERROR

And my App is down -- it's getting the error

This site can’t provide a secure connection scbqrtiaoyjyyqxuxzjl.supabase.co sent an invalid response. ERR_SSL_PROTOCOL_ERROR

I started a new project and it's having the same problem !

I'd buy Pro +/Usage on my Live project if it's going to get this solved !!

And I can't find where to open support tickets...

[ajoslin103]

SOLUTION

Very odd for me as only 1 out of 3 projects was working

I turned off the "advanced security" in my Cox Panoramic WiFi device and everything works now

[inian]

Hi, yeah we had few reports of this recently. Google had flagged one of the subdomains recently, but we worked to Google to sort this out and Google has removed the flag. I guess local routers also make use of the Google Safe Browsing dataset but have not updated yet. So, the solution is to add an exception to your local firewall or to force an update of the firewall rules in your router.

[ajoslin103]

after a couple of hours I tried turning Panoramic Wifi Advanced Security back on

that did not update the rules, so I turned it off again

I had previously tried turning off SafeBrowsing with no effect

So my Completely Non Technical users may still be borked

They won’t be able to turn off: Panoramic Wifi Advanced Security

Nor can I expect my hoped for millions of customers to do so either

This one is somewhere else, I guess you better start here...

Cos Support: 1 (800) 234-3993

On Jun 27, 2021, at 11:47 AM, Inian @.***> wrote:

Hi, yeah we had few reports of this recently. Google had flagged one of the subdomains recently, but we worked to Google to sort this out and Google has removed the flag. I guess local routers also make use of the Google Safe Browsing dataset but have not updated yet. So, the solution is to add an exception to your local firewall or to force an update of the firewall rules in your router.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/supabase/supabase/issues/2096#issuecomment-869184128, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADML5N6VTEMNJHU6JZQ4XTTU5BYNANCNFSM47KYAPGQ.

[HuynhBrian]

I'm experiencing a similar issue

[romlytech]

Same problem here. Was able to allow access via my Xfinity router.... one hour at a time...

[kiwicopple]

Hi all, the investigation is still ongoing but we have some updates. This was caused by user who signed up to host malicious content on Supabase storage. We took down the content within an hour and blocked the account - it looks like they were a very new GitHub account created specifically for this

We worked closely with our contact at Google to clear the subdomain: https://transparencyreport.google.com/safe-browsing/search?url=supabase.co

We are taking steps to mitigate this:

• We are adding supabase.co to the Public Suffix List which would indicate any site under supabase.co are to be treated as separate websites
• We are limiting Storage Hosting to a small subset of content types (we will even block HTML hosting until we are certain we can manage malicious actors at scale)
• We will implement support for custom domains so that you are not impacted by other users on the platform
• We've setup an abuse@supabase.io email to receive notifications when we receive abuse reports from upstream providers
• We will block suspicious Github accounts from creating new projects until they have been reviewed by a team member

[jaredramirez]

Awesome, things are working for me now. I'll close this for now, and if others experience this then I think follow w3b6x9's original message about opening a support ticket.

[themattmayfield]

Still having the same issue. Has anyone found out a solid solution. ALL my apps are down with the same errors everyone else is having. A tad bit frustrating but I'm here lol

[inian]

Unfortunately we have little control over local routers still using stale data from Google Safe Browsing dataset. If this is just affecting you, whitelisting your supabase domain in the router firewall will help. If this is affecting your users as well, I understand that's not a feasible solution - reach out to beta@supabase.io and we will see if we can work something out.

We are working hard on making sure this doesn't happen again though. Some udpates

• We rolled out a fix which renders html pages as text/plain instead of text/html, so that it will not be possible to host (deceptive) webpages on Supabase storage.
• We have submitted a PR to include supabase.co domains in the public suffix list
• We have an abuse email now and better processes when an upstream provider sends us an abuse report.

Next we are going to work on blocking suspicious new Supabase accounts and implementing custom domain support.

[ghost]

I am not entirely sure how to "whitelist your supabase domain in the router firewall will help". Does anyone have any helpful or direct links as to where / how to do this?

Thank you for any help!

[Danieltech99]

@conor-gaughan You want to look up / Google your router or internet provider router + "whitelist". Some internet providers, like Xfinity, don't allow you to whitelist or don't have functionality to add a domain to a whitelist, so for these providers you will want to turn off "Advanced Security" (for example with Xfinity this is called "xFi Advanced Security")/

[ghost]

That fixed it - thank you very much @Danieltech99 !

[MattWIP]

Hey guys - I have a user that is still experiencing this from his Cox router. I screen-shared with him last night and noted that his security settings were turned all the way down on his router. Any other clues as to how I might resolve this?

[inian]

Hi @MattWIP, another idea is to use a VPN. Not sure if the router is blocking at the DNS level, if it is changing your DNS resolver to a different resolver like 1.1.1.1 or 8.8.8.8 might work

[MattWIP]

Hi @MattWIP, another idea is to use a VPN. Not sure if the router is blocking at the DNS level, if it is changing your DNS resolver to a different resolver like 1.1.1.1 or 8.8.8.8 might work

Hey @inian Thanks for the response! Yeah I understand this is a possibility, but I can't reasonably as my user(s) to access our site via VPN or changing DNS settings in order to access their content. I need a more blanket & reasonable resolution for our non-tech-savvy users. Do you think initializing a fresh project on Supabase might help?

[inian]

From all the reports, it looks like this is happening only for Xfinity subscribers. Their support hasn't been too helpful in this. They recommend affected users contacting Xfinity support directly. So, if you can file an issue with them, it will help! I am also planning to submit this form to see if that helps.

[MattWIP]

Hey, @inian just to add to that I'm having 2 Cox ISP users experiencing the issues w/ their Cox Panoramic Wifi Modems (which seems to be the only common thread I can identify between my two users experiencing issues). I was able to get them access by having them shut off their modem's security all together, obviously not ideal but works for now.

[joeriddles]

I was previously experiencing this issue while using CenturyLink. It's resolved for me now, but it may not be limited to just Xfinity.

[inian]

If you are a customer of the affected ISPs, please open support tickets with them. If they have been using Google's Safe Browsing for their security rules, the domain has been whitelisted by them a few weeks ago already.

I will try reaching out to CenturyLink and Cox too, but since I am not a customer (or based in US), it is easier for existing customers to raise support tickets with them. If you are able to flag this out to them, reach out to security@supabase.io with what their support rep says and we will send some swag over for helping us fix this issue ❤️

[monsieurBoutte]

@inian I came here for the same issue and luckily saw the comment @Danieltech99 about shutting off my advanced security on my xfinity router.

But it doesn't really feel like a long term solution and I'd be happy to open a support ticket with my ISP. If you wouldn't mind sharing a template for reaching out to our ISPs — I think that would be helpful for anyone else in the future who lands on this thread.

[ajoslin103]

I'm sorry to say this, but anything we do as developers that helps us keep moving is good -- but I can in no way ask even one of my testers, let alone my eventual users do anything of the sort.

I just have to trust that the real reason it's causing trouble is because I am doing developery things and that production code from the backend of my production copy will never ever ever hit this problem.

[AWaselnuk]

But it doesn't really feel like a long term solution and I'd be happy to open a support ticket with my ISP. If you wouldn't mind sharing a template for reaching out to our ISPs — I think that would be helpful for anyone else in the future who lands on this thread.

@monsieurBoutte suggestion for a template would help me out. How should I direct people to reach out to ISPs about this? Asking people to figure out their router settings and shut off security does not seem sustainable.

This issue continues to be a support burden for me, blocking real customers from using my site. I shudder to think of how many potential customers I've lost to it (you need to sign up/in to buy my product).

Happy to reach out to you guys by email if that's easier than on this thread - just let me know @inian

[quicksnap]

Just to add to the pile, I was testing my app on a friend's network and it failed, likely because of this. I believe they're using a Comcast router.

(PS Hey Victor!)

[inian]

Another update - Supabase domains have been accepted into the Public Suffix List. Over the next few days, we will be offering an alternate domain for affected users. Everyone affected please reach out to beta@supabase.io referencing this issue and I will set you up the alternate domain right away.

[xerod]

Guys, I'm having this issue on vercel and I'm trying to replicate it on my local server. it turns out having the same result (NS_ERROR_DOM_BAD_URL) like #2160.

I have encountered similar issue earlier when using different instance with free tier, probably caused by a config error by supabase when the instance is revived after being paused for a week. But now I'm using this Pro tier and still having this problem. Any solution to this? I'm experiencing this issue on production instance and I'm not sure whitelisting supabase subdomain on local router would help.

[inian]

Can you create a support ticket https://app.supabase.io/support/new @xerod?