pub.dev is down (global outage)

timsneath commented 3 years ago

Overview

At approximately 7pm, the pub infrastructure began to respond with a HTTP 502 Site Error message. The site infra is deployed to the us-central GCP region. This impacts core Flutter services, specifically:

The pub.dev website itself, which presents the error above;
Calls made by the flutter or pub commands (e.g. flutter upgrade or pub upgrade).

Please do not reply to this bug with "me too" or +1 messages, it makes it harder for folk to track. Thanks!

Updates

7:10pm Pacific We're currently experiencing an outage on pub.dev, which appears to be related to a load balancer issue. We don't have an ETA for a resolution at this time; we're currently working to understand the issue.

7:59pm Pacific The pub.dev site is still down. We have a Google on-call engineer currently investigating. We have not yet identified a root cause.

8:15pm Pacific We apologize for the inconvenience. We're seeing load balancer errors and are escalating to the appropriate team. Still no ETA, unfortunately, since we're still haven't determined the root cause.

8:27pm Pacific We have multiple Google Cloud engineers on-call investigating, but I'm sorry to report that we still don't have a root cause. We'll continue to post updates regularly. Thank you for your patience.

9:00pm Pacific We are continuing to debug the problem. We have declared a Google escalated outage while we attempt to identify the root cause. Some folk have been successful using the Chinese mirror site at https://pub.flutter-io.cn.

9:20pm Pacific Again, apologies.

9:35pm Pacific We are currently exploring the theory that we have exceeded a quota, but that the error didn't show in the log. Paging an oncall team to try and increase the quota to see if this resolves. Again, this really sucks -- we recognize that it's a major inconvenience to you all, and we're feeling sick that we're down. Thank you for being patient with us :(

9:45pm Pacific We have updated the quota and are resetting the VM instances, to see if we have successfully identified the root cause.

9:51pm Pacific We are seeing evidence of partially restored service.

9:55pm Pacific The pub service appears to be fully restored.

10:15pm Pacific Here's what we think we know at this point in time. At some point within the last day or two, a change was made to the pub.dev landing page that includes a call to the YouTube API. There is a quota limit for YouTube calls that we didn't hit over the last few days, but today we hit it. Confounding the issue, the code was missing exception handling and the logging was inadequate or obfuscated sufficiently that we were unable to immediately spot the problem. The immediate resolution was to raise the quota temporarily to give us time to revert the original change.

At this time we think the issue is resolved, but we'll obviously be monitoring closely. Again, apologies on behalf of the Flutter & Dart teams for the disruption. We take this very seriously, and we will perform a full post-mortem and share the learnings and actions we'll take as a result of this.

timsneath commented 3 years ago

Internal bug: https://buganizer.corp.google.com/issues/183665226

timsneath commented 3 years ago

No ETA to fix yet. We're figuring out what's gone wrong; we'll update this bug with status as we learn more.

Levi-Lesches commented 3 years ago

The package I uploaded last night must've been so useful and efficient, the server couldn't handle it 😎

yuebenshan commented 3 years ago

whats happening

GiteshDalal commented 3 years ago

Is there any back up website that we can point to while pub.dev is down? This is a big blocker for business.

duongtruong12 commented 3 years ago

my pub get and pub upgrade is failed now, cannot do anything else...

boyan01 commented 3 years ago

Is there any back up website that we can point to while pub.dev is down? This is a big blocker for business.

~~If you can access pub.flutter-io.cn, then you can try to use this URL as a temporary solution.~~

~~export PUB_HOSTED_URL=https://pub.flutter-io.cn~~

~~ref: https://flutter.dev/community/china~~

Did not work.

timsneath commented 3 years ago

Updates will be posted at the top of this bug. To make it easier to follow along, please don't post "me too" or +1 comments.

xuning0 commented 3 years ago

@boyan01 Can only access the homepage

eseidelGoogle commented 3 years ago

Status: All of pub.dev remain down. Internal load-balancing errors. We're diagnosing with SREs on call. No ETA yet.

GiteshDalal commented 3 years ago

Is there any back up website that we can point to while pub.dev is down? This is a big blocker for business.

If you can access pub.flutter-io.cn, then you can try to use this URL as a temporary solution.
export PUB_HOSTED_URL=https://pub.flutter-io.cn
ref: https://flutter.dev/community/china

Didn't work, seems like the backend is common for both websites. Unable to query packages using it, even though the website homepage is available.

LeoAiolia commented 3 years ago

Kpi is gone this month

BytesZero commented 3 years ago

I provide the following solutions, If you can't pack it, please try as follows. If you have successfully run the project before, please use the second plan. The newly acquired package is not applicable. You need to wait for the website to be completely restored.

pub.dev

export PUB_HOSTED_URL=~/.pub-cache/hosted/pub.dev

pub.flutter-io.cn （中国地区请使用下面的命令）
```
export PUB_HOSTED_URL=~/.pub-cache/hosted/pub.flutter-io.cn
```
See effect

Running "flutter pub get" in example...                             0.7s

Please see below for new solutions https://github.com/dart-lang/pub-dev/issues/4663#issuecomment-806354636

paurakhsharma commented 3 years ago

Is this the reason for me being stuck at this? This happened after I downgraded from 2.0.1 to 1.22.6. I even reinstalled flutter but still stuck at it.

Rquinz commented 3 years ago

Is this the reason for me being stuck at this? This happened after I downgraded from 2.0.1 to 1.22.6. I even reinstalled flutter but still stuck at it.

Yeah, it is

almpazel commented 3 years ago

It's happen , @flutter team can fixed that yeah.... Good luck...... We are wating..............

yubaokang commented 3 years ago

挂球了

Anoirwork commented 3 years ago

PUBdemic 😂😂

YowFung commented 3 years ago

Flutter 团队开启了 996ICU 拯救模式……

xuanhe168 commented 3 years ago

Pub is down,挂球了😂😂😂😂😂

milewski commented 3 years ago

Does aliyun has any pub.dev mirrors like it does for composer/npm/others package manegers? or something we can use meanwhile?

Rquinz commented 3 years ago

Does aliyun has any pub.dev mirrors? or something we can use meanwhile?

Saw this above. You can try this.

If you can access pub.flutter-io.cn, then you can try to use this URL as a temporary solution.

export PUB_HOSTED_URL=https://pub.flutter-io.cn

ref: https://flutter.dev/community/china

doesnt work for me though. goodluck

xuanhe168 commented 3 years ago

the "pub.flutter-io.cn" is accessible

azhon commented 3 years ago

Can't write code, forced to paddle

stargt commented 3 years ago

help

rmasonh18 commented 3 years ago

Thanks for being on top the issue ~ good to know problem isn't issue on my mac.

hienlh commented 3 years ago

Many businesses are being blocked. Please resolve it soon! Waiting for a good news

BytesZero commented 3 years ago

许多企业被封锁。请尽快解决！等待一个好消息

Please see my solution https://github.com/dart-lang/pub-dev/issues/4663#issuecomment-806338861

BytesZero commented 3 years ago

Thanks for being on top the issue ~ good to know problem isn't issue on my mac.

Please see my solution https://github.com/dart-lang/pub-dev/issues/4663#issuecomment-806338861

ahyangnb commented 3 years ago

help

Levi-Lesches commented 3 years ago

Guys please stop spamming this issue. We all get notified every time. Just press the subscribe button on the side and wait for an update please, we're all being patient.

Levi-Lesches commented 3 years ago

Again, if you have this error -- just wait. Please don't post unless you're on the team fixing this. (I'll stop so I don't spam either).

milewski commented 3 years ago

Run this command, assuming the packages you are using are common enough and have been previously cached via this flutter mirror before the outage, if it doesn't work for you that's because the package didn't exist on mirror cache and it will attempt to fetch pub.dev which obviously will not work.. so good luck

PUB_HOSTED_URL=https://mirrors.tuna.tsinghua.edu.cn/dart-pub flutter pub get

or

PUB_HOSTED_URL=https://mirrors.cloud.tencent.com/dart-pub flutter pub get

or

PUB_HOSTED_URL=https://pub.flutter-io.cn flutter pub get

jo3ght commented 3 years ago

I'm becoming short on a deadline

davidlawson commented 3 years ago

pub.flutter-io.cn isn't working for me, but the following mirrors are:

https://mirrors.tuna.tsinghua.edu.cn/dart-pub
https://mirrors.cloud.tencent.com/dart-pub

example terminal usage (non-permanent):

PUB_HOSTED_URL=https://mirrors.tuna.tsinghua.edu.cn/dart-pub flutter pub get
PUB_HOSTED_URL=https://mirrors.cloud.tencent.com/dart-pub flutter pub get

blupointmedia commented 3 years ago

Run this command, assuming the packages you are using are common enough and have been previously cached via this flutter mirror before the outage, if it doesn't work for you that's because the package didn't exist on mirror cache and it will attempt to fetch pub.dev which obviously will not work.. so good luck
PUB_HOSTED_URL=https://pub.flutter-io.cn flutter pub get

How would I clear this after it is back up again?

awaisahmad94 commented 3 years ago

@davidlawson How to set to this mirror to fetch pls guide for mac. thanks

milewski commented 3 years ago

How would I clear this after it is back up again?

You don't need to do anything. this is not setting anything up permanently

simonpham commented 3 years ago

Run this command, assuming the packages you are using are common enough and have been previously cached via this flutter mirror before the outage, if it doesn't work for you that's because the package didn't exist on mirror cache and it will attempt to fetch pub.dev which obviously will not work.. so good luck
PUB_HOSTED_URL=https://pub.flutter-io.cn flutter pub get
How would I clear this after it is back up again?

Just close your Terminal

awaisahmad94 commented 3 years ago

pub.flutter-io.cn isn't working for me, but the following mirrors are:
https://mirrors.tuna.tsinghua.edu.cn/dart-pub
https://mirrors.cloud.tencent.com/dart-pub

☝️

1l0 commented 3 years ago

I've confirmed that one of the mirrors works for flutter upgrade. Thanks for the info.

PUB_HOSTED_URL=https://mirrors.cloud.tencent.com/dart-pub flutter upgrade

ensaryusuf commented 3 years ago

TencentsCDN network can be partially distributed. Packages keep not working.

pub.flutter-io.cn isn't working for me, but the following mirrors are:
https://mirrors.tuna.tsinghua.edu.cn/dart-pub
https://mirrors.cloud.tencent.com/dart-pub
example terminal usage (non-permanent):
PUB_HOSTED_URL=https://mirrors.tuna.tsinghua.edu.cn/dart-pub flutter pub get
PUB_HOSTED_URL=https://mirrors.cloud.tencent.com/dart-pub flutter pub get

Tencents CDN network can be partially distributed. Packages keep not working. (in Turkey)

xxived commented 3 years ago

server is up now, at least for me

Salahadin48 commented 3 years ago

Running this command in the terminal of the current project resolved my issue. PUB_HOSTED_URL=https://mirrors.cloud.tencent.com/dart-pub flutter pub get