Closed wbond closed 8 years ago
Thanks for the heads up on this - it's appreciated.
Definitely big thanks for the updates, since of course, you can imagine (Murphy's law?) -- I started for the first time really putting my time into getting ST to be my editor of choice, and as the first steps were easy, I didn't expect to get mysteriously stuck on the part which basically fetches metadata from PackageControl site. "Something seemed clicking", since it's not that often I've come across having used a service that is under attack. So, it's really useful to get feedback and understand that this isn't the way things usually run. It was annoyance, but much less so once this news came about.
Wow. Didn't realize Linode's gotten hammered. Good remediation plan for next time. Cheers.
Thanks for the update. Right now, i can open https://packagecontrol.io/channel_v3.json in browser. But it returns an almost empty json. {"packages_cache": {}, "dependencies_cache": {}, "schema_version": "3.0.0", "repositories": []}
@Grisson Thanks for the info, I believe I've temporarily resolved the issue for now, but I may need to do another fix later.
@wbond It works. Million thanks for your hard work. :)
+1 to all the votes of thanks. Any chance you can set up a yearly donation option as well? As a non-US supporter, I get clobbered with international currency fees every time I make a donation, which adds up if done monthly.
@ijdickinson Yes, it does look like I can set up a yearly payment option also. When I am on later I'll get that set up.
I'd like to add a suggestion for a way to decrease bandwidth usage. I'm guessing most of the 100GB/day is from PackageControl clients checking for updates to their installed plugins by downloading the channel json file. Instead of downloading the whole JSON file, why not generate a JSON file per package and deploy that to s3, GitHub, or some other free/cheap static file hosting?
That means PackageControl no longer needs to download the complete channel JSON file when checking for updates, it only needs to request the specific named JSON files for the user's installed packages.
Honestly, bandwidth reduction isn't a high priority. Making 40 requests instead of one would likely cause different issues, and would require lots of debugging and development work.
The most expensive thing is development and maintenance work. One hour of billable time for a senior engineer is more than the monthly cost of 8TB of bandwidth from Linode.
The thing we really want from the upgrade is a fault tolerant system, which means multiple providers and failover, preferably mostly automated.
thanks for the detailed update. Had no idea about the DDoS attacks.
Many thanks @wbond for your efforts. :) . now I can use this URL https://packagecontrol.io/channel_v3.json.
I'm new to Sublime Text and it looks like there are still missing packages in the channel_v3.json. For example, I couldn't find Tag, Color Highlighter, etc. Just before I finish typing this report, the channel_v3.json has once again become empty as following:
{ schema_version: "3.0.0", repositories: [ ], dependencies_cache: { }, packages_cache: { } }
@lightwave as stated above, you can use that from archive.org. so you can add a channel like this: https://web.archive.org/web/20160103232808/https://packagecontrol.io/channel_v3.json
Thanks @wbond! Good to hear there's some plans to make this more robust.
Thanks for the detailed updates! Package control is still not working for me, but good to know it's being looked at.
https://packagecontrol.io/channel_v3.json is still returning an empty response. And I am afraid our corporate proxy does not allow access to the web archive URL. Is there another alternative?
@kaiser101 You can download the web-archive locally and use the file path to the downloaded JSON file as a channel in "Packages/Package Control/Package Control.sublime-settings" -> "channels" for the time-being.
Edit: I just made a gist using the web-archive page. Here's the URL to the raw version of the channel: https://gist.githubusercontent.com/pradyunsg/874fdebb915980cafc81/raw/6e0a9f1775bfd95812e16194059e07cb48acdfe7/channel_v3.json
@pradyunsg
Thanks a lot for the gist. But it still does not work for me, since the channel entry requires a http/https url which I cannot create locally. Also btw, the json is not complete.
@kaiser101
I'm currently using this as my channel setting:
"channels": [
// "https://packagecontrol.io/channel_v3.json",
"/home/pradyunsg/code/channel_v3.json"
],
And trust me, it works. :smile:
[EDIT] You can download the web-archive locally and use the file path to the downloaded JSON file as a channel in "Packages/Package Control/Package Control.sublime-settings" -> "channels" for the time-being as I'm doing (see above).
Or if you want to use the gist as a fallback while packagecontrol.io is down, change the "channels" key in Packages/Package Control/Package Control.sublime-settings
to
"channels": [
// "https://packagecontrol.io/channel_v3.json"
"https://gist.githubusercontent.com/pradyunsg/874fdebb915980cafc81/raw/6e0a9f1775bfd95812e16194059e07cb48acdfe7/channel_v3.json"
],
@wbond I suggest putting up some sort of notice on packagecontrol.io about this, redirecting to this issue.
@pradyunsg
The json on the gist is incomplete. Would you mind taking a look?
Should be fixed now.
(Sorry for the noise everyone)
Thanks for keeping us updated @wbond, much appreciated.
@kaiser101 I also made a gist containing the archived channel_v3.json
which I have tested working: https://gist.githubusercontent.com/Trippnology/218044242634b30617e9/raw/cdd1d1aa8bd54f8d98c677d89bd37c4c99a53f61/channel_v3.json
I also copied the file to a server of my own so it could be served with gzip to speed up delivery: https://cdn.trippnology.net/spc/channel_v3.json
The official channel is still coming back with no entries at the moment (for me anyway), but please do remember to remove these temporary channels from your settings once it comes back online.
It seems the crawler is running much more slowly, and getting stuck sometimes, possibly due to the mitigation that Linode put in place. It got stuck late last night and the automated package removal kicked in. I've disabled that for now and fixed the status of existing packages.
The main channel is back online, and hopefully should remain so from now on out. If you have questions about the status, view this in your browser:
@wbond Im getting this issue:
Package Control: Installing 1 missing dependency Package Control: The dependency specified, bz2, is not available Package Control: No updated packages
It's also related due this DDoS? Regards.
@wbond
Why don't you use Amazon or some CDN for hosting? Bandwith from Amazon and other CDNs is expensive. At last calculation, using such a hosting provider would likely cost $1,000 USD a month.
so this would be free with maxcdn... why not? could set a pull proxy to update the proxy data every time the channels etc. change data. seems like a great solution to me...
I have not reached out to maxcdn, however that would not by itself make the system fault tolerant. There are some small things that can be done to make an outage less black and white (using the local cache when a connection-level error occurs), but the larger scale things will help future-proof the system and simplify administration.
The current outage was a network issue which was likely to be resolved in a relatively short term without any long-term affect of Package Control, however I've previous dealt with other sorts of outages, such as when the EBS system at AWS locked up machines for over a week back in 2011, and various dedicated hosting provider hardware and power issues. The plan to update the infrastructure to be more distributed, and improve the orchestration of it will allow Package Control to roll with all sorts of different outages in the future. If there is a physical disaster at Linode's Atlanta datacenter, there would need to be a restoration of backups to a new location. The improvements I want to work on would turn that from a mostly hands-on exercise to a mostly automated exercise.
My focus on looking for donations is to support my time in maintaining the system, but also to provide a robust architecture. Honestly, my time is probably about 90%+ of the cost. Part of that is doing things like setting up a status page, testing failure scenarios, converting existing server setup shell scripts to Ansible, setting up Postgres replication and dealing with failover (DNS and DB).
On a typical weekday there are hundreds of thousands of users hitting the server. If I could convince 0.2% of users to donate the equivalent of $2 a month, I could pay myself market rates to work on the service a couple of days a month. So, that is what I am trying to do. I think for many users Package Control is worth the cost of a coffee a month, but I suppose we shall see. :smile:
@wbond maybe @jdorfman from maxcdn can help
We run large deployments with them (aside from using S3/elastic beanstalk etc.) and it's just absolutely great - the pull zones especially for content that gets hit hard. It's a set up and forget process. This would solve pretty much all your bandwidth and uptime issues. It's worth looking into it in case they're up for it to provide it as part of their open source contributions.
@tobsn The real issue is needing to distribute the hosting to multiple providers. Linode has been great, and I am planning on sticking with them because of the excellent service and upgrades I've received over the past four years. However, I do want to get PC's architecture improved for fault tolerance. Adding a CDN in front could help in some ways, but won't help me when the hardware the site is hosted on goes belly-up. Sure, users won't be clamoring for old JSON files from archive.org, but that should be solvable using the existing PC HTTP cache that is saved to disk on every user's machine. I would live to have everything keep humming on the site (usage submissions, websockets, etc) with a failover when an issue occurs. Ideally I want to become more provider agnostic, and diversify so the service doesn't rely on a single provider.
@tobsn thanks for the mention.
@wbond we do offer an "use stale" option that will serve older cache if the origin is down. We would love to be of service free of charge as we do with all of these projects: https://www.maxcdn.com/open-source/
If interested drop me a line: jdorfman at maxcdn dot com
@wbond next if you need help with amazon and elastic beanstalk I can help and I can get you in contact with integrations engineers at amazon - they also have free tier programs for startups (called AWS activate) which, as far as I heard, can be applied for open source projects: https://aws.amazon.com/activate/
with elastic beanstalk you could automatically scale up/down based on traffic/performance of servers etc. without doing anything and, best of it, without any complex puppet/chef scripting. through RDS and elastic caching you also solve the scaling of mysql, memcache, and redis. same process as maxcdn, set up and forget.
for mongodb, in case that is used, my last recommendation would be compose.io, they might be able to help out as well. maybe @Winslett can answer that or one of his coworkers.
essentially you can host for free and CDN for free if you want to take that route. at a minimum I would instantly go with pull zones from maxcdn to offload the biggest amount of traffic. you'd instantly drop from a bazillion requests to almost nothing. can't get better than that as a first step.
(I'll stop now, rest is your decision)
As this issue is long over, I am now closing this status issue. Hopefully in the next month or so I'll have some tweaks made to PC to handle reusing cached data during an outage.
Getting the error again. Looks like the server is down right now. http://www.downforeveryoneorjustme.com/packagecontrol.io
EDIT · Settings workaround using latest snapshot from Jan 9, 2016:
"channels": [
"https://packagecontrol.io/channel_v3.json",
"https://web.archive.org/web/20160109204050/https://packagecontrol.io/channel_v3.json"
]
@mattborn Yup, another Linode outage. Unfortunately I don't have an ETA for moving off of them to another provider yet.
Indeed... http://status.linode.com/
Time to learn how to manually install a package... I bet it is easy, but never had to do it before...
I will try what I've just found:
I am facing the same problem "There is no package for installation", in sublime text 3 build 3114, ubuntu 14.04, below is the log
DPI scale: 1
startup, version: 3114 linux x32 channel: stable
executable: /opt/sublime_text/sublime_text
working dir: /
packages path: /home/flexsin/.config/sublime-text-3/Packages
state path: /home/flexsin/.config/sublime-text-3/Local
zip path: /opt/sublime_text/Packages
zip path: /home/flexsin/.config/sublime-text-3/Installed Packages
ignored_packages: ["Vintage"]
pre session restore time: 0.190626
startup time: 0.339059
first paint time: 0.347232
first paint time: 0.35137
reloading plugin Default.auto_indent_tag
reloading plugin Default.block
reloading plugin Default.comment
reloading plugin Default.convert_syntax
reloading plugin Default.copy_path
reloading plugin Default.delete_word
reloading plugin Default.detect_indentation
reloading plugin Default.duplicate_line
reloading plugin Default.echo
reloading plugin Default.exec
reloading plugin Default.fold
reloading plugin Default.font
reloading plugin Default.goto_line
reloading plugin Default.history_list
reloading plugin Default.indentation
reloading plugin Default.kill_ring
reloading plugin Default.mark
reloading plugin Default.new_templates
reloading plugin Default.open_context_url
reloading plugin Default.open_file_settings
reloading plugin Default.open_in_browser
reloading plugin Default.pane
reloading plugin Default.paragraph
reloading plugin Default.paste_from_history
reloading plugin Default.profile
reloading plugin Default.quick_panel
reloading plugin Default.run_syntax_tests
reloading plugin Default.save_on_focus_lost
reloading plugin Default.scroll
reloading plugin Default.set_unsaved_view_name
reloading plugin Default.show_scope_name
reloading plugin Default.side_bar
reloading plugin Default.sort
reloading plugin Default.swap_line
reloading plugin Default.switch_file
reloading plugin Default.symbol
reloading plugin Default.transform
reloading plugin Default.transpose
reloading plugin Default.trim_trailing_white_space
reloading plugin CSS.css_completions
reloading plugin Diff.diff
reloading plugin HTML.encode_html_entities
reloading plugin HTML.html_completions
reloading plugin 0_package_control_loader.00-package_control
reloading plugin 0_package_control_loader.01-ssl-linux
Package Control: Linux SSL: successfully loaded _ssl module for libssl.so.1.0.0
reloading plugin Package Control.1_reloader
reloading plugin Package Control.2_bootstrap
reloading plugin Package Control.Package Control
plugins loaded
Package Control: Error submitting usage information. HTTP exception InvalidCertificateException (Host packagecontrol.io returned an invalid certificate ([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:548))) downloading https://packagecontrol.io/submit?package_control_version=3.1.2&sublime_platform=linux&operation=install&version=3.1.2&package=Package+Control&sublime_version=3114.
Package Control: Installing 1 missing dependency
Package Control: Error downloading channel. HTTP exception InvalidCertificateException (Host packagecontrol.io returned an invalid certificate ([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:548))) downloading https://packagecontrol.io/channel_v3.json.
Package Control: The dependency specified, bz2, is not available
Package Control: Skipping automatic upgrade, last run at 2016-05-16 16:51:25, next run at 2016-05-16 17:51:25 or after
Package Control: Error downloading channel. HTTP exception InvalidCertificateException (Host packagecontrol.io returned an invalid certificate ([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:548))) downloading https://packagecontrol.io/channel_v3.json.
Package Control: Error downloading channel. HTTP exception InvalidCertificateException (Host packagecontrol.io returned an invalid certificate ([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:548))) downloading https://packagecontrol.io/channel_v3.json.
Please provide some information how to fix it?
On Windows, this error was showing for me this morning, until I updated Package Control and ran the code for Windows users listed in the release notes.
https://github.com/wbond/package_control/blob/master/messages/3.0.0.txt
There looks like there's another outage. sublime.wbond.net is down.
I am also not able to get the packages.
Debug log:
WINDOW COMMAND ENABLED True
WINDOW COMMAND ENABLED True
WINDOW COMMAND ENABLED True
WINDOW COMMAND ENABLED True
WINDOW COMMAND ENABLED True
WINDOW COMMAND ENABLED False
Package Control: Fetching list of available packages and dependencies
Platform: linux-x64
Sublime Text Version: 3126
Package Control Version: 3.2.1
Package Control: Download Debug
URL: https://packagecontrol.io/channel_v3.json
Timeout: 30
Resolved IP: 50.116.34.243
Resolved IPv6: 2600:3c02:e000:42::1
Package Control: Urllib Debug Proxy
http_proxy:
https_proxy:
proxy_username:
proxy_password:
Package Control: Found system CA bundle at /usr/lib/ssl/certs/ca-certificates.crt (274340 bytes)
Package Control: Urllib HTTPS Debug General
Connecting to packagecontrol.io on port 443
Package Control: Urllib HTTPS Debug General
Upgrading connection to SSL using CA certs file at /home/meetsc/.config/sublime-text-3/Packages/User/Package Control.merged-ca-bundle
Successfully upgraded connection to packagecontrol.io:443 with SSL Using TLSv1/SSLv3 with cipher ECDHE-RSA-AES128-SHA
Server SSL certificate:
subject: CN=packagecontrol.io
subject alt name: packagecontrol.io, www.packagecontrol.io
expire date: Nov 27 23:59:59 2019 GMT
Certificate validated for packagecontrol.io
Package Control: Urllib HTTPS Debug Write
GET /channel_v3.json HTTP/1.1
If-None-Match: "589dc05e-46023"
If-Modified-Since: Fri, 10 Feb 2017 13:30:06 GMT
Host: packagecontrol.io
User-Agent: Package Control v3.2.1
Accept-Encoding: bzip2,gzip,deflate
Connection: Keep-Alive
Package Control: Urllib HTTPS Debug Read
HTTP/1.1 200 OK
Server: nginx
Date: Mon, 13 Feb 2017 14:10:50 GMT
Content-Type: application/json
Content-Length: 287338
Last-Modified: Mon, 13 Feb 2017 14:10:05 GMT
Connection: keep-alive
ETag: "58a1be3d-4626a"
Content-Encoding: bzip2
Vary: Accept-Encoding
Strict-Transport-Security: max-age=15552000
Accept-Ranges: bytes
Package Control: Caching https://packagecontrol.io/channel_v3.json in /home/meetsc/.config/sublime-text-3/Packages/User/Package Control.cache/01524fae79697630d0454ba3fabd9414
Package Control: Connection went away while trying to download https://packagecontrol.io/channel_v3.json, trying again
Package Control: Urllib Debug Proxy
http_proxy:
https_proxy:
proxy_username:
proxy_password:
Package Control: Found system CA bundle at /usr/lib/ssl/certs/ca-certificates.crt (274340 bytes)
Package Control: Urllib HTTPS Debug General
Connecting to packagecontrol.io on port 443
Package Control: Urllib HTTPS Debug General
Upgrading connection to SSL using CA certs file at /home/meetsc/.config/sublime-text-3/Packages/User/Package Control.merged-ca-bundle
Successfully upgraded connection to packagecontrol.io:443 with SSL Using TLSv1/SSLv3 with cipher ECDHE-RSA-AES128-SHA
Server SSL certificate:
subject: CN=packagecontrol.io
subject alt name: packagecontrol.io, www.packagecontrol.io
expire date: Nov 27 23:59:59 2019 GMT
Certificate validated for packagecontrol.io
Package Control: Urllib HTTPS Debug Write
GET /channel_v3.json HTTP/1.1
If-None-Match: "589dc05e-46023"
If-Modified-Since: Fri, 10 Feb 2017 13:30:06 GMT
Host: packagecontrol.io
User-Agent: Package Control v3.2.1
Accept-Encoding: bzip2,gzip,deflate
Connection: Keep-Alive
Package Control: Urllib HTTPS Debug Read
HTTP/1.1 200 OK
Server: nginx
Date: Mon, 13 Feb 2017 14:10:56 GMT
Content-Type: application/json
Content-Length: 287338
Last-Modified: Mon, 13 Feb 2017 14:10:05 GMT
Connection: keep-alive
ETag: "58a1be3d-4626a"
Content-Encoding: bzip2
Vary: Accept-Encoding
Strict-Transport-Security: max-age=15552000
Accept-Ranges: bytes
Package Control: Caching https://packagecontrol.io/channel_v3.json in /home/meetsc/.config/sublime-text-3/Packages/User/Package Control.cache/01524fae79697630d0454ba3fabd9414
Package Control: Connection went away while trying to download https://packagecontrol.io/channel_v3.json, trying again
Package Control: Urllib Debug Proxy
http_proxy:
https_proxy:
proxy_username:
proxy_password:
Package Control: Found system CA bundle at /usr/lib/ssl/certs/ca-certificates.crt (274340 bytes)
Package Control: Urllib HTTPS Debug General
Connecting to packagecontrol.io on port 443
Package Control: Urllib HTTPS Debug General
Upgrading connection to SSL using CA certs file at /home/meetsc/.config/sublime-text-3/Packages/User/Package Control.merged-ca-bundle
Successfully upgraded connection to packagecontrol.io:443 with SSL Using TLSv1/SSLv3 with cipher ECDHE-RSA-AES128-SHA
Server SSL certificate:
subject: CN=packagecontrol.io
subject alt name: packagecontrol.io, www.packagecontrol.io
expire date: Nov 27 23:59:59 2019 GMT
Certificate validated for packagecontrol.io
Package Control: Urllib HTTPS Debug Write
GET /channel_v3.json HTTP/1.1
If-None-Match: "589dc05e-46023"
If-Modified-Since: Fri, 10 Feb 2017 13:30:06 GMT
Host: packagecontrol.io
User-Agent: Package Control v3.2.1
Accept-Encoding: bzip2,gzip,deflate
Connection: Keep-Alive
Package Control: Urllib HTTPS Debug Read
HTTP/1.1 200 OK
Server: nginx
Date: Mon, 13 Feb 2017 14:11:01 GMT
Content-Type: application/json
Content-Length: 287338
Last-Modified: Mon, 13 Feb 2017 14:10:05 GMT
Connection: keep-alive
ETag: "58a1be3d-4626a"
Content-Encoding: bzip2
Vary: Accept-Encoding
Strict-Transport-Security: max-age=15552000
Accept-Ranges: bytes
Package Control: Caching https://packagecontrol.io/channel_v3.json in /home/meetsc/.config/sublime-text-3/Packages/User/Package Control.cache/01524fae79697630d0454ba3fabd9414
Package Control: Connection went away while trying to download https://packagecontrol.io/channel_v3.json, trying again
Package Control: Urllib Debug Proxy
http_proxy:
https_proxy:
proxy_username:
proxy_password:
Package Control: Found system CA bundle at /usr/lib/ssl/certs/ca-certificates.crt (274340 bytes)
Package Control: Unable to download https://packagecontrol.io/channel_v3.json after 3 attempts
I had the same issue the proxy in sublime was not working:
Package Control: Error downloading channel. HTTP exception InvalidCertificateException (Host 127.0.0.1:3129 returned an invalid certificate ([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:548))) downloading https://web.archive.org/web/20160109204050/https://packagecontrol.io/channel_v3.json.
and the way I solved it is to download the file locally and put it in a folder and put this line in the channels:
{
"bootstrapped": true,
"installed_packages":
[
"Package Control"
],
"debug": true,
"channels": [
"/Users/dproios/Documents/channel_v3.json"
]
}
(I post it as a workaround because a lot of issues i saw dont mention it and in documentation of Package control Settings there was not a mention in the local file workaround)
~~@partizanos your issue is https://github.com/wbond/package_control/issues/1334. ~~
Nvm, reading this on mobile caused me to skip the actual exception text. Just make sure that you update your locally cached file every now and then.
I'll set a longer cache period in a PR at some point so that packages can still be installed when pc.io is down.
All issues created related to the DDoS of Linode Atlanta will be redirected here. If you have questions other than "when will it be fixed," please post them here.
Current Status: Online
Linode, where Package Control is hosted, has been a wonderful host. Over the past four years, the traffic to PC has increased from a couple hundred requests per day, to around 5TB of bzip2'ed data per month. Uncompressed, PC ships over a terabyte of JSON per day. With the custom bz2 HTTP transport encoding compression, that is usually around 120-150GB.
During that time, there have only been a few minor outages, largely due to some network routing and server maintenance.
Starting on December 25th, 2015, some organization has been attacking various Linode data centers with a distributed denial of service (DDoS) attack. On January 1st, 2016 they started targeting the Linode data center in Atlanta. http://status.linode.com/ has details of Linode's response, and some high-level information about the varied attacks.
It appears that as of 8pm UTC on January 3rd, we are back online, however there have been a few smaller outages in the past few hours.
Infrastructure Plan
Even though this has been the first major outage of Package Control, it is not something I'd like to repeat. My high-level plan to mitigate such outages in the future is:
Software Plan
The following work will ideally be implemented over time to help deal with outages, both in the server, crawler and client software:
Donating
In order for Package Control to get the attention it needs to "just work," both time and resources are required. There will be monthly costs associated with at least one more server and the DNS through Amazon. There will also be work needed to set up an infrastructure that continues to be secure, but is fault tolerant.
Because of these issues, my biggest need in donation is recurring monthly support. If we can get a significant number of users to donate just $2.50 a month (about the cost of one coffee), the total amount will be sufficient to both give me time to work on the infrastructure, and pay for the various hosting fees.
https://packagecontrol.io/say_thanks
Finally, I would like to say a huge thank you to each and every one of you who have donated and supported Package Control over the past four years. When I started out, I definitely never expected to be providing a service to hundreds of thousands of users every day.
Frequently Asked Questions
Workarounds
If you have implemented a workaround, such as adding an archive.org URL to fetch old channel data, please be sure to remove that once things are running smoothly. That data is over 3 months old, and could likely cause issues in the future