We should have a status page at status.crates.io

carols10cents commented 7 years ago

I think crates.io is a big enough deal to the Rust community at this point that we should have a place people can check if crates.io isn't working. Even if crates.io isn't big enough yet, I think this is a good step to take in preparation for when we are :)

Requirements/ideas:

Hosted somewhere that's not heroku so that it's accessible even if heroku is down
- Github pages?
MVP: a static page hosted on github pages that just displays tweets from a crates.io status twitter account that we can post to, things like "we're currently investigating failed crate publishes" etc.
- Advantages: don't have to actually touch the website to make an update, people can follow the twitter account as well
I think the page should post something when we've deployed or rolled back, and what commits were deployed or rolled back, in the hopes that people might be able to help debug issues that happen because of a deploy. I think heroku has a post-deploy API hook thing we could use?
It'd be super awesome to show graphs like the heroku metrics about throughput/response time, sort of like github's status graphs.

I'd love ideas or help implementing this.

steveklabnik commented 7 years ago

I have pre-reserved a twitter account for this purpose.

carols10cents commented 7 years ago

I have pre-reserved a twitter account for this purpose.

share? :) :3

cafreeman commented 7 years ago

@carols10cents @steveklabnik so, here's a thought, but feel free to shoot this down as "no, already too complicated.": Glimmer standalone + now.sh.

Gives us a pretty lightweight way to make a FE-only page that can still be pretty dynamic, is Ember-y so it's not a huge departure from what the rest of crates.io is doing, and ember build -prod && now ./dist is a pretty awesome deploy step.

cafreeman commented 7 years ago

Oh wait, just saw the $14.99 a month for custom domain with Now, oof.

cafreeman commented 7 years ago

non-Ember-y solutions are cool too, btw. Mostly I'd like to help out with this and thought I'd start the conversation out with an idea :)

steveklabnik commented 7 years ago

I think Glimmer would make a lot of sense, as you want some dynamism but also fundamentally want a static page.

... and totally not because I'd like an excuse to play with it 😅

cafreeman commented 7 years ago

Well, it's real ugly and simple, but here's something that might get the ball rolling? https://github.com/cafreeman/crates-status

I've pointed it at the @rustlang account, but you can change it here.

cafreeman commented 7 years ago

@carols10cents do you happen to know if Crates.io currently has the "github stats"-like numbers available?

cafreeman commented 7 years ago

Starting to look a little nicer!

If you want to point me to the crates-status twitter account and talk about other content you'd like to include, I'm happy to keep heading this direction :)

carols10cents commented 7 years ago

Hi @cafreeman! Thank you for your work on this so far, sorry that I've let this slip through the cracks!

do you happen to know if Crates.io currently has the "github stats"-like numbers available?

The only metrics we have on crates.io right now are the builtin heroku metrics and it doesn't look like they have a, like, dashboard widget API or anything but if you could search around and see if there's anything, that would be a good first step I think.

If not, could you investigate what other heroku metrics addons might work with a rust backend, have a dashboard widget thing you could embed on the status page, and not be a million dollars? Does newrelic work with Rust backends, for example?

The twitter account is https://twitter.com/cratesiostatus :)

cafreeman commented 7 years ago

@carols10cents @steveklabnik Hi there! I want to apologize about having just gone dark on this for a while. We were launching a product at work and I just had zero bandwidth left over there for a while, but I found enough time to at least do some research on this problem!

So, to my thinking, in order to get at "github stats"-like numbers, we basically need two things:

The ability to collect metrics and possibly aggregate them depending on how fine-grained the reporting is
The ability to serve those metrics as data over a REST API that we can access from the glimmer app in order to populate some neat (or at least passable) visualizations.

Heroku clearly has the first item covered, as evidenced by the their logplex stuff and the metrics dashboard that Carol linked. I started going down this route to see how we might get at that underlying data, but eventually found this blog post about the launch of the metrics dashboard. The tl;dr on this is that those dashboards are populated by metrics APIs, but "We would like to eventually expose these endpoints officially as part of the Heroku API, but that work is not currently scheduled." In other words, we're not likely to access the data directly from Heroku any time soon.

So, next up, I started looking for possible add-ons and integrations and came across a heroku add-on called Librato. I spoke to a friend of mine who works at Heroku and he said that they actually use Librato for a lot of internal monitoring stuff as well, and he said he'd definitely recommend it. I dug into a bit and it seems like it would probably fit our needs exactly. It slurps up log/metric info from heroku apps and transforms them however you like. It gives you the custom dashboard stuff, but more importantly, it also appears to expose all your metrics via an API as well!

Their API docs can be found here and they seem to offer up exactly the kind of stuff we'd want access to. Furthermore, I found a blog post from TravisCI where they talk about using Librato to monitor their own Heroku apps with great success, so it seems like this might be a good place to look.

To sum up, I think there is a way to get metrics on the health of crates.io and display them in a cool gitub-y way, but I believe it'll require a couple intermediate steps:

Someone who actually controls the Heroku stuff will probably need to poke around at the Librato add-on, try adding it to crates.io, and see if we can actually starts exposing the metrics we care about via a REST API. This is almost certainly a thing I can't help with since you don't want me poking around in your heroku stuff
We will likely want to expose this stuff via an intermediate API that lives separate from crates.io. Or least on a separate dyno or something? Basically I think we want this intermediate layer in charge of authenticating with librato and pulling out/transforming data before passing it on to the stats page, and we don't it to be down whenever crates.io is down. This is a thing I would be glad to help with once the data for that stuff is available from Librato
Start pulling that data into the front-end and building out some viz for whichever metrics we decide we'd like to see right off the bat. This is also a thing I would be glad to help with

Sorry about the small book I just wrote you, I recognize this might be more work than you were hoping to put into it and I understand if you decide to go another direction, but I wanted to at least save y'all the trouble of doing all the research!

Let me know what you think and if you have suggestions for other options/places to look :)

cafreeman commented 7 years ago

Also, I don't think I ever posted the link to the actual deployed version of this, but this status page does actually exist in real life now: http://crates-status.surge.sh/

carols10cents commented 7 years ago

I wanted to at least save y'all the trouble of doing all the research!

This is amazing!! Thank you!!! I heard about librato a while ago but hadn't done the research to see if it's something we should enable, and the info you have here is exactly what I wanted to know ❤️ ❤️ ❤️ ❤️

I've enabled the free Librato add-on for now, I can talk to more people about the $ plans if we need something from those. It looks like the biggest limitation on the free plan is that you only get the last hour of data, which should be fine for now.

Although it looks like I can't make API tokens without a paid plan either. I made a public shared space though, which said "This is your publicly sharable link for this space. Links (tokens) can be be managed and updated within Share History" when I created it? I'm not sure if this gives you a read-only way to access the API somehow, it looks like it's making API requests though?

https://metrics.librato.com/s/public/ntsit4dsx is the shared space, I see it making requests like https://metrics.librato.com/metrics-api/v1/share/ntsit4dsx/spaces/429726/sdk_charts/4245042/streams/29082193?start_time=1495023250&end_time=1495026850&resolution=1&sources%5B%5D=*&complex_gauge=true

There's nothing sensitive in these metrics, no one freak out, the point of this is to make these metrics public :P

caspark commented 7 years ago

Hey, I didn't realise we already had some discussion on this, so I started a thread suggesting using statuspage.io (I'm on the dev team) a day or two ago: https://github.com/rust-lang/rust/issues/42308 which led to https://internals.rust-lang.org/t/feedback-on-ruststatus-twitter-and-anything-else-status-related/5331

And by the way @cafreeman your research into displaying metrics on a status page is pretty spot on actually: the Librato approach is pretty close to how we do it at StatusPage.io, with some complications around letting people push their own data points to us directly as well as supporting slurping in data from metrics providers other than Librato :)

cafreeman commented 7 years ago

@caspark nice! I came across statuspage.io a while ago it did look very promising, but I have no clue what the budget for that kind of service is as far as Rust itself is concerned.

@carols10cents if you'd prefer to go with statuspage.io that seems like a reasonable option? Otherwise, I'm clearing time this weekend to dig into all the Librato stuff you added and start pulling metrics.

cafreeman commented 7 years ago

@carols10cents heads up, it looks like you probably still need an API key to query the metrics API itself? I can definitely spy on the XHRs going out from the dashboard that you linked and curl those URLs successfully, but they don't appear to expose the actual API options that you see in the documentation here.

For instance, I tried hitting the "list all metrics" endpoint with this:

curl -X GET \
  https://metrics.librato.com/metrics-api/v1/share/ntsit4dsx/metrics \
  -H 'cache-control: no-cache' \
  -H 'content-type: application/json'

And got a 401 Unauthorized

{
  "errors": {
    "request": [
      "Authorization Required"
    ]
  }
}

I looked into the "Shared Spaces" thing and found this article which seems to indicate that it's probably just intended for sharing the entire dashboard, rather than being a publicly exposed API.

If you still want to go this route (rather than statuspage.io), and you're able to generate an API key I can try using that?

carols10cents commented 7 years ago

@cafreeman Sorry for the delay on this! Here's a read-only API key to librato: 4d0c9f61f7ce2dba4e1112183de6242af2451d924963595ed748e452d8651edb

I don't think we've decided which route we'd rather go, I'm up for trying either librato or statuspage or a combination of both or idk? :)

cafreeman commented 7 years ago

No worries! Now that I've got an API key I can see what we can get out of Librato :)

ashleygwilliams commented 6 years ago

i've taken over working on this thanks ya'll. we've removed the librato integration and are using apex ping and i'll be setting up status page to talk to our heroku metrics. you can check out http://ping.crates.io/ and i'll write about when i get status page set up!

caspark commented 6 years ago

No worries @ashleygwilliams , thanks for getting this going again :)

If Apex Ping doesn't work out for whatever reason, we over at Statuspage.io would be more than happy to hand out a free page - plenty of Rust fans on the dev team here who'd be proud to help out!

jtgeibel commented 6 years ago

@ashleygwilliams since this is a CNAME to status.ping.apex.sh I'm guessing that we can't use https on this endpoint. If that is the case, then I'll likely close issue #1325 which would require all subdomains to be on https.

kzys commented 5 years ago

Hello folks,

Now we have http://ping.crates.io and https://twitter.com/cratesiostatus. What's remaining now?

sgrif commented 5 years ago

Nothing AFAICT. Thanks.

rust-lang / crates.io

We should have a status page at status.crates.io #662