Offering openstreetmap a free Skylight account

[wycats]

Per @tomhughes (https://github.com/openstreetmap/openstreetmap-website/issues/1713#issuecomment-356072322), I'm reposting this here:

---

Hello OpenStreetMap maintainers!

We’re planning on making Skylight free for open source projects this year and want to offer you a free beta account to use Skylight on your Rails app for free.

If you’re not already familiar with Skylight, it is a smart profiler for Ruby on Rails apps. Skylight makes it easy to pinpoint performance issues in Rails applications.

I work on a lot of open source projects myself, and in my experience it can be pretty hard to get existing contributors to work on application performance issues. Few contributors consider working on performance problems, and the ones that might be interested may not even know where to start. By making performance information more accessible, we hope to inspire potential contributors to tackle slow parts of your app, and have a good way to see if their contributions helped.

This feature is still in beta at the moment for a few reasons. First, we expect that open source apps will add a significant load to our servers, so we plan to phase in free accounts so we can make sure we scale appropriately as the new accounts come online. We also want to be able to address any issues that you as maintainers may find, as this feature required some significant changes to our existing codebase.

Adding Skylight to a Rails application is very easy. All you have to do is add the gem to your Gemfile and set an environment variable with your authentication token.

If you’re interested in participating or if you have any questions, you can reach us at support@skylight.io. We'd also love to feature you on our website as one of the first open source organizations using Skylight. We hope you’ll join us in helping to make performance an accessible way to contribute to an open source project.

---

I'm not sure if opening a github issue is the ideal way of getting in touch but I didn't have better ideas. At this point, we're offering a free account to a handful of apps; each one was selected by someone on our team as an open source Rails app with a good community or an app they personally care about. This is not a broad-based message we're sending to hundreds of accounts.)

[tomhughes]

Sorry I mean to open a ticket here but hadn't got around to it yet...

[zerebubuth]

Are there any open source Skylight alternatives that we should be evaluating against with respect to the OSMF FOSS policy?

[pnorman]

Would this be on OSMF servers, or a development tool similar to coverity?

[tomhughes]

@pnorman it's a hosted thing where we add a gem that's submits statistics to a hosted service, so coverity is the right kind of model.

[pnorman]

If it's like Coverity, I don't see that the OSMF policy applies, since it's the development community which makes those decisions. I certainly don't see that the OSMF policy impacts my (infrequent) use of coverity with osm2pgsql.

It's still a useful question, as if there is anything equivalent that's open source, I'd prefer it even if the policy doesn't apply.

[wycats]

@pnorman I'm not aware of anything equivalent (specifically, application performance monitoring for Rails apps rather than server monitoring) that's open source, but I could be missing something.

[tomhughes]

@pnorman This is collecting stats from the production site, not something that the devs would run periodically as a test.

[zerebubuth]

The closest thing a quick search on Github turned up was AppPerf, which looks interesting, but as the author disclaims: "This application is in extremely beginning stages".

There seems to be some standardisation around OpenTracing, which it's possibly to use from Ruby but seems to me like a very low-level tool for investigating performance problems rather than discovering them.

We had a blunt attempt at a very high-level kind of performance monitoring in munin, but it wasn't very easy to dig down into.

As @tomhughes says, to be most useful, this would need to run on production servers, although it wouldn't be required for the site to work properly from the users' perspectives. However, it would send data to a 3rd party service, so we should give careful consideration to the implications of that.

[pnorman]

Aside from the OSMF policy, isn't there also a long-standing OWG policy of only running open-source software on our hardware? I realize there's question on if this is "running" or not, but it gets close to the intent of the policy.

The PII implications are a good question, and we'd have to look at what gets sent.

[tomhughes]

@pnorman I'm not aware of any such policy... I mean I'm not sure if we've ever run anything user facing that isn't open source but we certainly have admin tools (hp-health, various raid tools, etc) that aren't.

[chancancode]

Hello! I also work at Skylight. I realize there is still an ongoing policy discussion going on, but if anyone wants to preview/evaluate/play with this, we just finished setting this up for a few other orgs. For example, this is the Skylight dashboard for The Odin Project. There are more apps on the Skylight for OSS homepage as well (more are being added).

[mmd-osm]

I tested skylight locally to analyze one of our most expensive calls, namely the changset upload. First observation: changesets with more than 50 buildings already hit skylight's tracing limit:

ERROR:skylight::c_api: failed to start tracing span; err="Exceeded maximum number of trace spans; MAX=2048"
[SKYLIGHT] internal error: failed to start tracing span

Fair enough, we can still test some smaller changesets for the time being. First results were not really conclusive: there are many allocations reported by the tool, but it's not clear what's going on. Let’s add some instrumentation code to commit and upload methods (hey, that’s adding instrumentation code to productive code and redeploying to prod!!), still not clear where those allocations are coming from and where exactly the time is spent:

At least the skylightd daemon running in background all the time seems to use encryption when communicating with the skylight server.

[Firefishy]

We use newrelic at work and it has really helped the team identify issues and make improvements to our production apps.

Skylight looks like a good alternative.

[pnorman]

I had a look at some data from a more "typical" site that I know as a user, LinuxFest Northwest's conference site. It looks useful for that site for identifying performance problems. They also link to it in their footer, which is how I found it there.

The number of trace spans issue above makes me wonder if it'll be as useful for a site like ours.

[mmd-osm]

Beta period seems to be over, Skylight offers this for free for Open Source projects, see https://www.skylight.io/support/skylight-for-open-source#requirements

[pnorman]

As the beta is over this issue is no longer necessary.