search

w3c / modern-tooling

Work of the modern tooling task force
http://w3c.github.io/modern-tooling/
MIT License
44 stars 39 forks source link

Consider fedmsg as data-transport protocol and backbone for Unified feeds? #44

Open odinho opened 9 years ago

odinho commented 9 years ago

I took a look at the Unified feeds-part, and was reminded of http://www.fedmsg.com/ which I've been considering transforming some of our internal systems in Opera to use.

Some of the benefits would be that some other big, open projects (notably Fedora and Debian) are using this already to expose their infrastructure to third-parties, and to have a more common way for services to connect.

There's a tool called 'datanommer', which stores all of the messages that was seen on the bus, that one has a normal REST-API where it is possible to query everything that has happened.

There's also a 'fnm' a notification center where usrse can set up what they want notifications for (on IRC and email currently): https://github.com/fedora-infra/fmn http://threebean.org/blog/category/fmn/.

darobin commented 9 years ago

I looked at this, but I see some limitations. At the input layer, it would seem that we'd have to write adapters to input data from our sources (most of the data comes in from GitHub), unless I missed something. For querying, it looks like you can query high-level attributes but you can't query inside the JSON you store (again, maybe there's a way I haven't seen). This would limit the usefulness of the system compared to just dumping the events into something like Elastic, no?

odinho commented 9 years ago

Well, fedmsg is mostly just a high level infra communication protocol built on the awesome ZeroMQ. Querying is not part of it at all, you have to have applications to do that.

They have some documentation on "why's", though a reason I want it at Opera is that it's battle-tested and created as a solution to problems arising from having many services. We want to do more of our internal stuff as microservices, but many of them need to communicate effectively. There's quite a lot coming for free since it's been in use in a huge project for a long time already, so some things like FNM we get without even writing it. There's still many things which would need to be written, but that's no different from not using fedmsg. It actually just makes everything speak the same language, which is a huge benefit.

Stuffing the messages and other things into ElasticSearch sounds like a good idea, that is basically a simple thing to do as well. So actually something like Unified Feed could use fedmsg as the way of getting data from other sources. You'll build a better microservice architecture that way, where other people also can hook in to the actual streams they want to listen to. It's also makes for less of a single-point-of-failure architecture.

Datanommer is just an application using fedmsg, so this Unified Feeds app would also just be an app that did this. The feeds app could probably also be direct dependency for some other things like e.g. the Dashboard. So that the dashboard itself doesn't know anything about fedmsg, it only talks Unified Feeds-native API's. Things like this is fully possible, and indeed is the way some of the services are done.

About the github thing, I was going to say you'd need an adapter for that, probably to write one, but doing a qucik search I see one already exists: https://github.com/fedora-infra/github2fedmsg https://apps.fedoraproject.org/github2fedmsg