convert it to new RCStream?

[skierpage]

Awesome project, it's one of the projects that https://meta.wikimedia.org/wiki/WikimediaKiosk cycles through.

But the IRC recent changes will eventually go away, instead there's the Socket.IO-based RCStream feed. Do you have any plans to convert to it? If so I would feature this in the tech article Recent changes stream. Fame and fortune awaits :)

[edsu]

Is the IRC recent changes really going away? Do you know where this is being discussed? The stream.wikimedia.org/rc was recently down for days and nobody seemed to notice... There must be a great number of anti-vandalism tools that use the IRC streams aren't there? Or are they being converted over?

All that being said, it would greatly simplify wikistream if it didn't need to watch the channels.

Also, @WardCunningham already wrote an email to wiki-research saying how much he liked Wikistream. That's all the fame & fortune I need :smile_cat:

[WardCunningham]

Wikistream was painless to install because it was already in node which I understood.

I had a go at RCStream which seemed straight forward enough. I had some hitch getting it going but the RCStream folks at Wikimedia Foundation were kind enough to help me straighten it out. I hope Wikistream can be revised to provide the same service with the new mechanism underneath.

[mahmoud]

Hi there! As a maintainer of Listen to Wikipedia, another site on the Kiosk, I hope no one minds if I butt in a bit. :)

I know Ori (@atdt), the author of the new change streaming protocol, and spoke with him about it before its release. Unless something has changed, the main reason for the new protocol was to ease integration for certain newer stacks which may have better WebSocket than IRC support (i.e. JavaScript/frontend), as well as make it easier for WMF to add more features to the stream (the IRC format is already hampered by nontrivial parsing).

I very much doubt that the IRC features are going away. They offer key transparency for many Wikipedias with slower changerates. They are also a feature of MediaWiki that requires very little external setup, as opposed to the WebSockets approach, which I believe requires Redis. So, there are likely many non-WMF MediaWiki installations out there using the IRC method of change notification, so to break with that would be conspicuously nonstandard.

Finally, how difficult is it to run IRC servers? I understand this sort of FUD can be tough to quash in a time when companies like Google are constantly deprecating functional software people use, but I remain firmly convinced WMF is different in this respect. ;)

[skierpage]

Is the IRC recent changes really going away? Do you know where this is being discussed?

It seems to be a shared assumption that motivated the development of RCStream/stream.wikimedia.org, e.g. T87780 "Add deprecation notice to IRC RC feed", and https://wikitech.wikimedia.org/wiki/Irc.wikimedia.org "It is a vague future goal to deprecate this service". I've asked for a more definitive answer.

The stream.wikimedia.org/rc was recently down for days and nobody seemed to notice...

D'oh! Please file bugs at https://phabricator.wikimedia.org/tag/wikimedia-stream/

@mahmoud, I don't think there are plans to remove the IRCColourfulRCFeedFormatter from MediaWiki code, even if and when Wikimedia wikis prefer publishing JSONRCFeedFormatter changes using the RedisPubSubFeedEngine.

Thanks for everything y'all do! (I'm just the writer :) )

[atdt]

The stream.wikimedia.org/rc was recently down for days and nobody seemed to notice...

@edsu Are you sure? We have alerts set up for it. And I don't see evidence of an outage in the network graphs:

RCStream needs to be substantially better if we want usage to take off. One trivially simple thing I have been meaning to do but haven't yet is upload a package to npm that expresses a dependency on the proper version of Socket.IO, so you could just npm install rcstream and get a working client.

[edsu]

@atdt No I'm not sure. Does that measure traffic out to the Internet? Is it possible that something was down in between, a proxy of some kind?

What I noticed is that the demo linked from here was broken and a couple days after I asked about the stream possibly being broken on wiki-research the demo started working again. Perhaps the demo was fixed instead of the stream? I'm sorry if it seems like I'm pointing fingers.

[edsu]

@skierpage thanks for the link to that discussion. I'll see if I can move this over to using the stream when I have a chance.

[hqtan]

Hi! I would like to also confirm @edsu's finding - I couldn't get the CodePen demo mentioned here to work either (running it from the codepen website). Running the code snippet today (Fri 2015-05-16 17ish UTC+1000) showed the following output:

{
    "event": "error"
}

Not sure whether it's the code snippet that is at fault, or the codepen website (or me!)... Once i have node.js installed on localhost, i'll run that code snippet again to see if i get the same result.

Also, I tried running the python code from here as well, with no success either:

$ cat wikipedia_rcstream.py
#!/usr/bin/python
# -*- coding: utf-8 -*-
import socketIO_client

class WikiNamespace(socketIO_client.BaseNamespace):
    def on_change(self, change):
        print('%(user)s edited %(title)s' % change)

    def on_connect(self):
        self.emit('subscribe', 'en.wikipedia.org')
        print('CONNECTED')

socketIO = socketIO_client.SocketIO('stream.wikimedia.org', 80)
socketIO.define(WikiNamespace, '/rc')

socketIO.wait()

$ python wikipedia_rcstream.py
WARNING:root:stream.wikimedia.org:80/socket.io [waiting for connection] unexpected status code (404 404 Not Found)

Is it the code that's borked, or is RCStream not working as expected?

Apologies in advance for what might be silly questions!

Also, awesome work with wikistream as well, @edsu !

[edsu]

@hqtan looks like the demo is working again?