Should we use a framework/library and if so which one.

[neilcollins]

Suggestions so far:

• IRC
• 0MQ: http://www.zeromq.org/bindings:python
• HTTP client: http://docs.python.org/2/library/urllib2.html or http://docs.python-requests.org/en/latest/
• HTTP server: http://www.cherrypy.org/
• Web application framework: https://www.djangoproject.com/

Feels at this stage that IRC is best fit but needs some thought/investigation.

[peterbrittain]

I think that there is little point reinventing the wheel, so we should probably use a framework of some sorts. Before we really decide what it is, though, we probably need to decide what the basic interop requirements are.

Based on the HLD discussion, I think we're probably safest using HTTP in order to avoid firewall issues. There are some concerns about polling, though. IIUC, the expectation is that each Pi runs a client that can:

• register/deregister with the server
• check connectivity to the server
• create a conga (with optional password)
• post a message to be published on a conga
• recieve a notification that a message has arrived on a conga
• list who is in a conga
• report abuse/kick someone out of the conga (if you created it)
• check for any nearby Pis (though I'm not quite sure how this feeds in).

Most of that fits fairly well with HTTP. The only issue I see is the notifications. Anyone got experience of COMET for this sort of thing? I see that it's supposed to be possible with CherryPy - see http://tools.cherrypy.org/wiki/Comet - but have never tried myself and so don't know the scalability limits of such a solution.

Failing that, does anyone have a better idea for which protocol to use - and how we would ensure that it can safely traverse school firewalls?

[Lukasa]

If we go the HTTP route, I strongly strongly strongly recommend Requests as the client, if only because I'm intimately involved in the development of that library.

[Lukasa]

Circling back to this with some more time to consider, I think @peterbrittain has covered most of the main points. I'll tackle some others:

• HTTP Notifications: COMET is basically a dirty hack. It would be easier and just as effective to use HTTP caching combined with regular polling to turn the requests around quickly if there's nothing to say. Requests plus a cache I wrote would give us that. (Might have to add some more features to the cache, but that's not too bad.) As an alternative, we could use HTTP streaming like Twitter does, and send small byte loads on a semi regular basis (Requests can handle this use case too.)
• Intra-Conga messages: HTTP with a JSON payload is likely to be the way to go, unless we think we're going to want to send binary data (e.g. images).

[peterbrittain]

I'm not sure caching is sufficient. If we want to support 1500 clients polling at a sensible interval (say 5s) to make feedback, that could be 300 requests a second. I presume that we don't have a huge amount of server CPU and bandwidth, so this might prove a bottleneck even with caching.

Out of the 2 options, the HTTP streaming therefore gets my vote (as long as we can make a clean client API to access it).

[Lukasa]

I can build a wrapper around some Requests code that should abstract most of the streaming problems away. It'll be a little bit of work on the server-side as well (we'll have to send the occasional empty packet to keep the connection open), but nothing too horrible.

[peterbrittain]

Looks like we have some decent options with Requests on the client-side, then. Not much being said about the server, though.

Assuming that we're going with HTTP (with streaming notifications), I instantly err towards django for the administrative function we need to provide. However, those streams will be a bit awkward. Maybe we could try a Django + Tornado (http://www.tornadoweb.org/en/stable/) mix?

[Lukasa]

I have similar concerns. Writing a streaming HTTP response in Django is not itself a big deal (Python generator). However, I think we'd need as many Django processes as there are concurrent connections to serve them: that means 1500 processes. I don't think that's going to happen.

Tornado is likely to be the better way to handle that. We can probably put Tornado in front of Django (my understanding is that Tornado has some fancy WSGIContainer magic that can just run Django and forward most of the work on to Django except for the streaming notifications work.

[peterbrittain]

So for now let's assume that implementation (of tornado + django + requests) to handle the client server interface.

Any objections?

[Lukasa]

I'm +1 on that. =)