Add multicore CPU utilization support to rtmp.py

[adding more responses]
--
Hey,

The rtmplite software is not multi-threaded.

If you are interested in only the SIP-RTMP gateway feature (where
multi-processing does not break things), you can try out the
siprtmp_gevent.py with the "-f" option which allows you to use
multi-processing and potentially use all the cores.

If you need the RTMP server feature of rtmplite, then it won't work in
multiprocessing mode currently. The reason is that for RTMP server the
different connections from client need to share same context for
streaming/distribution to other clients. Whereas for SIP-RTMP gateway,
every connection from client is separate from the other connections
since a media from one client does not go back to another client but
to a SIP side.

Please add an issue in the project issue page for adding
multiprocessing support in rtmplite/rtmp.py. Hopefully we can resolve
it using the shared memory primitives of Python.

Hope this helps
--
Kundan,
thank you for you reply. I need RTMP server feature for video
broadcasting, so ss you suggested I added new issue on Google Code.

I was thinking about temporary workaround this issuue
- N instances of rtmp.py on one machine (where N=nmbr of CPU cores)
- restreaming streams from one instance to all other using rtmpclient
- load balancing logic on Flash Client side
It'a hypotetic workaround reasonable only when subscibers/broadcasters
ratio is high.
What do you think about such idea?
Maybe there is some better way to utilize all cores of CPU..

Best Regards, Maciek
--

Hi Maciek,

I agree with you.

1) Unlike web servers, RTMP server is stateful so standard server side
TCP load balancer is not very useful (unless the load balancer can
understand RTMP and distribute based on say RTMP URL path).

2) Client side failover and load sharing is much more robust and scalable.

3) There are generally two types of server load pattern: (a) large
number of 2-3 party calls, (b) very large number of participants.

Suppose we run N instances of rtmp.py on one machine, say on different
ports s1=rtmp://server:1935/path, s2=rtmp://server:1936/path, ...

For (a) where subscriber/broadcaster ratio is 1 or 2, it is better to
do out-of-band server selection. For example, let the participating
clients pick the same server, and connect to the same server. With
large number of calls distributed across multiple servers, generally
the load sharing works well. Either a server side API could be used or
a simple hash function could be used to map an RTMP URL to the same
server. For example, the client connects to port=1935+hash(path)%N and
where path is different for different calls. Another approach is to
let the primary server redirect incoming new connection to secondary
if the primary is overloaded, but I personally don't prefer this idea.

For(b) where subscriber/broadcaster ratio is very high, an
architecture similar to FMS edge-origin makes sense. This is same as
restreaming from one instance to others as you mentioned. To implement
it in rtmplite, we need to design some mechanism to automate this.
Generally, the publish task takes more CPU (~34MHz) than the play task
(~12MHz). An IPC mechanism such as shared memory or queue could be
used to restream from one to another instead of rtmpclient, since I
believe using rtmpclient will further reduce the performance
drastically.

While (a) is client side solution; If you are interested in
contributing to implementing (b), lets explore it in more detail.

References:
[1] http://docs.python.org/library/multiprocessing.html
[2] http://p2p-sip.blogspot.com/2011/04/performance-of-siprtmp-multitask-vs.html
Original comment by kundan10 on 25 Feb 2012 at 6:40
hlooalklampc / rtmplite

Add multicore CPU utilization support to rtmp.py #73
