The C10k Plan

[nemasu]

My current plan for handling 10k+ connections, will edit as it's improved. Comments are welcome :)

Idea: Don't block on anything until it's ready. When waiting for I/O, accept more connections. +Asynchronous I/O. +Non-blocking sockets. +Multi-threaded. +Self contained threads. +Single thread can handle multiple connections, more threads = more connections +Only one thread design needed +No synchronization of threads needed +Self load balancing threads. ie. Threads with more free cycles will call accept more often. -/+ One connection will use about 10 KB of memory. -More complicated (but more awesome too). -Will likely be a tad slower on many small requests.

struct Data = socket fd, file fd, send_offset, length, 8KB buf(recv, path, url) = 8 + 8 + 8 + 8 + 8192 + 104 + 2000 = 10328 bytes

Storing pointer to struct in epoll user data, no need for extra data structure.

worker-thread pseudocode

top:
accept
    epol_ctl( socket fd, read )

epoll_wait ( socket recv, send fds, and file read fds )
    socket recv ( if file fd == 0 ):
        allocate data mem & add socket fd to data
        recv
        parse for file & open, store fd in data & create header.
        epoll_ctl( file fd, read )
    socket send:
        send
        if done, free data, epoll_ctl delete else inc offset
    file read ( if file fd != 0 ):
        get info from Data struct: offset, length
        read()
        update Data, if done remove&close file fd, add socket send fd
goto top

[smougel]

instead of opening a file for each request... maybe you should try to put it into memory ?

But : if the file content change... You need to refresh your data stored in memory...

[nemasu]

Hmm, that would be interesting for smaller files actually. Basically caching them in memory. I don't think it would be practical for large files though.

Maybe with some sort of file access threshold being passed it would store it in memory ( assuming it's not huge ). Interesting Idea.

Edit: Seems like the case of adding something to the cache will require synchronization ... my mutex lives!! xD

[arwyn]

Caching files is likely to be counter productive.

The OS filesystem cache has better domain knowledge than you have available. Whatever nanoseconds gained in caching in ram would be wiped out in consistency checks and flushing operations. Not only that, but by adding a second layer of cache you are disrupting the FS usage statistics, decreasing its effectiveness.

[nemasu]

@arwyn Ah yeah, that's true. Well then, less work for me. :)

[smougel]

@arwyn the only way to verify these assertions is to perform some benchmarks... Disk Access are VERY VERY costly... There is a factor 1000 in terms of latency and throughput between RAM and HDD.

It will be very interesting to compare serving static content from RAM and compare the results with content serving from the filesystem. Maybe it's possible to develop a quick and dirty test by placing static data in the code (3 gifs and 3 html files).

But you're true... It's difficult to have a good cache strategy... There is cache in so many levels (CPU, Disk controller, OS)

An interesting reading about kernel cache strategies: http://www.thomas-krenn.com/en/wiki/Linux_Page_Cache_Basics

@nemasu Have you identified where are the bottlenecks in the current code ?

[nemasu]

@smougel Thanks for the input and info. Yes, it's currently limited by I/O and thread count. Because it's a 1 to 1 relationship right now, 1 connection = 1 thread. The current plan looks really promising though, I'm looking forward to implementing it. :) I ditched select for poll, because select is ancient and terrible.

Side note: I just did a benchmark with version 0.1, 10k successful transfers in 15 seconds using 10 concurrent connections with no failures. Also it was in a VM and I left thread count at 10 ( this ones tricky to tweak because it's on a VM at the moment ). Not too bad for a non async implementation. Even as it stands it might be useful with it being only a 5.5 KB library-less binary.

[nemasu]

@dariolah Thanks for that link, it was a very interesting read. I think the possibility is high I will try that at some point. That being said, c10k as a stepping stone needs to be accomplished anyways, but after that I will be thinking about implementing some of those designs.