search

lunixbochs / uberserver

matchmaking/chat lobby server for the spring rts project
MIT License
8 stars 4 forks source link

server dies under IO load #4

Closed abma closed 12 years ago

abma commented 13 years ago

when IO on physical server reached a certain point of slowness and accounts.txt is written, entire server is affected. this shouldn't happen, as accounts.txt is written in a thread and releases the interpreter lock when waiting on IO.

something else must be conflicting to cause a deadlock.

edit by lunixbochs: was "server seems to die when >~350 users" with more than 350 users online, the server seems to die... looks like memory usage kills it?

lunixbochs commented 13 years ago

please be more specific. if you're talking about production server, that has nothing to do with users - it's a file IO issue

the server has been tested with thousands of active connections.

abma commented 13 years ago

it looks like this is the magic number, where the server is overloaded.

or maybe some algorithm performs very bad starting with greater number than this.

(for example: every status-change is reported to every user)

i'm reporting because apache does still can ship webpages but lobbyserver doesn't respond.

lunixbochs commented 13 years ago

If you're talking about production server, this has nothing to do with user count. I've 100% duplicated it as an IO bug triggered when saving accounts.txt.

abma commented 13 years ago

maybe give uberserver a higher priority + ulimit some processes?

lunixbochs commented 13 years ago

I just need to shove the accounts.txt write somewhere it won't deadlock the server. Something else (maybe some socket operations?) are causing a deadlock that doesn't normally happen when writing to a file in a thread under heavy IO.

I started up a test script during heavy load and I was able to write extensively to a file without causing any lag in main thread. I'm thinking I might just start a subprocess (or fork?) to save accounts.txt as a quick fix, then get the server moved as a more permanent one.

I'll also try to reproduce in a sandbox to see if I can figure out what exactly is causing the conflict.

abma commented 13 years ago

i talk about springrts.com main lobby server...

how big is accounts.txt? maybe you should better switch to a database and try to keep fewer stuff in memory (if it makes sense)?

lunixbochs commented 13 years ago

The database layer is already designed and working, I've planned to switch to using it when I move to my dedicated server.

abma commented 13 years ago

if you need testers... let me know! :)

abma commented 12 years ago

duplicate of #8 + #6