search

infolab-csail / wikipedia-mirror

Makefiles that will download and setup a local wikipedia instance.
1 stars 2 forks source link

Too slow mysql dump load with no cpu or memory usage #4

Closed fakedrake closed 10 years ago

fakedrake commented 10 years ago

Turns out disk io is very slow. Looking at htop (or top) the memory is full of cache and iotop shows that the cache is probably being flushed as fast as possible by the [jbd/dm-8-0] process which takes up 99% of io.

Thus I claim that we can't really go much faster. I run make with -j12 to leave no room at all but if you kill that chances are it wont die gracefully. The server will die before some clients and then I have kno clue where I left off.

Anyway if you do actually kill a half digested dump you want to continue with --force(default in makefile) or mysql will complain about duplicates and refuse to continue.

fakedrake commented 10 years ago

Also note this article about flushing caches. Haven't tried it but should be good.