Now that we've got the size of the db to <.5GB prior to compression, we can also try to build it in-memory and dump it to disk only when we're done. To do:
check whether this is really faster;
check whether it makes sense to hold the redirect table in the db as well, because that's currently what's taking memory in the parser (I think?);
check whether we get a "vacuum" for free (45f0414628ec82257fa406dc9d9f167c572c6535).
larsmans
commented
10 years ago
Now that we've got the size of the db to <.5GB prior to compression, we can also try to build it in-memory and dump it to disk only when we're done. To do: