Some of the slowness on the big build's MySQL index creation can probably be ameliorated by fixing a few settings in my.cnf for the server. Here are some tips from the web to think about next time we try a 1TB+ build.
myisam_sort_buffer_size defaults to 2M on many installations. We have it currently set at 512M, which seemed insanely high to me at the time. (For multiple queries, this might get complicated). But as a session variable during the build, it may make sense to set it yet far higher still--somewhere between 40 and 90% of total memory on the machine. (Depending on whether we can guarantee there won't be another build simultaneously.) So on the Hathi server, that would be something like 50,000M. It could substantially reduce disk read times.
myisam_repair_threads should allow several threads to be used. It's rare a table has more than 2 indexes, but most of them do have two. Defaulting to 4 might make sense on large instances.
Of course, we also need to ensure that multiple threads are available at all.
Some of the slowness on the big build's MySQL index creation can probably be ameliorated by fixing a few settings in my.cnf for the server. Here are some tips from the web to think about next time we try a 1TB+ build.
myisam_sort_buffer_sizedefaults to 2M on many installations. We have it currently set at 512M, which seemed insanely high to me at the time. (For multiple queries, this might get complicated). But as a session variable during the build, it may make sense to set it yet far higher still--somewhere between 40 and 90% of total memory on the machine. (Depending on whether we can guarantee there won't be another build simultaneously.) So on the Hathi server, that would be something like 50,000M. It could substantially reduce disk read times.myisam_repair_threadsshould allow several threads to be used. It's rare a table has more than 2 indexes, but most of them do have two. Defaulting to 4 might make sense on large instances.