Closed sbulen closed 3 years ago
sbulen
commented
3 years ago Actually, it speeds up if you don't trigger background processing with something like Selenium... And yes, that makes sense...
I just double-checked my pg, which was quicker, and in fact, I have 3,000,000 background tasks in the queue, to add notifications for the 3M added posts! 😂
I have sped up the mysql8 processing dramatically by changing COUNT(*) to MAX({pk})... Seems more stable, too...
sbulen
commented
3 years ago EDITING THIS....
After trying many different things, I ultimately realized that the issue was due to InnoDB configuration. Experimented around a bit & ultimately got InnoDB perform acceptably. The 3 main variables for me were: innodb_buffer_pool_size - increased innodb_buffer_pool_instances - increased innodb_flush_log_at_trx_commit = 2
Since I can now get it to perform OK, I am going to close this.
Very handy tool, but it's gotta be quicker... And it's gotta be more stable.
It fails a lot with no error messages at all. It then needs to be restarted. The good news is it happily picks up where it left off.
I was able to get a couple million rows in pg with this tool, but it took a few hours and I needed to manually restart it several times. Not too big a problem for an infrequently used tool.
OTOH, I've been at it with mysql 8.0.19 for several hours & only have ~150K rows added. It's silently stopping a lot.
Part of what is happening is that "CreatePost-Notify" background tasks are queueing up, and the test environment has too little activity to push that along. I can get past that by automating occasional queries using Selenium. So it moves.
(((Part of my problem here is specific to mysql 8 < 8.0.20. Early versions of mysql 8 had a massive issue with COUNT(*). Apparently it must do a physical count... And performance drops dramatically as the record counts get bigger. E.g., 10 seconds per count when records > 500K or so. And Populate.php issues ~ 7 of these each cycle. So it spends a LOT of time counting over & over again, doing nothing... I am of course running 8.0.19, before the fix... Not sure we need to address this, since it's fixed in 8.0.20, but noting for awareness. And whining....)))