测试数据库速度

[gqqnbig]

multiprocessing可以加快速度。SQLite只能对整个数据库加锁，所以并行只存在于classification的地方，虽然使用了4个进程，但速度是原来的不到2倍。

SQL Server对单表加锁，速度是原来的3倍。

[gqqnbig]

https://www.red-gate.com/simple-talk/sql/performance/comparing-multiple-rows-insert-vs-single-row-insert-with-three-data-load-methods/

[gqqnbig]

在Google Cloud上测试，不运行其他应用，避免影响。

SQLite顺序 Used time: 983.9608352184296

SQLite并行 Used time: 983.6607518196106

MySQL顺序 [Thread 1] User 132955 is done. Progress is 96%. Used time is 1700s, Remaining time is 70s. [Thread 1] User 134340 is done. Progress is 97%. Used time is 1719s, Remaining time is 53s. [Thread 1] User 135725 is done. Progress is 98%. Used time is 1737s, Remaining time is 35s. [Thread 1] User 137110 is done. Progress is 99%. Used time is 1753s, Remaining time is 17s. Fixed 47 empty prediction in table ValidationRatings. Fixed 698 empty prediction in table TestRatings. Used time: 1774.4654133319855

MySQL并行（userId有无索引速度差不多） [Thread 2] User 68556 is done. Progress is 98%. Used time is 621s, Remaining time is 12s. [Thread 1] User 34279 is done. Progress is 99%. Used time is 621s, Remaining time is 6s. [Thread 4] User 138148 is done. Progress is 99%. Used time is 624s, Remaining time is 6s. [Thread 3] User 103525 is done. Progress is 99%. Used time is 624s, Remaining time is 6s. [Thread 2] User 68902 is done. Progress is 99%. Used time is 627s, Remaining time is 6s. Fixed 47 empty prediction in table ValidationRatings. Fixed 698 empty prediction in table TestRatings. Used time: 635.8703634738922

SQL Server on Docker顺序 User 132954 is done. Progress is 96%. Used time is 2342s, Remaining time is 97s User 134339 is done. Progress is 97%. Used time is 2369s, Remaining time is 73s User 135724 is done. Progress is 98%. Used time is 2393s, Remaining time is 48s User 137109 is done. Progress is 99%. Used time is 2419s, Remaining time is 24s Fixed 47 empty prediction in table ValidationRatings. Fixed 698 empty prediction in table TestRatings. Used time: 2445.146753549576

SQL Server on Docker并行 [Thread 3] User 99138 is done. Progress is 99%. Used time is 916s, Remaining time is 9s. [Thread 2] User 99785 is done. Progress is 98%. Used time is 920s, Remaining time is 18s. [Thread 1] User 133523 is done. Progress is 98%. Used time is 922s, Remaining time is 18s. [Thread 2] User 33419 is done. Progress is 99%. Used time is 928s, Remaining time is 9s. [Thread 1] User 119589 is done. Progress is 99%. Used time is 930s, Remaining time is 9s. Fixed 47 empty prediction in table ValidationRatings. Fixed 698 empty prediction in table TestRatings. Used time: 938.446896314621 (2408168, 3999236, 0.60215701198929)

[gqqnbig]

https://domino.research.ibm.com/library/cyberdig.nsf/papers/0929052195DD819C85257D2300681E7B/$File/rc25482.pdf

https://www.v2ex.com/t/394313

Docker和本地应用的性能只差4%。

[gqqnbig]

有了并行后，主要时间花在模型训练和预测上，数据库时间不是重点。