HearthStats / hearthstats

Details Hearthstone match statistics and tracking.
hearthstats.net
Other
88 stars 44 forks source link

`get_type_popularity` + DB seeding improvements #512

Closed crawsible closed 9 years ago

crawsible commented 9 years ago
  1. Improves the efficiency of the get_type_popularity DB query by an order of magnitude. Introduces an index on match_deck to quickly filter out old match_deck records.
  2. Makes use of the Match multi_create method to seed a local db far more quickly. Creating 800K matches takes around 2 minutes now.
  3. Takes steps towards cleaning up the repo. Removes unused files, removes executable bits from files which are not executable.

-csj

crawsible commented 9 years ago

Logs of the get_type_popularity before and after:

2.1.2 :093 > UniqueDeckType.get_type_popularity(3)
   (920.8ms)  SELECT COUNT(*) AS count_all, unique_deck_types.name AS unique_deck_types_name FROM `unique_deck_types` INNER JOIN `unique_decks` ON `unique_decks`.`unique_deck_type_id` = `unique_deck_types`.`id` INNER JOIN `decks` ON `decks`.`unique_deck_id` = `unique_decks`.`id` INNER JOIN `match_decks` ON `match_decks`.`deck_id` = `decks`.`id` INNER JOIN `matches` ON `matches`.`id` = `match_decks`.`match_id` WHERE (matches.created_at >= '2015-10-26 03:13:11') AND (unique_deck_types.name IS NOT NULL) GROUP BY unique_deck_types.name HAVING count_all >= 10
 => {"Unique Deck Type 0"=>25.77665827036104, "Unique Deck Type 1"=>25.692695214105793, "Unique Deck Type 3"=>24.181360201511335, "Unique Deck Type 4"=>24.349286314021832} 
2.1.2 :094 > UniqueDeckType.get_type_popularity(3)
   (720.6ms)  SELECT COUNT(*) AS count_all, unique_deck_types.name AS unique_deck_types_name FROM `unique_deck_types` INNER JOIN `unique_decks` ON `unique_decks`.`unique_deck_type_id` = `unique_deck_types`.`id` INNER JOIN `decks` ON `decks`.`unique_deck_id` = `unique_decks`.`id` INNER JOIN `match_decks` ON `match_decks`.`deck_id` = `decks`.`id` INNER JOIN `matches` ON `matches`.`id` = `match_decks`.`match_id` WHERE (matches.created_at >= '2015-10-26 03:13:16') AND (unique_deck_types.name IS NOT NULL) GROUP BY unique_deck_types.name HAVING count_all >= 10
 => {"Unique Deck Type 0"=>25.77665827036104, "Unique Deck Type 1"=>25.692695214105793, "Unique Deck Type 3"=>24.181360201511335, "Unique Deck Type 4"=>24.349286314021832} 
2.1.2 :095 > UniqueDeckType.get_type_popularity(3)
   (714.1ms)  SELECT COUNT(*) AS count_all, unique_deck_types.name AS unique_deck_types_name FROM `unique_deck_types` INNER JOIN `unique_decks` ON `unique_decks`.`unique_deck_type_id` = `unique_deck_types`.`id` INNER JOIN `decks` ON `decks`.`unique_deck_id` = `unique_decks`.`id` INNER JOIN `match_decks` ON `match_decks`.`deck_id` = `decks`.`id` INNER JOIN `matches` ON `matches`.`id` = `match_decks`.`match_id` WHERE (matches.created_at >= '2015-10-26 03:13:19') AND (unique_deck_types.name IS NOT NULL) GROUP BY unique_deck_types.name HAVING count_all >= 10
 => {"Unique Deck Type 0"=>25.77665827036104, "Unique Deck Type 1"=>25.692695214105793, "Unique Deck Type 3"=>24.181360201511335, "Unique Deck Type 4"=>24.349286314021832} 
2.1.2 :096 > UniqueDeckType.get_type_popularity(3)
   (712.7ms)  SELECT COUNT(*) AS count_all, unique_deck_types.name AS unique_deck_types_name FROM `unique_deck_types` INNER JOIN `unique_decks` ON `unique_decks`.`unique_deck_type_id` = `unique_deck_types`.`id` INNER JOIN `decks` ON `decks`.`unique_deck_id` = `unique_decks`.`id` INNER JOIN `match_decks` ON `match_decks`.`deck_id` = `decks`.`id` INNER JOIN `matches` ON `matches`.`id` = `match_decks`.`match_id` WHERE (matches.created_at >= '2015-10-26 03:13:22') AND (unique_deck_types.name IS NOT NULL) GROUP BY unique_deck_types.name HAVING count_all >= 10
 => {"Unique Deck Type 0"=>25.77665827036104, "Unique Deck Type 1"=>25.692695214105793, "Unique Deck Type 3"=>24.181360201511335, "Unique Deck Type 4"=>24.349286314021832} 
2.1.2 :097 > 
2.1.2 :098 >   
2.1.2 :099 >   
2.1.2 :100 >   
2.1.2 :101 >   
2.1.2 :102 >   
2.1.2 :103 >   
2.1.2 :104 >   
2.1.2 :105 >   
2.1.2 :106 >   
2.1.2 :107 >   
2.1.2 :108 >   
2.1.2 :109 >   
2.1.2 :110 >   
2.1.2 :111 >   
2.1.2 :112 >   
2.1.2 :113 >   
2.1.2 :114 >   
2.1.2 :115 >   reload!
Reloading...
 => true 
2.1.2 :116 > UniqueDeckType.get_type_popularity(3)
   (30.5ms)  SELECT COUNT(*) AS count_all, unique_deck_types.name AS unique_deck_types_name FROM `unique_deck_types` INNER JOIN `unique_decks` ON `unique_decks`.`unique_deck_type_id` = `unique_deck_types`.`id` INNER JOIN `decks` ON `decks`.`unique_deck_id` = `unique_decks`.`id` LEFT JOIN
 (SELECT `match_decks`.* FROM `match_decks` WHERE (created_at >= '2015-10-26 03:13:55')) as `match_decks`
ON
 `match_decks`.`deck_id` = `decks`.`id` WHERE (unique_deck_types.name IS NOT NULL) GROUP BY unique_deck_types.name HAVING count_all >= 10
 => {"Unique Deck Type 0"=>25.77665827036104, "Unique Deck Type 1"=>25.692695214105793, "Unique Deck Type 3"=>24.181360201511335, "Unique Deck Type 4"=>24.349286314021832} 
2.1.2 :117 > UniqueDeckType.get_type_popularity(3)
   (30.5ms)  SELECT COUNT(*) AS count_all, unique_deck_types.name AS unique_deck_types_name FROM `unique_deck_types` INNER JOIN `unique_decks` ON `unique_decks`.`unique_deck_type_id` = `unique_deck_types`.`id` INNER JOIN `decks` ON `decks`.`unique_deck_id` = `unique_decks`.`id` LEFT JOIN
 (SELECT `match_decks`.* FROM `match_decks` WHERE (created_at >= '2015-10-26 03:14:22')) as `match_decks`
ON
 `match_decks`.`deck_id` = `decks`.`id` WHERE (unique_deck_types.name IS NOT NULL) GROUP BY unique_deck_types.name HAVING count_all >= 10
 => {"Unique Deck Type 0"=>25.77665827036104, "Unique Deck Type 1"=>25.692695214105793, "Unique Deck Type 3"=>24.181360201511335, "Unique Deck Type 4"=>24.349286314021832} 
2.1.2 :118 > UniqueDeckType.get_type_popularity(3)
   (30.0ms)  SELECT COUNT(*) AS count_all, unique_deck_types.name AS unique_deck_types_name FROM `unique_deck_types` INNER JOIN `unique_decks` ON `unique_decks`.`unique_deck_type_id` = `unique_deck_types`.`id` INNER JOIN `decks` ON `decks`.`unique_deck_id` = `unique_decks`.`id` LEFT JOIN
 (SELECT `match_decks`.* FROM `match_decks` WHERE (created_at >= '2015-10-26 03:14:24')) as `match_decks`
ON
 `match_decks`.`deck_id` = `decks`.`id` WHERE (unique_deck_types.name IS NOT NULL) GROUP BY unique_deck_types.name HAVING count_all >= 10
 => {"Unique Deck Type 0"=>25.77665827036104, "Unique Deck Type 1"=>25.692695214105793, "Unique Deck Type 3"=>24.181360201511335, "Unique Deck Type 4"=>24.349286314021832} 
2.1.2 :119 > 
trigun0x2 commented 9 years ago

2Sexy4Me