This repo contains the source code used to run SpartanFinder from May 2020 to about August 2021. The site originally started out as a way to just get a sense of who is playing Halo: Combat Evolved and which version of the game I can find people playing it on. Since the Halo franchise has numerous titles, on numerous platforms, with numerous ways to play I figured it would be useful for those trying to find a game on the less popular titles. I've also had a curiousty about the historical popularity of every title, and think it would be fun one day to have a all time population chart for every halo title on the same plot.
The site definitely got into a good place and I thought provided fairly accurate numbers. Most games I was able to go straight to the source, except for MCC and H5. H5 had its own api, which is great, and I was able to build a decent crawler that got fairly accurate population numbers. For MCC the crawler I built was limited since there was no API to use and I had to use the XSAPI xbox live api to get player presence and rich presence information. Along the way a site called gamstat.com popped up and provided very well calibrated, but not pecise, population numbers for all xbox live titles. Combining that number with my crawlers number provided a good balance between calibration and precision for legacy 360 titles as well as MCC on console (MCC on PC leveraged the steam api to get direct numbers. Yes, I didn't have a great way of factoring in windows store users). Things were going good.
People even began stumbling upon the site which was great. It seemed somewhat controversial on reddit as fans of specific titles seemed to use it as fodder to support why their game did or didn't have a healthy player base. Yes this site is not offical, and now everyone knows how it largely worked so take from it what you will.
Eventually gamstat.com shut down, because the guy who ran it said there was not enough interest in it for the amount of work he was putting in it. While this was disappointed, and impacted the quality of my MCC numbers, I understood because I ended up feeling the same way. While my site also did not cost much to run, it did require a fair amount of upkeep. Scripts would stop running, or freeze, or apis/data sources would change and all of those things required updating the code to get things back up and running. I tried monetizing links to each halo title on the main page with amazon's referral system, which would never actually allow me to hold an account past the trial phase even though it was generating roughly ~$50-$100 of referral income a month. That probably would have been enough for me to keep the site going, but amazon kept rejecting my referral account requests once the trial period was over. Ebay's referral system didn't require a probationary period, but paid significantly less. I think I would earn about $10/month through them as people just don't buy stuff from eBay as much as they do from Amazon.
So eventually it got to a point where it didn't have a super strong userbase (I think it peaked at 200 quality unique views a day), it wasn't earning me good beer money, and no one else was interested in getting involved with running the site. So similar to gamstat, I just had to stop operating it. I do want to help others out if they want to pick up any of this, so I have created this repo with the source code for the scrapers and the population data this site collected over it's one year run.
In the scrapers folder, you will find the source code to each scraper I built out to support this site. Feel free to use them, but I will concede the codebase isn't amazing, and some of them probably don't work anymore as the apis/endpoints they used have changed. I did some commentating and can do more upon request.
In the data folder, you will find the historical logs for population counts for each scraper. Additionally, for a while I was automatically saving the homepage to the wayback machine. So if you want the "official" numbers generated from this project, you can look at them via the wayback machine
So that's pretty much it. If you have any questions or need any clarifications just open up an issue and I'll respond. If you are involved in an effort that is similar to this, please let me know as well! My discord name is CYRiX#5864.
xboxstat.ru - A russian site that does a great job of tracking all xbox live game statistics