search

rrcampbell-exe / real-fake-birds

A guessing game featuring real fake birds.
https://www.realfakebirds.app
1 stars 0 forks source link

Test Algorithm for Balance #42

Closed rrcampbell-exe closed 11 months ago

rrcampbell-exe commented 12 months ago

Create function to run birdCall 1000 times and see how balanced the current algorithm is in terms of providing real bird names and fake bird names. The current observation is that more fake names are being provided than real, and by a noticeable margin.

To test this hypothesis and make updates as necessary, create a reusable file (that sits in the .gitignore) that can be run locally on a Node server. Track the number of times isReal returns true over the course of x number of runs (1000 for initial testing purposes).

rrcampbell-exe commented 12 months ago

A new file has been created called algo-eval.js. It sits on the server side of the app and can be run from the server folder by running node utils/algo-val.js. It will tell you how many times out of timesToCallAPI that isReal is true.

Calling the API 1,200 times returned isReal a total of 527 times, or 43.9% of the time. This seems to uphold the hypothesis that triggered this issue's creation in the first place.

In order to restore balance to the algorithm, the value for fakeBird in the birdCall function should be adjusted. Currently, we are asking to return a fakeBird 47% of the time, but it is returning a fakeBird 56.1% of the time.

fakeBird will likely have to be lowered to approximately 41% in order to return fake birds and real birds in equal measure.

rrcampbell-exe commented 11 months ago

Adding the back parameter to the URL for birdCall and giving it a value of 30 may have restored the necessary balance. Rather than only pull in recent bird sightings from the previous 14 days, it now does 30 days, which makes it more likely that more countries will have more records available at a given point in time.

The most recent run of the algorithm test returned real birds 501 out of 1000 times, which is about as balanced as possible.

rrcampbell-exe commented 11 months ago

A second run of the algorithm test returned isReal 482 out of 1000 times. Not exactly balanced, but still an improvement over the approximately 44% that we saw prior to the addition of the back parameter. The working branch can be merged and this issue can subsequently be closed for the time being.