bayesian_testing
is a small package for a quick evaluation of A/B (or A/B/C/...) tests using
Bayesian approach.
Implemented tests:
[0, 1, 0, ...]
)[1, 0, 3, ...]
)Implemented evaluation metrics:
Posterior Mean
Probability of Being Best
the best
is equivalent to the greatest
(from a data/metric point of view),
however it is possible to change this by using min_is_best=True
in the evaluation method
(this can be useful if we try to find the variant while minimizing the tested measure).Expected Loss
Probability of Being Best
and Expected Loss
are calculated using simulations from posterior
distributions (considering given data).
bayesian_testing
can be installed using pip:
pip install bayesian_testing
Alternatively, you can clone the repository and use poetry
manually:
cd bayesian_testing
pip install poetry
poetry install
poetry shell
The primary features are classes:
BinaryDataTest
NormalDataTest
DeltaLognormalDataTest
DeltaNormalDataTest
DiscreteDataTest
PoissonDataTest
ExponentialDataTest
All test classes support two methods to insert the data:
add_variant_data
- Adding raw data for a variant as a list of observations (or numpy 1-D array).add_variant_data_agg
- Adding aggregated variant data (this can be practical for a large data,
as the aggregation can be done already on a database level).Both methods for adding data allow specification of prior distributions (see details in respective docstrings). Default prior setup should be sufficient for most of the cases (e.g. cases with unknown priors or large amounts of data).
To get the results of the test, simply call the method evaluate
.
Probability of being best and expected loss are approximated using simulations, hence the evaluate
method can return slightly different values for different runs. To stabilize it, you can set the
sim_count
parameter of the evaluate
to a higher value (default value is 20K), or even use the
seed
parameter to fix it completely.
Class for a Bayesian A/B test for the binary-like data (e.g. conversions, successes, etc.).
Example:
import numpy as np
from bayesian_testing.experiments import BinaryDataTest
# generating some random data
rng = np.random.default_rng(52)
# random 1x1500 array of 0/1 data with 5.2% probability for 1:
data_a = rng.binomial(n=1, p=0.052, size=1500)
# random 1x1200 array of 0/1 data with 6.7% probability for 1:
data_b = rng.binomial(n=1, p=0.067, size=1200)
# initialize a test:
test = BinaryDataTest()
# add variant using raw data (arrays of zeros and ones):
test.add_variant_data("A", data_a)
test.add_variant_data("B", data_b)
# priors can be specified like this (default for this test is a=b=1/2):
# test.add_variant_data("B", data_b, a_prior=1, b_prior=20)
# add variant using aggregated data (same as raw data with 950 zeros and 50 ones):
test.add_variant_data_agg("C", totals=1000, positives=50)
# evaluate test:
results = test.evaluate()
results # print(pd.DataFrame(results).set_index('variant').T.to_markdown(tablefmt="grid"))
+-----------------+--------------+--------------+--------------+
| variant | A | B | C |
+=================+==============+==============+==============+
| totals | 1500 | 1200 | 1000 |
+-----------------+--------------+--------------+--------------+
| positives | 80 | 80 | 50 |
+-----------------+--------------+--------------+--------------+
| positive_rate | 0.05333 | 0.06667 | 0.05 |
+-----------------+--------------+--------------+--------------+
| posterior_mean | 0.05363 | 0.06703 | 0.05045 |
+-----------------+--------------+--------------+--------------+
| prob_being_best | 0.06575 | 0.894 | 0.04025 |
+-----------------+--------------+--------------+--------------+
| expected_loss | 0.0138918 | 0.0004528 | 0.0171382 |
+-----------------+--------------+--------------+--------------+
Class for a Bayesian A/B test for the normal data.
Example:
import numpy as np
from bayesian_testing.experiments import NormalDataTest
# generating some random data
rng = np.random.default_rng(21)
data_a = rng.normal(7.2, 2, 1000)
data_b = rng.normal(7.1, 2, 800)
data_c = rng.normal(7.0, 4, 500)
# initialize a test:
test = NormalDataTest()
# add variant using raw data:
test.add_variant_data("A", data_a)
test.add_variant_data("B", data_b)
# test.add_variant_data("C", data_c)
# add variant using aggregated data:
test.add_variant_data_agg("C", len(data_c), sum(data_c), sum(np.square(data_c)))
# evaluate test:
results = test.evaluate(sim_count=20000, seed=52, min_is_best=False)
results # print(pd.DataFrame(results).set_index('variant').T.to_markdown(tablefmt="grid"))
+-----------------+-------------+-------------+--------------+
| variant | A | B | C |
+=================+=============+=============+==============+
| totals | 1000 | 800 | 500 |
+-----------------+-------------+-------------+--------------+
| sum_values | 7294.68 | 5685.86 | 3736.92 |
+-----------------+-------------+-------------+--------------+
| avg_values | 7.29468 | 7.10733 | 7.47383 |
+-----------------+-------------+-------------+--------------+
| posterior_mean | 7.29462 | 7.10725 | 7.4737 |
+-----------------+-------------+-------------+--------------+
| prob_being_best | 0.1707 | 0.00125 | 0.82805 |
+-----------------+-------------+-------------+--------------+
| expected_loss | 0.196874 | 0.385112 | 0.0169998 |
+-----------------+-------------+-------------+--------------+
Class for a Bayesian A/B test for the delta-lognormal data (log-normal with zeros). Delta-lognormal data is typical case of revenue per session data where many sessions have 0 revenue but non-zero values are positive values with possible log-normal distribution. To handle this data, the calculation is combining binary Bayes model for zero vs non-zero "conversions" and log-normal model for non-zero values.
Example:
import numpy as np
from bayesian_testing.experiments import DeltaLognormalDataTest
test = DeltaLognormalDataTest()
data_a = [7.1, 0.3, 5.9, 0, 1.3, 0.3, 0, 1.2, 0, 3.6, 0, 1.5, 2.2, 0, 4.9, 0, 0, 1.1, 0, 0, 7.1, 0, 6.9, 0]
data_b = [4.0, 0, 3.3, 19.3, 18.5, 0, 0, 0, 12.9, 0, 0, 0, 10.2, 0, 0, 23.1, 0, 3.7, 0, 0, 11.3, 10.0, 0, 18.3, 12.1]
# adding variant using raw data:
test.add_variant_data("A", data_a)
# test.add_variant_data("B", data_b)
# alternatively, variant can be also added using aggregated data
# (looks more complicated, but it can be quite handy for a large data):
test.add_variant_data_agg(
name="B",
totals=len(data_b),
positives=sum(x > 0 for x in data_b),
sum_values=sum(data_b),
sum_logs=sum([np.log(x) for x in data_b if x > 0]),
sum_logs_2=sum([np.square(np.log(x)) for x in data_b if x > 0])
)
# evaluate test:
results = test.evaluate(seed=21)
results # print(pd.DataFrame(results).set_index('variant').T.to_markdown(tablefmt="grid"))
+---------------------+----------+------------+
| variant | A | B |
+=====================+==========+============+
| totals | 24 | 25 |
+---------------------+----------+------------+
| positives | 13 | 12 |
+---------------------+----------+------------+
| sum_values | 43.4 | 146.7 |
+---------------------+----------+------------+
| avg_values | 1.80833 | 5.868 |
+---------------------+----------+------------+
| avg_positive_values | 3.33846 | 12.225 |
+---------------------+----------+------------+
| posterior_mean | 2.09766 | 6.19017 |
+---------------------+----------+------------+
| prob_being_best | 0.04815 | 0.95185 |
+---------------------+----------+------------+
| expected_loss | 4.09411 | 0.158863 |
+---------------------+----------+------------+
Note: Alternatively, DeltaNormalDataTest
can be used for a case when conversions are not
necessarily positive values.
Class for a Bayesian A/B test for the discrete data with finite number of numerical categories (states), representing some value. This test can be used for instance for dice rolls data (when looking for the "best" of multiple dice) or rating data (e.g. 1-5 stars or 1-10 scale).
Example:
from bayesian_testing.experiments import DiscreteDataTest
# dice rolls data for 3 dice - A, B, C
data_a = [2, 5, 1, 4, 6, 2, 2, 6, 3, 2, 6, 3, 4, 6, 3, 1, 6, 3, 5, 6]
data_b = [1, 2, 2, 2, 2, 3, 2, 3, 4, 2]
data_c = [1, 3, 6, 5, 4]
# initialize a test with all possible states (i.e. numerical categories):
test = DiscreteDataTest(states=[1, 2, 3, 4, 5, 6])
# add variant using raw data:
test.add_variant_data("A", data_a)
test.add_variant_data("B", data_b)
test.add_variant_data("C", data_c)
# add variant using aggregated data:
# test.add_variant_data_agg("C", [1, 0, 1, 1, 1, 1]) # equivalent to rolls in data_c
# evaluate test:
results = test.evaluate(sim_count=20000, seed=52, min_is_best=False)
results # print(pd.DataFrame(results).to_markdown(tablefmt="grid", index=False))
+---------+--------------------------------------+---------------+----------------+-----------------+---------------+
| variant | concentration | average_value | posterior_mean | prob_being_best | expected_loss |
+=========+======================================+===============+================+=================+===============+
| A | {1: 2, 2: 4, 3: 4, 4: 2, 5: 2, 6: 6} | 3.8 | 3.73077 | 0.54685 | 0.199953 |
+---------+--------------------------------------+---------------+----------------+-----------------+---------------+
| B | {1: 1, 2: 6, 3: 2, 4: 1, 5: 0, 6: 0} | 2.3 | 2.75 | 0.008 | 1.18268 |
+---------+--------------------------------------+---------------+----------------+-----------------+---------------+
| C | {1: 1, 2: 0, 3: 1, 4: 1, 5: 1, 6: 1} | 3.8 | 3.63636 | 0.44515 | 0.287025 |
+---------+--------------------------------------+---------------+----------------+-----------------+---------------+
Class for a Bayesian A/B test for the poisson data.
Example:
from bayesian_testing.experiments import PoissonDataTest
# goals received - so less is better (duh...)
psg_goals_against = [0, 2, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 3, 1, 0]
city_goals_against = [0, 0, 3, 2, 0, 1, 0, 3, 0, 1, 1, 0, 1, 2]
bayern_goals_against = [1, 0, 0, 1, 1, 2, 1, 0, 2, 0, 0, 2, 2, 1, 0]
# initialize a test:
test = PoissonDataTest()
# add variant using raw data:
test.add_variant_data('psg', psg_goals_against)
# example with specific priors
# ("b_prior" as an effective sample size, and "a_prior/b_prior" as a prior mean):
test.add_variant_data('city', city_goals_against, a_prior=3, b_prior=1)
# test.add_variant_data('bayern', bayern_goals_against)
# add variant using aggregated data:
test.add_variant_data_agg("bayern", len(bayern_goals_against), sum(bayern_goals_against))
# evaluate test (since fewer goals is better, we explicitly set the min_is_best to True)
results = test.evaluate(sim_count=20000, seed=52, min_is_best=True)
results # print(pd.DataFrame(results).set_index('variant').T.to_markdown(tablefmt="grid"))
+------------------+------------+-----------+-----------+
| variant | psg | city | bayern |
+==================+============+===========+===========+
| totals | 15 | 14 | 15 |
+------------------+------------+-----------+-----------+
| sum_values | 9 | 14 | 13 |
+------------------+------------+-----------+-----------+
| observed_average | 0.6 | 1 | 0.86667 |
+------------------+------------+-----------+-----------+
| posterior_mean | 0.60265 | 1.13333 | 0.86755 |
+------------------+------------+-----------+-----------+
| prob_being_best | 0.78175 | 0.0344 | 0.18385 |
+------------------+------------+-----------+-----------+
| expected_loss | 0.0369998 | 0.562055 | 0.300335 |
+------------------+------------+-----------+-----------+
_note: Since we set min_is_best=True
(because received goals are "bad"), probability and loss are
in a favor of variants with lower posterior means._
Class for a Bayesian A/B test for the exponential data.
Example:
import numpy as np
from bayesian_testing.experiments import ExponentialDataTest
# waiting times for 3 different variants, each with many observations,
# generated using exponential distributions with defined scales (expected values)
waiting_times_a = np.random.exponential(scale=10, size=200)
waiting_times_b = np.random.exponential(scale=11, size=210)
waiting_times_c = np.random.exponential(scale=11, size=220)
# initialize a test:
test = ExponentialDataTest()
# adding variants using the observation data:
test.add_variant_data('A', waiting_times_a)
test.add_variant_data('B', waiting_times_b)
test.add_variant_data('C', waiting_times_c)
# alternatively, add variants using aggregated data:
# test.add_variant_data_agg('A', len(waiting_times_a), sum(waiting_times_a))
# evaluate test (since a lower waiting time is better, we explicitly set the min_is_best to True)
results = test.evaluate(sim_count=20000, min_is_best=True)
results # print(pd.DataFrame(results).set_index('variant').T.to_markdown(tablefmt="grid"))
+------------------+-------------+------------+-------------+
| variant | A | B | C |
+==================+=============+============+=============+
| totals | 200 | 210 | 220 |
+------------------+-------------+------------+-------------+
| sum_values | 1937.3 | 2451.68 | 2266.11 |
+------------------+-------------+------------+-------------+
| observed_average | 9.68649 | 11.6747 | 10.3005 |
+------------------+-------------+------------+-------------+
| posterior_mean | 9.68215 | 11.6696 | 10.2963 |
+------------------+-------------+------------+-------------+
| prob_being_best | 0.72605 | 0.0122 | 0.26175 |
+------------------+-------------+------------+-------------+
| expected_loss | 0.146675 | 2.13691 | 0.755893 |
+------------------+-------------+------------+-------------+
To set up a development environment, use Poetry and pre-commit:
pip install poetry
poetry install
poetry run pre-commit install
Additional metrics:
Potential Value Remaining
Credible Interval
bayesian_testing
package itself depends only on numpy package.