Why Does The UI Show a 60/40 Control/Test Split

[bpromo]

@cvillecsteele @tomsouthall @smnirven If I remember our earlier 60/40 concern was Visits (2303) versus Qualified (1928) in the screenshot below:

Lets look at visit count since the offer was launched:

SELECT COUNT(*) FROM events WHERE type = 'session-start' AND site_id = '995ead32-3c3b-4e4d-aeb3-5bb250fa410c AND created_at > '2015-05-13 00:01:00';

4678

Now lets look at offered count since the offer was launched:

SELECT COUNT(*) FROM offer_qualifications WHERE site_id = '995ead32-3c3b-4e4d-aeb3-5bb250fa410c' AND created_at > '2015-05-13 00:01:00';

3854

So now lets look at other metrics. First bucket-assigned for that time period:

SELECT COUNT(*) FROM events WHERE type = 'bucket-assigned' AND site_id = '995ead32-3c3b-4e4d-aeb3-5bb250fa410c' AND created_at > '2015-05-13 00:01:00';

4678 (same as session-start?)

Now lets look at control group of bucket-assigned:

SELECT COUNT(*) FROM events WHERE type = 'bucket-assigned' AND control_group = true AND site_id = '995ead32-3c3b-4e4d-aeb3-5bb250fa410c' AND created_at > '2015-05-13 00:01:00';

2443 of those 4678 were in control (52%)

Now lets look at test group of bucket-assigned:

SELECT COUNT(*) FROM events WHERE type = 'bucket-assigned' AND control_group = false AND site_id = '995ead32-3c3b-4e4d-aeb3-5bb250fa410c' AND created_at > '2015-05-13 00:01:00';

2235 of those 4678 were in test (48%)

Sooo I think our issue is that we are comparing session-start in events with offer_qualifications. What if we compared session-start in events in the control group ...

SELECT COUNT(*) FROM events WHERE type = 'session-start' AND control_group = true AND site_id = '995ead32-3c3b-4e4d-aeb3-5bb250fa410c' AND created_at > '2015-05-13 00:01:00';

2443 of those 4678 were in control (52%)

... with the test group:

SELECT COUNT(*) FROM events WHERE type = 'session-start' AND control_group = false AND site_id = '995ead32-3c3b-4e4d-aeb3-5bb250fa410c' AND created_at > '2015-05-13 00:01:00';

2235 of those 4678 were in test (48%)

Looks good. Now lets forget offer_qualifications and look at shopper-qualified-offers in events:

SELECT COUNT(*) FROM events WHERE type = 'shopper-qualified-offers' AND control_group = false AND site_id = '995ead32-3c3b-4e4d-aeb3-5bb250fa410c' AND created_at > '2015-05-13 00:01:00';

2221

SELECT COUNT(*) FROM events WHERE type = 'shopper-qualified-offers' AND control_group = true AND site_id = '995ead32-3c3b-4e4d-aeb3-5bb250fa410c' AND created_at > '2015-05-13 00:01:00';

2293

[bpromo]

Ok so I'm thinking:

Visits should be the total count of session-start events as it is now: 4678

Offered should be ~ 50% of Visits assuming no real conditions. If we use shopper-qualified-offers where control_group = false we get: 2221 which is 47.4% of Visits

I do not know why offer_qualifications isn't showing us what we need but I think it may have to do with control_group which I raised in another ticket a bit ago. I also am not entirely sure why we need the table in the first place.