Add eligible segment filter for bandits

[kjozwiak]

Labelling as Android as per confirmation from @tmancey & @moritzhaller from today's meeting that epsilon greedy bandit classifier will also be on Android. Also, https://github.com/brave/brave-browser/issues/12595 was labelled with OS/Android as well.

[stephendonner]

Verified using the test plan in https://github.com/brave/brave-core/pull/7533 with additional contextual details, from @moritzhaller and @tmancey (and thx to @btlechowski for pro-tips, too!)

Brave	1.20.97 Chromium: 88.0.4324.96 (Official Build) dev (x86_64)
Revision	68dba2d8a0b149a1d3afac56fa74648032bcf46b-refs/branch-heads/4324@{#1784}

Case 1: Default models enabled (Text Classification and Purchase Intent)

Used --args --enable-logging=stderr --vmodule="*/bat-native-ledger/*"=6,"*/brave_rewards/*"=6,"*/bat-native-ads/*"=6,"*/bat-native-confirmations/*"=6,"*/brave_ads/*"=6,"*/brave_user_model/*"=6 --brave-ads-staging --brave-ads-debug --rewards=staging=true,reconcile-interval=5

Verify that Text Classification and Purchase Intent models are enabled and Epsilon Greedy Bandit is disabled

```[14716:775:0129/091938.057797:VERBOSE1:features.cc(59)] Text classification feature is enabled [14716:775:0129/091938.057839:VERBOSE1:features.cc(61)] Epsilon greedy bandit feature is disabled [14716:775:0129/091938.057884:VERBOSE1:features.cc(63)] Purchase intent feature is enabled ```

Trigger ad notification and verify that ads are served as usual via text classification

``` [14716:775:0129/092959.983897:VERBOSE1:text_classification_processor.cc(62)] Classified text with the top segment as gaming-gaming [14716:775:0129/093025.448373:VERBOSE1:ad_notification_serving.cc(101)] Ad notification delivered: uuid: f41fdce0-62b0-42dc-99b6-77fe84db5083 creativeInstanceId: ed6c467a-8643-42eb-a7cd-32e878fa3cd4 creativeSetId: a9f0b49c-3298-4c1b-a8b5-76a71e8d7332 campaignId: 0e70c893-b0c0-4dab-860d-7e65446f0f0f segment: untargeted title: This is a test body: This is a test targetUrl: https://www.brave.com [14716:775:0129/093025.448421:VERBOSE3:ad_notification_event_viewed.cc(24)] Viewed ad notification with uuid f41fdce0-62b0-42dc-99b6-77fe84db5083 and creative instance id ed6c467a-8643-42eb-a7cd-32e878fa3cd4 [14716:775:0129/093025.448808:VERBOSE1:confirmations.cc(90)] Confirming view ad for creative instance id ed6c467a-8643-42eb-a7cd-32e878fa3cd4 ```

Process purchase intent segment and verify keywords are extracted

``` [14716:775:0129/091938.122877:VERBOSE1:purchase_intent_resource.cc(58)] Successfully loaded kkjipiepeooghlclkedllogndmohhnhi purchase intent resource [14716:775:0129/091938.124779:VERBOSE1:purchase_intent_resource.cc(235)] Parsed purchase intent user model version 1 [14716:775:0129/091938.125023:VERBOSE1:purchase_intent_resource.cc(68)] Successfully initialized kkjipiepeooghlclkedllogndmohhnhi purchase intent resource ```

Verify that bandit is processing (despite the model being disabled, that is by design)

``` [14716:775:0129/093027.940733:VERBOSE1:epsilon_greedy_bandit_processor.cc(118)] Epsilon greedy bandit processed BraveAdsAdNotificationEventType::kClicked [14716:775:0129/093027.947347:VERBOSE1:ad_notification_event_clicked.cc(36)] Successfully logged ad notification clicked event ```

Verify that the model is not suggesting segments to the ad serve pipeline

Nothing shows up under Exploring... or Exploiting...

Case 2: Variant models enabled (Epsilon Greedy Bandit and Purchase Intent: --disable-features="TextClassification" --enable-features="EpsilonGreedyBandit")

Used --args --enable-logging=stderr --vmodule="*/bat-native-ledger/*"=6,"*/brave_rewards/*"=6,"*/bat-native-ads/*"=6,"*/bat-native-confirmations/*"=6,"*/brave_ads/*"=6,"*/brave_user_model/*"=6 --brave-ads-debug --disable-features="TextClassification" --enable-features="EpsilonGreedyBandit"

Verify that Epsilon Greedy Bandit and Purchase Intent models are enabled and Text Classification is disabled

``` [15207:775:0129/103356.166699:VERBOSE1:features.cc(59)] Text classification feature is disabled [15207:775:0129/103356.166742:VERBOSE1:features.cc(61)] Epsilon greedy bandit feature is enabled [15207:775:0129/103356.166784:VERBOSE1:features.cc(63)] Purchase intent feature is enabled ```

Process purchase intent segment and verify keywords are extracted

``` [15207:775:0129/104513.181256:VERBOSE1:purchase_intent_processor.cc(101)] Extracted purchase intent signal from visited URL ```

Verify that bandit is processing

``` [15207:775:0129/103356.123110:VERBOSE2:epsilon_greedy_bandit_processor.cc(51)] Epsilon greedy bandit invalid arm was reset for architecture segment (The above repeats for each segment) ```

Verify that the model is suggesting segments to the ad serve pipeline

``` [15207:775:0129/104244.918132:VERBOSE2:epsilon_greedy_bandit_model.cc(165)] Exploiting epsilon greedy bandit segments: [15207:775:0129/104244.920762:VERBOSE1:ad_notification_serving.cc(160)] No segments to serve targeted ads [15207:775:0129/104244.920807:VERBOSE1:ad_notification_serving.cc(226)] Serve untargeted ad ```

Case 3: Restart after one round of training and make sure state wasn't deleted in user Preferences JSON key epsilon_greedy_bandit_arms

In addition to the details below, I verified that the pulls and values data persisted between sessions/restarts of the browser, and were accurately reflected via brave://rewards 7-day History panel.

Used --args --enable-logging=stderr --vmodule="*/bat-native-ledger/*"=6,"*/brave_rewards/*"=6,"*/bat-native-ads/*"=6,"*/bat-native-confirmations/*"=6,"*/brave_ads/*"=6,"*/brave_user_model/*"=6 --brave-ads-debug --disable-features="TextClassification" --enable-features="EpsilonGreedyBandit"

Here, I'm covering those ads from segments pulled from the catalog (and shown), which I either `clicked` or `dismissed`. Represented as a percentage (**%**), `clicking` will _retain_ or _increase_ (if multiple pulls and views) the `value`, and conversely, `dismissing` or `timing out` will _decrease_ the `value`. So, with 2 `pulls` for a particular segment, but only 1 `click`, `value 0.5` = **50%** click-through rate. With only 1 `pull` and 1 `click`, `value = 1.0` (**100%**). 2 `pulls` **without** clicks will set `value` to `0.0` (**0%**). Before training, each segment's (ad) `pulls` is set to 0, and its `value` is set to `1.0` (**100%**). key for the below: segment category, _ad name_, clicked or dismissed (`pulls`, `value`, as a **%** of click-throughs) * arts & entertainment, _Splinterlands_, **dismissed** (**1** `pull`, `value 0.0` = **0%**) * business, _Money Metals Exchange_, **dismissed** (**1** `pull`, `value 0.0` = **0%**) * cell phones, _Stay Connected w/SIMPLE_, **clicked** (**1** `pull`, `value 1.0` = **100%**) * education, _BAT = Chances to Win_, **clicked** (**1** `pull`, `value 1.0` = **100%**) * family & parenting, _Lustre for Brave_, **clicked** (**1** `pull`, `value 1.0` = **100%**) * general (unlisted in bandit training; from new tab page), 6 _Gala Games_ impressions (new tab page), clicked 1 (not represented here in JSON/pulls) * hobbies & interests (1/2), _Gemini_, **dismissed** (_1st_ of **2 total** `pulls`, total `value 0.5` = **50%**) * hobbies & interests (2/2), _Gala Games_, **clicked** (_2nd_ of **2 total** `pulls`, total `value 0.5` = **50%**) * personal finance (1/2), _BAT: First Private Ad Network_, **clicked** (_1st_ of **2 total** `pulls`, total `value 1.0` = **100%**) * personal finance (2/2), _Lending Tree_, **clicked** (_2nd_ of **2 total** `pulls`, `value 1.0` = **100%**) * travel, _Coinbase_, **dismissed** (**1** `pull`, `value 0.0` = **0%**) References: a) **before** and **after** training comparison, in `Preferences.json`, found in `~/Library/Application Support/BraveSoftware/Brave-Browser-Dev/Default/Preferences/` b) relevant `brave://rewards/` ad history

---

Verification passed on

Brave	1.20.97 Chromium: 88.0.4324.96 (Official Build) dev (64-bit)
Revision	68dba2d8a0b149a1d3afac56fa74648032bcf46b-refs/branch-heads/4324@{#1784}
OS	Ubuntu 18.04 LTS

Verified using the test plan in https://github.com/brave/brave-core/pull/7533

Default models enabled (Text Classification and Purchase Intent)

Verify that Text Classification and Purchase Intent models are enabled and Epsilon Greedy Bandit is disabled

``` [2792:2792:0208/234131.018377:VERBOSE1:features.cc(59)] Text classification feature is enabled [2792:2792:0208/234131.018400:VERBOSE1:features.cc(61)] Epsilon greedy bandit feature is disabled [2792:2792:0208/234131.018435:VERBOSE1:features.cc(63)] Purchase intent feature is enabled ```

Trigger ad notification and verify that ads are served as usual via text classification

``` [2792:2792:0208/234320.947896:VERBOSE1:text_classification_processor.cc(62)] Classified text with the top segment as technology & computing-software [4134:4134:0208/235705.246825:VERBOSE1:ad_notification_serving.cc(165)] Serve ad for segments: [4134:4134:0208/235705.246953:VERBOSE1:ad_notification_serving.cc(167)] technology & computing-software [4134:4134:0208/235705.247044:VERBOSE1:ad_notification_serving.cc(167)] personal finance-personal finance [4134:4134:0208/235705.247144:VERBOSE1:ad_notification_serving.cc(167)] home-appliances [4134:4134:0208/235705.248500:VERBOSE2:ad_notification_serving.cc(273)] 1 eligible ads before pacing [4134:4134:0208/235705.248621:VERBOSE2:eligible_ads_priority_filter.cc(81)] 1 eligible ads with a priority of 1 [4134:4134:0208/235705.248736:VERBOSE2:ad_notification_serving.cc(278)] 1 eligible ads after pacing [4134:4134:0208/235705.248838:VERBOSE1:ad_notification_serving.cc(264)] Found 1 eligible ads [4134:4134:0208/235705.252155:VERBOSE1:ads_service_impl.cc(1769)] Timeout ad notification with uuid db014190-893c-4543-9479-3255206e2e86 in 120 seconds [4134:4134:0208/235705.252671:VERBOSE1:ad_notification_serving.cc(101)] Ad notification delivered: uuid: db014190-893c-4543-9479-3255206e2e86 creativeInstanceId: 30ecd70f-f56d-4745-a457-4035fcab54b6 creativeSetId: 0a697a3b-233a-44bc-a337-e2441fb277a7 campaignId: d4d0474c-277a-4714-b25d-7b484cdb580a segment: technology & computing-software title: This is a test body: This is a test targetUrl: https://www.brave.com ```

Process purchase intent segment and verify sites are extracted

``` [4748:4748:0208/235931.075227:VERBOSE1:purchase_intent_processor.cc(101)] Extracted purchase intent signal from visited URL ``` ``` "purchaseIntentSignalHistory":{"AC001":[{"timestamp_in_seconds":1612825171,"weight":1}]} ```

Process purchase intent segment and verify keywords are extracted

``` [6014:6014:0209/000651.762424:VERBOSE1:purchase_intent_processor.cc(101)] Extracted purchase intent signal from visited URL ``` ``` "purchaseIntentSignalHistory":{"BF001":[{"timestamp_in_seconds":1612825609,"weight":1} ```

Verify that bandit is processing (despite the model being disabled, that is by design)

``` [6014:6014:0209/001033.366224:VERBOSE1:epsilon_greedy_bandit_processor.cc(118)] Epsilon greedy bandit processed BraveAdsAdNotificationEventType::kDismissed ```

Verify that the model is not suggesting segments to the ad serve pipeline

Nothing shows up under Exploring... or Exploiting...

Variant models enabled (Epsilon Greedy Bandit and Purchase Intent: --disable-features="TextClassification" --enable-features="EpsilonGreedyBandit")

Used --args --enable-logging=stderr --vmodule="*/bat-native-ledger/*"=6,"*/brave_rewards/*"=6,"*/bat-native-ads/*"=6,"*/bat-native-confirmations/*"=6,"*/brave_ads/*"=6,"*/brave_user_model/*"=6 --brave-ads-debug --disable-features="TextClassification" --enable-features="EpsilonGreedyBandit"

Verify that Epsilon Greedy Bandit and Purchase Intent models are enabled and Text Classification is disabled

``` [7253:7253:0209/001451.561996:VERBOSE1:features.cc(59)] Text classification feature is disabled [7253:7253:0209/001451.562154:VERBOSE1:features.cc(61)] Epsilon greedy bandit feature is enabled [7253:7253:0209/001451.562343:VERBOSE1:features.cc(63)] Purchase intent feature is enabled ```

Process purchase intent segment and verify keywords are extracted

``` [7253:7253:0209/001642.064751:VERBOSE1:purchase_intent_processor.cc(101)] Extracted purchase intent signal from visited URL ``` ``` "purchaseIntentSignalHistory":{"Mortgage and Refinance":[{"timestamp_in_seconds":1612826202,"weight":1} ```

Verify that bandit is processing

``` [7253:7253:0209/001451.287621:VERBOSE2:epsilon_greedy_bandit_processor.cc(51)] Epsilon greedy bandit invalid arm was reset for family & parenting segment [7253:7253:0209/001451.287707:VERBOSE2:epsilon_greedy_bandit_processor.cc(51)] Epsilon greedy bandit invalid arm was reset for fashion segment [7253:7253:0209/001451.287801:VERBOSE2:epsilon_greedy_bandit_processor.cc(51)] Epsilon greedy bandit invalid arm was reset for folklore segment [7253:7253:0209/001451.287887:VERBOSE2:epsilon_greedy_bandit_processor.cc(51)] Epsilon greedy bandit invalid arm was reset for food & drink segment [7253:7253:0209/001451.288014:VERBOSE2:epsilon_greedy_bandit_processor.cc(51)] Epsilon greedy bandit invalid arm was reset for health & fitness segment [7253:7253:0209/001451.288100:VERBOSE2:epsilon_greedy_bandit_processor.cc(51)] Epsilon greedy bandit invalid arm was reset for history segment [7253:7253:0209/001451.288194:VERBOSE2:epsilon_greedy_bandit_processor.cc(51)] Epsilon greedy bandit invalid arm was reset for hobbies & interests segment [7253:7253:0209/001451.288279:VERBOSE2:epsilon_greedy_bandit_processor.cc(51)] Epsilon greedy bandit invalid arm was reset for home segment [7253:7253:0209/001451.288373:VERBOSE2:epsilon_greedy_bandit_processor.cc(51)] Epsilon greedy bandit invalid arm was reset for law segment [7253:7253:0209/001451.288489:VERBOSE2:epsilon_greedy_bandit_processor.cc(51)] Epsilon greedy bandit invalid arm was reset for military segment [7253:7253:0209/001451.288586:VERBOSE2:epsilon_greedy_bandit_processor.cc(51)] Epsilon greedy bandit invalid arm was reset for personal finance segment [7253:7253:0209/001451.288671:VERBOSE2:epsilon_greedy_bandit_processor.cc(51)] Epsilon greedy bandit invalid arm was reset for pets segment [7253:7253:0209/001451.288765:VERBOSE2:epsilon_greedy_bandit_processor.cc(51)] Epsilon greedy bandit invalid arm was reset for politics segment [7253:7253:0209/001451.288851:VERBOSE2:epsilon_greedy_bandit_processor.cc(51)] Epsilon greedy bandit invalid arm was reset for real estate segment [7253:7253:0209/001451.289155:VERBOSE2:epsilon_greedy_bandit_processor.cc(51)] Epsilon greedy bandit invalid arm was reset for religion segment [7253:7253:0209/001451.289262:VERBOSE2:epsilon_greedy_bandit_processor.cc(51)] Epsilon greedy bandit invalid arm was reset for science segment [7253:7253:0209/001451.289348:VERBOSE2:epsilon_greedy_bandit_processor.cc(51)] Epsilon greedy bandit invalid arm was reset for society segment [7253:7253:0209/001451.289497:VERBOSE2:epsilon_greedy_bandit_processor.cc(51)] Epsilon greedy bandit invalid arm was reset for sports segment [7253:7253:0209/001451.289584:VERBOSE2:epsilon_greedy_bandit_processor.cc(51)] Epsilon greedy bandit invalid arm was reset for technology & computing segment [7253:7253:0209/001451.289678:VERBOSE2:epsilon_greedy_bandit_processor.cc(51)] Epsilon greedy bandit invalid arm was reset for travel segment [7253:7253:0209/001451.289763:VERBOSE2:epsilon_greedy_bandit_processor.cc(51)] Epsilon greedy bandit invalid arm was reset for weather segment [7253:7253:0209/001451.289858:VERBOSE2:epsilon_greedy_bandit_processor.cc(51)] Epsilon greedy bandit invalid arm was reset for crypto segment ```

Verify that the model is suggesting segments to the ad serve pipeline

``` [7253:7253:0209/001630.676907:VERBOSE2:epsilon_greedy_bandit_model.cc(165)] Exploiting epsilon greedy bandit segments: [7253:7253:0209/001630.676950:VERBOSE2:epsilon_greedy_bandit_model.cc(167)] hobbies & interests [7253:7253:0209/001630.677069:VERBOSE2:epsilon_greedy_bandit_model.cc(167)] business [7253:7253:0209/001630.677102:VERBOSE2:epsilon_greedy_bandit_model.cc(167)] technology & computing ```

Restart after one round of training and make sure state wasn't deleted in user Preferences JSON key epsilon_greedy_bandit_arms

In addition to the details below, I verified that the pulls and values data persisted between sessions/restarts of the browser, and were accurately reflected via brave://rewards 7-day History panel.

First `technology & computing-software` ad clicked ``` [9702:9702:0209/003830.104953:VERBOSE1:ads_impl.cc(186)] Browser state changed to unidle [9702:9702:0209/003830.105835:VERBOSE2:epsilon_greedy_bandit_model.cc(165)] Exploiting epsilon greedy bandit segments: [9702:9702:0209/003830.105881:VERBOSE2:epsilon_greedy_bandit_model.cc(167)] technology & computing [9702:9702:0209/003830.109432:VERBOSE1:ad_notification_serving.cc(165)] Serve ad for segments: [9702:9702:0209/003830.109474:VERBOSE1:ad_notification_serving.cc(167)] technology & computing [9702:9702:0209/003830.128211:VERBOSE2:ad_notification_serving.cc(273)] 2 eligible ads before pacing [9702:9702:0209/003830.128261:VERBOSE2:eligible_ads_priority_filter.cc(81)] 2 eligible ads with a priority of 1 [9702:9702:0209/003830.128302:VERBOSE2:ad_notification_serving.cc(278)] 2 eligible ads after pacing [9702:9702:0209/003830.128322:VERBOSE1:ad_notification_serving.cc(264)] Found 2 eligible ads [9702:9702:0209/003830.130100:VERBOSE1:ads_service_impl.cc(1769)] Timeout ad notification with uuid a7e8bd1b-5c36-4258-9312-446075ba4eda in 120 seconds [9702:9702:0209/003830.130191:VERBOSE1:ad_notification_serving.cc(101)] Ad notification delivered: uuid: a7e8bd1b-5c36-4258-9312-446075ba4eda creativeInstanceId: ed6c467a-8643-42eb-a7cd-32e878fa3cd4 creativeSetId: a9f0b49c-3298-4c1b-a8b5-76a71e8d7332 campaignId: 0e70c893-b0c0-4dab-860d-7e65446f0f0f segment: technology & computing title: This is a test 1 body: This is a test 1 targetUrl: https://www.brave.com ``` After ad is clicked the value is increased to 1 `{\"pulls\":1,\"segment\":\"technology & computing\",\"value\":1.0}` ``` "epsilon_greedy_bandit_arms": "{\"architecture\":{\"pulls\":0,\"segment\":\"architecture\",\"value\":1.0},\"arts & entertainment\":{\"pulls\":0,\"segment\":\"arts & entertainment\",\"value\":1.0},\"automotive\":{\"pulls\":0,\"segment\":\"automotive\",\"value\":1.0},\"business\":{\"pulls\":0,\"segment\":\"business\",\"value\":1.0},\"careers\":{\"pulls\":0,\"segment\":\"careers\",\"value\":1.0},\"cell phones\":{\"pulls\":0,\"segment\":\"cell phones\",\"value\":1.0},\"crypto\":{\"pulls\":0,\"segment\":\"crypto\",\"value\":1.0},\"drugs\":{\"pulls\":0,\"segment\":\"drugs\",\"value\":1.0},\"education\":{\"pulls\":0,\"segment\":\"education\",\"value\":1.0},\"family & parenting\":{\"pulls\":0,\"segment\":\"family & parenting\",\"value\":1.0},\"fashion\":{\"pulls\":0,\"segment\":\"fashion\",\"value\":1.0},\"folklore\":{\"pulls\":0,\"segment\":\"folklore\",\"value\":1.0},\"food & drink\":{\"pulls\":0,\"segment\":\"food & drink\",\"value\":1.0},\"health & fitness\":{\"pulls\":0,\"segment\":\"health & fitness\",\"value\":1.0},\"history\":{\"pulls\":0,\"segment\":\"history\",\"value\":1.0},\"hobbies & interests\":{\"pulls\":0,\"segment\":\"hobbies & interests\",\"value\":1.0},\"home\":{\"pulls\":0,\"segment\":\"home\",\"value\":1.0},\"law\":{\"pulls\":0,\"segment\":\"law\",\"value\":1.0},\"military\":{\"pulls\":0,\"segment\":\"military\",\"value\":1.0},\"personal finance\":{\"pulls\":0,\"segment\":\"personal finance\",\"value\":1.0},\"pets\":{\"pulls\":0,\"segment\":\"pets\",\"value\":1.0},\"politics\":{\"pulls\":0,\"segment\":\"politics\",\"value\":1.0},\"real estate\":{\"pulls\":0,\"segment\":\"real estate\",\"value\":1.0},\"religion\":{\"pulls\":0,\"segment\":\"religion\",\"value\":1.0},\"science\":{\"pulls\":0,\"segment\":\"science\",\"value\":1.0},\"society\":{\"pulls\":0,\"segment\":\"society\",\"value\":1.0},\"sports\":{\"pulls\":0,\"segment\":\"sports\",\"value\":1.0},\"technology & computing\":{\"pulls\":1,\"segment\":\"technology & computing\",\"value\":1.0},\"travel\":{\"pulls\":0,\"segment\":\"travel\",\"value\":1.0},\"weather\":{\"pulls\":0,\"segment\":\"weather\",\"value\":1.0}}", "epsilon_greedy_bandit_eligible_segments": "[\"technology & computing\"]", ``` Second ad was dismissed ``` [10393:10393:0209/005930.100768:VERBOSE1:ads_impl.cc(186)] Browser state changed to unidle [10393:10393:0209/005930.103106:VERBOSE2:epsilon_greedy_bandit_model.cc(165)] Exploiting epsilon greedy bandit segments: [10393:10393:0209/005930.103179:VERBOSE2:epsilon_greedy_bandit_model.cc(167)] technology & computing [10393:10393:0209/005930.108020:VERBOSE1:ad_notification_serving.cc(165)] Serve ad for segments: [10393:10393:0209/005930.108067:VERBOSE1:ad_notification_serving.cc(167)] technology & computing [10393:10393:0209/005930.109104:VERBOSE1:eligible_ad_notifications.cc(70)] All advertisers have been shown, so round robin [10393:10393:0209/005930.109181:VERBOSE1:client.cc(385)] Resetting seen advertisers [10393:10393:0209/005930.111029:VERBOSE2:ad_notification_serving.cc(273)] 1 eligible ads before pacing [10393:10393:0209/005930.111072:VERBOSE2:eligible_ads_priority_filter.cc(81)] 1 eligible ads with a priority of 1 [10393:10393:0209/005930.111116:VERBOSE2:ad_notification_serving.cc(278)] 1 eligible ads after pacing [10393:10393:0209/005930.111170:VERBOSE1:ad_notification_serving.cc(264)] Found 1 eligible ads [10393:10393:0209/005930.113991:VERBOSE1:ads_service_impl.cc(1769)] Timeout ad notification with uuid 65b35e8c-9730-4c3d-ac8c-8dc1f2367062 in 120 seconds [10393:10393:0209/005930.114175:VERBOSE1:ad_notification_serving.cc(101)] Ad notification delivered: uuid: 65b35e8c-9730-4c3d-ac8c-8dc1f2367062 creativeInstanceId: 30ecd70f-f56d-4745-a457-4035fcab54b6 creativeSetId: 0a697a3b-233a-44bc-a337-e2441fb277a7 campaignId: d4d0474c-277a-4714-b25d-7b484cdb580a segment: technology & computing title: This is a test 2 body: This is a test 2 targetUrl: https://www.brave.com [10393:10393:0209/010030.639697:VERBOSE6:ad_notification_event_dismissed.cc(36)] Successfully logged ad notification dismissed event ``` After ad is dismissed the value is decreased to 0.5 `{\"pulls\":2,\"segment\":\"technology & computing\",\"value\":0.5}` ``` "epsilon_greedy_bandit_arms": "{\"architecture\":{\"pulls\":0,\"segment\":\"architecture\",\"value\":1.0},\"arts & entertainment\":{\"pulls\":0,\"segment\":\"arts & entertainment\",\"value\":1.0},\"automotive\":{\"pulls\":0,\"segment\":\"automotive\",\"value\":1.0},\"business\":{\"pulls\":0,\"segment\":\"business\",\"value\":1.0},\"careers\":{\"pulls\":0,\"segment\":\"careers\",\"value\":1.0},\"cell phones\":{\"pulls\":0,\"segment\":\"cell phones\",\"value\":1.0},\"crypto\":{\"pulls\":0,\"segment\":\"crypto\",\"value\":1.0},\"drugs\":{\"pulls\":0,\"segment\":\"drugs\",\"value\":1.0},\"education\":{\"pulls\":0,\"segment\":\"education\",\"value\":1.0},\"family & parenting\":{\"pulls\":0,\"segment\":\"family & parenting\",\"value\":1.0},\"fashion\":{\"pulls\":0,\"segment\":\"fashion\",\"value\":1.0},\"folklore\":{\"pulls\":0,\"segment\":\"folklore\",\"value\":1.0},\"food & drink\":{\"pulls\":0,\"segment\":\"food & drink\",\"value\":1.0},\"health & fitness\":{\"pulls\":0,\"segment\":\"health & fitness\",\"value\":1.0},\"history\":{\"pulls\":0,\"segment\":\"history\",\"value\":1.0},\"hobbies & interests\":{\"pulls\":0,\"segment\":\"hobbies & interests\",\"value\":1.0},\"home\":{\"pulls\":0,\"segment\":\"home\",\"value\":1.0},\"law\":{\"pulls\":0,\"segment\":\"law\",\"value\":1.0},\"military\":{\"pulls\":0,\"segment\":\"military\",\"value\":1.0},\"personal finance\":{\"pulls\":0,\"segment\":\"personal finance\",\"value\":1.0},\"pets\":{\"pulls\":0,\"segment\":\"pets\",\"value\":1.0},\"politics\":{\"pulls\":0,\"segment\":\"politics\",\"value\":1.0},\"real estate\":{\"pulls\":0,\"segment\":\"real estate\",\"value\":1.0},\"religion\":{\"pulls\":0,\"segment\":\"religion\",\"value\":1.0},\"science\":{\"pulls\":0,\"segment\":\"science\",\"value\":1.0},\"society\":{\"pulls\":0,\"segment\":\"society\",\"value\":1.0},\"sports\":{\"pulls\":0,\"segment\":\"sports\",\"value\":1.0},\"technology & computing\":{\"pulls\":2,\"segment\":\"technology & computing\",\"value\":0.5},\"travel\":{\"pulls\":0,\"segment\":\"travel\",\"value\":1.0},\"weather\":{\"pulls\":0,\"segment\":\"weather\",\"value\":1.0}}", "epsilon_greedy_bandit_eligible_segments": "[\"technology & computing\"]", ```

[GeetaSarvadnya]

Verification passed on

Brave | 1.20.100 Chromium: 88.0.4324.146 (Official Build) (64-bit)
-- | --
Revision | 406dc88511162d6598242f2c709be1414a042fb0-refs/branch-heads/4324@{#2088}
OS | Windows 10 OS Version 2004 (Build 19041.746)

• Verified the test plan from https://github.com/brave/brave-core/pull/7533

Case 1: Default models enabled

- Verified that `Text Classification` and `Purchase Intent models` are enabled and `Epsilon Greedy Bandit` is disabled by default ``` [18372:19208:0208/193551.986:VERBOSE1:features.cc(59)] Text classification feature is enabled [18372:19208:0208/193551.987:VERBOSE1:features.cc(61)] Epsilon greedy bandit feature is disabled [18372:19208:0208/193551.988:VERBOSE1:features.cc(63)] Purchase intent feature is enabled ``` - Viewed an AD and ensured ads are triggered via text classification ``` [15184:11084:0209/002915.741:VERBOSE1:text_classification_processor.cc(62)] Classified text with the top segment as personal finance-personal finance [15184:11084:0209/002915.747:VERBOSE1:conversions.cc(151)] No conversions found for visited URL [15184:11084:0209/002928.228:VERBOSE1:ads_impl.cc(178)] Browser state changed to idle [15184:11084:0209/002944.331:VERBOSE1:ads_impl.cc(186)] Browser state changed to unidle [15184:11084:0209/002944.368:VERBOSE1:ad_notification_serving.cc(165)] Serve ad for segments: [15184:11084:0209/002944.368:VERBOSE1:ad_notification_serving.cc(167)] personal finance-personal finance [15184:11084:0209/002944.373:VERBOSE1:ad_notification_serving.cc(167)] food & drink-cooking [15184:11084:0209/002944.381:VERBOSE1:ad_notification_serving.cc(167)] business-marketing [15184:11084:0209/002944.384:VERBOSE1:ad_notification_serving.cc(182)] No eligible ads found for segments [15184:11084:0209/002944.384:VERBOSE1:ad_notification_serving.cc(197)] Serve ad for parent segments: [15184:11084:0209/002944.384:VERBOSE1:ad_notification_serving.cc(199)] personal finance [15184:11084:0209/002944.385:VERBOSE1:ad_notification_serving.cc(199)] food & drink [15184:11084:0209/002944.386:VERBOSE1:ad_notification_serving.cc(199)] business [15184:11084:0209/002944.387:VERBOSE1:ad_notification_serving.cc(214)] No eligible ads found for parent segments [15184:11084:0209/002944.388:VERBOSE1:ad_notification_serving.cc(226)] Serve untargeted ad [15184:11084:0209/002944.390:VERBOSE2:ad_notification_serving.cc(273)] 3 eligible ads before pacing [15184:11084:0209/002944.390:VERBOSE2:eligible_ads_priority_filter.cc(81)] 3 eligible ads with a priority of 1 [15184:11084:0209/002944.391:VERBOSE2:ad_notification_serving.cc(278)] 3 eligible ads after pacing [15184:11084:0209/002944.391:VERBOSE1:ad_notification_serving.cc(264)] Found 3 eligible ads [15184:11084:0209/002944.392:VERBOSE1:ads_service_impl.cc(1769)] Timeout ad notification with uuid 3a4e401f-d470-4dda-b70b-6e1c29d54b2a in 120 seconds [15184:11084:0209/002944.392:VERBOSE1:ad_notification_serving.cc(101)] Ad notification delivered: uuid: 3a4e401f-d470-4dda-b70b-6e1c29d54b2a creativeInstanceId: 30ecd70f-f56d-4745-a457-4035fcab54b6 creativeSetId: 0a697a3b-233a-44bc-a337-e2441fb277a7 campaignId: d4d0474c-277a-4714-b25d-7b484cdb580a segment: untargeted title: This is a test body: This is a test targetUrl: https://www.brave.com [15184:11084:0209/002944.393:VERBOSE3:ad_notification_event_viewed.cc(24)] Viewed ad notification with uuid 3a4e401f-d470-4dda-b70b-6e1c29d54b2a and creative instance id 30ecd70f-f56d-4745-a457-4035fcab54b6 [15184:11084:0209/002944.402:VERBOSE1:confirmations.cc(90)] Confirming view ad for creative instance id 30ecd70f-f56d-4745-a457-4035fcab54b6 ``` - Viewed an AD using the search keyword `coinbase earn` and ensured purchase intent for **Keywords** are working as expected ``` [14368:16036:0209/001816.206:VERBOSE1:purchase_intent_resource.cc(58)] Successfully loaded kkjipiepeooghlclkedllogndmohhnhi purchase intent resource [14368:16036:0209/001816.208:VERBOSE1:purchase_intent_resource.cc(235)] Parsed purchase intent user model version 1 [14368:16036:0209/001816.208:VERBOSE1:purchase_intent_resource.cc(68)] Successfully initialized kkjipiepeooghlclkedllogndmohhnhi purchase intent resource ``` ``` [14368:16036:0209/001842.627:VERBOSE1:purchase_intent_processor.cc(101)] Extracted purchase intent signal from visited URL [14368:16036:0209/001842.628:VERBOSE1:ads_impl.cc(170)] Search engine pages are not supported for text classification [14368:16036:0209/001842.637:VERBOSE1:conversions.cc(151)] No conversions found for visited URL [14368:16036:0209/001843.100:VERBOSE2:tab_manager.cc(93)] Tab id 4 was updated [14368:16036:0209/001843.105:VERBOSE1:conversions.cc(130)] Checking URL for conversions [14368:16036:0209/001843.114:VERBOSE1:purchase_intent_processor.cc(101)] Extracted purchase intent signal from visited URL [14368:16036:0209/001843.115:VERBOSE1:ads_impl.cc(170)] Search engine pages are not supported for text classification [14368:16036:0209/001843.126:VERBOSE1:conversions.cc(151)] No conversions found for visited URL [14368:16036:0209/001857.123:VERBOSE1:ads_impl.cc(178)] Browser state changed to idle [14368:16036:0209/001911.246:VERBOSE1:ads_impl.cc(186)] Browser state changed to unidle [14368:16036:0209/001911.249:VERBOSE1:text_classification_model.cc(94)] No text classification probabilities found for en-US locale [14368:16036:0209/001911.262:VERBOSE1:ad_notification_serving.cc(165)] Serve ad for segments: [14368:16036:0209/001911.262:VERBOSE1:ad_notification_serving.cc(167)] untargeted [14368:16036:0209/001911.265:VERBOSE2:ad_notification_serving.cc(273)] 3 eligible ads before pacing [14368:16036:0209/001911.265:VERBOSE2:eligible_ads_priority_filter.cc(81)] 3 eligible ads with a priority of 1 [14368:16036:0209/001911.265:VERBOSE2:ad_notification_serving.cc(278)] 3 eligible ads after pacing [14368:16036:0209/001911.273:VERBOSE1:ad_notification_serving.cc(264)] Found 3 eligible ads [14368:16036:0209/001911.274:VERBOSE1:ads_service_impl.cc(1769)] Timeout ad notification with uuid 4ae50fd1-ab26-4422-a539-bc4d36f919ce in 120 seconds [14368:16036:0209/001911.274:VERBOSE1:ad_notification_serving.cc(101)] Ad notification delivered: uuid: 4ae50fd1-ab26-4422-a539-bc4d36f919ce creativeInstanceId: 4983dc92-09fd-4e5c-94c3-616a4728c1d9 creativeSetId: 5bdeab83-048f-48a7-9602-a1092ded123c campaignId: 63010037-c230-40db-b41a-3127619b88e4 segment: untargeted title: This is a test body: This is a test targetUrl: https://www.brave.com [14368:16036:0209/001911.275:VERBOSE3:ad_notification_event_viewed.cc(24)] Viewed ad notification with uuid 4ae50fd1-ab26-4422-a539-bc4d36f919ce and creative instance id 4983dc92-09fd-4e5c-94c3-616a4728c1d9 [14368:16036:0209/001911.275:VERBOSE1:confirmations.cc(90)] Confirming view ad for creative instance id 4983dc92-09fd-4e5c-94c3-616a4728c1d9 ``` - Ensured **Epsilon greedy bandit** is processed (despite the model being disabled, that is by design) ``` [14368:16036:0209/001913.532:VERBOSE1:epsilon_greedy_bandit_processor.cc(118)] Epsilon greedy bandit processed BraveAdsAdNotificationEventType::kClicked [14368:16036:0209/001913.541:VERBOSE1:ad_notification_event_clicked.cc(36)] Successfully logged ad notification clicked event ``` - Ensured model is not suggesting segments to the ad serve pipeline and didn't see `Exploring epsilon greedy bandit segments: ...` or `Exploiting epsilon greedy bandit segments: ...`

Case 2: Variant models enabled

- Verified that `Purchase Intent models` and `Epsilon Greedy Bandit` are enabled and `Text Classification` is disabled when the brave is launched using cmd line: `--enable-logging=stderr --vmodule="*/bat-native-ads/*"=9,"*/brave_ads/*"=9,"*/brave_user_model/*"=9,"*/bat_ads/*"=9 --brave-ads-staging --rewards=staging=true --disable-features="TextClassification" --enable-features="EpsilonGreedyBandit"` ``` [2292:8648:0208/232707.461:VERBOSE1:features.cc(59)] Text classification feature is disabled [2292:8648:0208/232707.462:VERBOSE1:features.cc(61)] Epsilon greedy bandit feature is enabled [2292:8648:0208/232707.463:VERBOSE1:features.cc(63)] Purchase intent feature is enabled ``` - Viewed an AD using the search keyword **best bitcoin lending sites** and ensured purchase intent for **Keywords** are working as expected ``` [17692:6200:0209/003226.279:VERBOSE1:purchase_intent_resource.cc(58)] Successfully loaded kkjipiepeooghlclkedllogndmohhnhi purchase intent resource [17692:6200:0209/003226.281:VERBOSE1:purchase_intent_resource.cc(235)] Parsed purchase intent user model version 1 [17692:6200:0209/003226.292:VERBOSE1:purchase_intent_resource.cc(68)] Successfully initialized kkjipiepeooghlclkedllogndmohhnhi purchase intent resource ``` ``` [17692:6200:0209/003312.822:VERBOSE1:purchase_intent_processor.cc(101)] Extracted purchase intent signal from visited URL [17692:6200:0209/003312.823:VERBOSE9:client.cc(455)] Saving client state [17692:6200:0209/003312.823:VERBOSE9:client.cc(455)] Saving client state [17692:6200:0209/003312.834:VERBOSE1:ads_impl.cc(170)] Search engine pages are not supported for text classification [17692:6200:0209/003312.837:VERBOSE9:client.cc(470)] Successfully saved client state [17692:6200:0209/003312.841:VERBOSE9:client.cc(470)] Successfully saved client state [17692:6200:0209/003312.846:VERBOSE1:conversions.cc(151)] No conversions found for visited URL [17692:6200:0209/003313.285:VERBOSE2:tab_manager.cc(93)] Tab id 4 was updated [17692:6200:0209/003313.287:VERBOSE1:conversions.cc(130)] Checking URL for conversions [17692:6200:0209/003313.291:VERBOSE1:purchase_intent_processor.cc(101)] Extracted purchase intent signal from visited URL [17692:6200:0209/003313.292:VERBOSE9:client.cc(455)] Saving client state [17692:6200:0209/003313.296:VERBOSE9:client.cc(455)] Saving client state [17692:6200:0209/003313.297:VERBOSE1:ads_impl.cc(170)] Search engine pages are not supported for text classification [17692:6200:0209/003313.315:VERBOSE9:client.cc(470)] Successfully saved client state [17692:6200:0209/003313.323:VERBOSE9:client.cc(470)] Successfully saved client state [17692:6200:0209/003313.323:VERBOSE1:conversions.cc(151)] No conversions found for visited URL [17692:6200:0209/003327.824:VERBOSE1:ads_impl.cc(178)] Browser state changed to idle [17692:6200:0209/003341.933:VERBOSE1:ads_impl.cc(186)] Browser state changed to unidle [17692:6200:0209/003341.936:VERBOSE2:epsilon_greedy_bandit_model.cc(165)] Exploiting epsilon greedy bandit segments: [17692:6200:0209/003341.947:INFO:notification_helper_win.cc(171)] Focus Assist is disabled [17692:6200:0209/003341.954:INFO:notification_helper_win.cc(208)] Notifications are enabled [17692:6200:0209/003341.961:VERBOSE1:ad_notification_serving.cc(160)] No segments to serve targeted ads [17692:6200:0209/003341.961:VERBOSE1:ad_notification_serving.cc(226)] Serve untargeted ad [17692:6200:0209/003341.967:VERBOSE2:ad_notification_serving.cc(273)] 3 eligible ads before pacing [17692:6200:0209/003341.967:VERBOSE2:eligible_ads_priority_filter.cc(81)] 3 eligible ads with a priority of 1 [17692:6200:0209/003341.968:VERBOSE2:ad_notification_serving.cc(278)] 3 eligible ads after pacing [17692:6200:0209/003341.968:VERBOSE1:ad_notification_serving.cc(264)] Found 3 eligible ads [17692:6200:0209/003341.969:VERBOSE9:client.cc(455)] Saving client state [17692:6200:0209/003341.971:VERBOSE9:ad_notifications.cc(390)] Saving ad notifications state [17692:6200:0209/003341.972:VERBOSE1:ads_service_impl.cc(1769)] Timeout ad notification with uuid 62c10745-36b3-44b5-b5e1-9f871f425be8 in 120 seconds [17692:6200:0209/003341.981:VERBOSE9:client.cc(455)] Saving client state [17692:6200:0209/003341.981:VERBOSE1:ad_notification_serving.cc(101)] Ad notification delivered: uuid: 62c10745-36b3-44b5-b5e1-9f871f425be8 creativeInstanceId: ed6c467a-8643-42eb-a7cd-32e878fa3cd4 creativeSetId: a9f0b49c-3298-4c1b-a8b5-76a71e8d7332 campaignId: 0e70c893-b0c0-4dab-860d-7e65446f0f0f segment: untargeted title: This is a test body: This is a test targetUrl: https://www.brave.com [17692:6200:0209/003341.989:VERBOSE3:ad_notification_event_viewed.cc(24)] Viewed ad notification with uuid 62c10745-36b3-44b5-b5e1-9f871f425be8 and creative instance id ed6c467a-8643-42eb-a7cd-32e878fa3cd4 ``` - Ensured **Epsilon greedy bandit** is processed (despite the model being disabled, that is by design) ``` [17692:6200:0209/003344.077:VERBOSE1:epsilon_greedy_bandit_processor.cc(118)] Epsilon greedy bandit processed BraveAdsAdNotificationEventType::kClicked [17692:6200:0209/003344.082:VERBOSE9:ad_notifications.cc(405)] Successfully saved ad notifications state [17692:6200:0209/003344.083:VERBOSE1:ad_notification_event_clicked.cc(36)] Successfully logged ad notification clicked event ``` - Verified that the model is suggesting segments to the ad serve pipeline `Exploring epsilon greedy bandit segments: ...` or `Exploiting epsilon greedy bandit segments: ...` ``` [17692:6200:0209/003341.936:VERBOSE2:epsilon_greedy_bandit_model.cc(165)] Exploiting epsilon greedy bandit segments: ```

• Verified that the pulls and values data persisted between sessions/restarts of the browser, and were accurately reflected via brave://rewards 7-day History panel.

Brave | 1.20.101 Chromium: 88.0.4324.152 (Official Build) (64-bit)
-- | --
Revision | 6579930fc53b4dc589c042bec9d0a3778326974d-refs/branch-heads/4324@{#2106}
OS | Windows 10 OS Version 2004 (Build 19041.746)

Case 3: Restart after one round of training and make sure state wasn't deleted in user Preferences JSON key epsilon_greedy_bandit_arms

• Verified the following checks in 3.x cases to ensure Case 3 is working as expected
  • Default: pull 0 value 1.0
  • Click on AD: pull 1 value 1.0
  • Dismiss an AD: pull 1 value 0.0
  • Time Out: Values are incorrect - logged https://github.com/brave/brave-browser/issues/14054
  • View 1st Ad and dismiss 2nd Ad - pull value is 2 and value is 0.5
  • Click two Ads of same segment - pull becomes 2 and value is 1.0
  • Dismiss two Ads of same segment - pull becomes 2 and value is 0.0

Case 3.1: Verified default pulls values is ` 0` and value is `1.0` in the Preference file for all the segments

- Verified that the default value `pulls` is set to `0` and `value` is set to `1.0` in the Preference file when Rewards is enabled ``` "epsilon_greedy_bandit_arms": "{\"architecture\":{\"pulls\":0,\"segment\":\"architecture\",\"value\":1.0}, \"arts & entertainment\":{\"pulls\":0,\"segment\":\"arts & entertainment\",\"value\":1.0}, \"automotive\":{\"pulls\":0,\"segment\":\"automotive\",\"value\":1.0}, \"business\":{\"pulls\":0,\"segment\":\"business\",\"value\":1.0}, \"careers\":{\"pulls\":0,\"segment\":\"careers\",\"value\":1.0}, \"cell phones\":{\"pulls\":0,\"segment\":\"cell phones\",\"value\":1.0}, \"crypto\":{\"pulls\":0,\"segment\":\"crypto\",\"value\":1.0}, \"drugs\":{\"pulls\":0,\"segment\":\"drugs\",\"value\":1.0}, \"education\":{\"pulls\":0,\"segment\":\"education\",\"value\":1.0}, \"family & parenting\":{\"pulls\":0,\"segment\": \"family & parenting\",\"value\":1.0}, \"fashion\":{\"pulls\":0,\"segment\":\"fashion\",\"value\":1.0}, \"folklore\":{\"pulls\":0,\"segment\":\"folklore\",\"value\":1.0}, \"food & drink\":{\"pulls\":0,\"segment\":\"food & drink\",\"value\":1.0}, \"health & fitness\":{\"pulls\":0,\"segment\":\"health & fitness\",\"value\":1.0}, \"history\":{\"pulls\":0,\"segment\":\"history\",\"value\":1.0}, \"hobbies & interests\":{\"pulls\":0,\"segment\":\"hobbies & interests\",\"value\":1.0}, \"home\":{\"pulls\":0,\"segment\":\"home\",\"value\":1.0}, \"law\":{\"pulls\":0,\"segment\":\"law\",\"value\":1.0}, \"military\":{\"pulls\":0,\"segment\":\"military\",\"value\":1.0}, \"personal finance\":{\"pulls\":0,\"segment\":\"personal finance\",\"value\":1.0}, \"pets\":{\"pulls\":0,\"segment\":\"pets\",\"value\":1.0}, \"politics\":{\"pulls\":0,\"segment\":\"politics\",\"value\":1.0}, \"real estate\":{\"pulls\":0,\"segment\":\"real estate\",\"value\":1.0}, \"religion\":{\"pulls\":0,\"segment\":\"religion\",\"value\":1.0}, \"science\":{\"pulls\":0,\"segment\":\"science\",\"value\":1.0}, \"society\":{\"pulls\":0,\"segment\":\"society\",\"value\":1.0}, \"sports\":{\"pulls\":0,\"segment\":\"sports\",\"value\":1.0}, \"technology & computing\":{\"pulls\":0,\"segment\":\"technology & computing\",\"value\":1.0}, \"travel\":{\"pulls\":0,\"segment\":\"travel\",\"value\":1.0}, \"weather\":{\"pulls\":0,\"segment\":\"weather\",\"value\":1.0}}", ```

Case 3.2: Clicked on AD and verified `pulls` values becomes `1` and `value` remained `1.0` in the Preference file - military segment

**Clicked on AD:** ``` [16324:4816:0209/141406.890:VERBOSE2:epsilon_greedy_bandit_model.cc(165)] Exploiting epsilon greedy bandit segments: [16324:4816:0209/141406.890:VERBOSE2:epsilon_greedy_bandit_model.cc(167)] military [16324:4816:0209/141406.897:VERBOSE2:epsilon_greedy_bandit_model.cc(167)] arts & entertainment [16324:4816:0209/141406.898:VERBOSE2:epsilon_greedy_bandit_model.cc(167)] business [16324:4816:0209/141406.899:INFO:notification_helper_win.cc(171)] Focus Assist is disabled [16324:4816:0209/141406.911:INFO:notification_helper_win.cc(208)] Notifications are enabled [16324:4816:0209/141406.913:VERBOSE1:ad_notification_serving.cc(165)] Serve ad for segments: [16324:4816:0209/141406.917:VERBOSE1:ad_notification_serving.cc(167)] military [16324:4816:0209/141406.917:VERBOSE1:ad_notification_serving.cc(167)] arts & entertainment [16324:4816:0209/141406.918:VERBOSE1:ad_notification_serving.cc(167)] business [16324:4816:0209/141406.925:VERBOSE2:ad_notification_serving.cc(273)] 17 eligible ads before pacing [16324:4816:0209/141406.925:VERBOSE2:ad_notification_pacing.cc(46)] Pacing ad delivery for creative instance id afa1c8c5-18f8-408a-81ef-d8e424ccd76c [Roll(0.5):0.716005] [16324:4816:0209/141406.925:VERBOSE2:eligible_ads_priority_filter.cc(81)] 16 eligible ads with a priority of 1 [16324:4816:0209/141406.933:VERBOSE2:ad_notification_serving.cc(278)] 16 eligible ads after pacing [16324:4816:0209/141406.933:VERBOSE1:ad_notification_serving.cc(264)] Found 16 eligible ads [16324:4816:0209/141406.934:VERBOSE1:ads_service_impl.cc(1769)] Timeout ad notification with uuid 83327656-fe1c-423e-b1f2-2796e3b6a3dc in 120 seconds [16324:4816:0209/141406.934:VERBOSE1:ad_notification_serving.cc(101)] Ad notification delivered: uuid: 83327656-fe1c-423e-b1f2-2796e3b6a3dc creativeInstanceId: 5a3c30d1-2d4e-4de2-82bb-2bac646e9e23 creativeSetId: 6b869181-7330-4b9c-80b7-d33c50493b22 campaignId: 37d14e05-fa1f-4730-8fd0-fa44a5ea41b3 segment: military title: Gala Games: $1.6m NFT! 🔥🔥 body: Want to find out why we sold an NFT for $1.6m? targetUrl: https://gala.games/gala-games-and-brave-present-mirandus-shops/ [16324:4816:0209/141406.938:VERBOSE3:ad_notification_event_viewed.cc(24)] Viewed ad notification with uuid 83327656-fe1c-423e-b1f2-2796e3b6a3dc and creative instance id 5a3c30d1-2d4e-4de2-82bb-2bac646e9e23 [16324:4816:0209/141406.944:VERBOSE1:confirmations.cc(90)] Confirming view ad for creative instance id 5a3c30d1-2d4e-4de2-82bb-2bac646e9e23 ``` - Ensured that `pulls` value changed to `1` and `value` remained `1.0` for click on AD 

Case 3.3: Clicked on AD and verified `pulls` value becomes `1` and `value` remained `1.0` in the Preference file - hobbies and interests segment

**Clicked on AD:** ``` [11668:4828:0209/143039.326:VERBOSE2:epsilon_greedy_bandit_model.cc(165)] Exploiting epsilon greedy bandit segments: [11668:4828:0209/143039.327:VERBOSE2:epsilon_greedy_bandit_model.cc(167)] technology & computing [11668:4828:0209/143039.327:VERBOSE2:epsilon_greedy_bandit_model.cc(167)] science [11668:4828:0209/143039.337:VERBOSE2:epsilon_greedy_bandit_model.cc(167)] hobbies & interests [11668:4828:0209/143039.339:INFO:notification_helper_win.cc(171)] Focus Assist is disabled [11668:4828:0209/143039.345:INFO:notification_helper_win.cc(208)] Notifications are enabled [11668:4828:0209/143039.346:VERBOSE1:ad_notification_serving.cc(165)] Serve ad for segments: [11668:4828:0209/143039.346:VERBOSE1:ad_notification_serving.cc(167)] technology & computing [11668:4828:0209/143039.353:VERBOSE1:ad_notification_serving.cc(167)] science [11668:4828:0209/143039.356:VERBOSE1:ad_notification_serving.cc(167)] hobbies & interests [11668:4828:0209/143039.386:VERBOSE2:ad_notification_serving.cc(273)] 28 eligible ads before pacing [11668:4828:0209/143039.386:VERBOSE2:ad_notification_pacing.cc(46)] Pacing ad delivery for creative instance id 98a93e60-6ede-44f7-85b7-51d6994f5f4d [Roll(0.5):0.608456] [11668:4828:0209/143039.387:VERBOSE2:eligible_ads_priority_filter.cc(81)] 27 eligible ads with a priority of 1 [11668:4828:0209/143039.394:VERBOSE2:ad_notification_serving.cc(278)] 27 eligible ads after pacing [11668:4828:0209/143039.396:VERBOSE1:ad_notification_serving.cc(264)] Found 27 eligible ads [11668:4828:0209/143039.397:VERBOSE1:ads_service_impl.cc(1769)] Timeout ad notification with uuid f00da124-73c8-4acd-8cbc-bfe191a5b080 in 120 seconds [11668:4828:0209/143039.405:VERBOSE1:ad_notification_serving.cc(101)] Ad notification delivered: uuid: f00da124-73c8-4acd-8cbc-bfe191a5b080 creativeInstanceId: fc702517-2158-46ca-ae19-9c459b0be723 creativeSetId: 6b869181-7330-4b9c-80b7-d33c50493b22 campaignId: 37d14e05-fa1f-4730-8fd0-fa44a5ea41b3 segment: hobbies & interests title: Gala Games: Join us & use BAT! body: Buy NFTs for the upcoming MMORPG Mirandus with BAT! targetUrl: https://gala.games/gala-games-and-brave-present-mirandus-shops/ [11668:4828:0209/143039.405:VERBOSE3:ad_notification_event_viewed.cc(24)] Viewed ad notification with uuid f00da124-73c8-4acd-8cbc-bfe191a5b080 and creative instance id fc702517-2158-46ca-ae19-9c459b0be723 [11668:4828:0209/143039.406:VERBOSE1:confirmations.cc(90)] Confirming view ad for creative instance id fc702517-2158-46ca-ae19-9c459b0be723 ``` - Ensured that `pulls` value changed to `1` and `value` remained `1.0` for click on AD 

Case 3.4: Dismissed an AD and verified that `pulls` value becomes `1` and `value` changed to `0.0` in the Preference file - personal finance segment

**Dismissed an AD:** ``` [7108:7452:0209/144509.554:VERBOSE2:epsilon_greedy_bandit_model.cc(165)] Exploiting epsilon greedy bandit segments: [7108:7452:0209/144509.555:VERBOSE2:epsilon_greedy_bandit_model.cc(167)] business [7108:7452:0209/144509.556:VERBOSE2:epsilon_greedy_bandit_model.cc(167)] personal finance [7108:7452:0209/144509.567:VERBOSE2:epsilon_greedy_bandit_model.cc(167)] military [7108:7452:0209/144509.571:INFO:notification_helper_win.cc(171)] Focus Assist is disabled [7108:7452:0209/144509.578:INFO:notification_helper_win.cc(208)] Notifications are enabled [7108:7452:0209/144509.582:VERBOSE1:ad_notification_serving.cc(165)] Serve ad for segments: [7108:7452:0209/144509.582:VERBOSE1:ad_notification_serving.cc(167)] business [7108:7452:0209/144509.582:VERBOSE1:ad_notification_serving.cc(167)] personal finance [7108:7452:0209/144509.583:VERBOSE1:ad_notification_serving.cc(167)] military [7108:7452:0209/144509.592:VERBOSE2:ad_notification_serving.cc(273)] 24 eligible ads before pacing [7108:7452:0209/144509.593:VERBOSE2:ad_notification_pacing.cc(46)] Pacing ad delivery for creative instance id 98a93e60-6ede-44f7-85b7-51d6994f5f4d [Roll(0.5):0.83242] [7108:7452:0209/144509.595:VERBOSE2:ad_notification_pacing.cc(46)] Pacing ad delivery for creative instance id afa1c8c5-18f8-408a-81ef-d8e424ccd76c [Roll(0.5):0.910722] [7108:7452:0209/144509.595:VERBOSE2:eligible_ads_priority_filter.cc(81)] 22 eligible ads with a priority of 1 [7108:7452:0209/144509.599:VERBOSE2:ad_notification_serving.cc(278)] 22 eligible ads after pacing [7108:7452:0209/144509.599:VERBOSE1:ad_notification_serving.cc(264)] Found 22 eligible ads [7108:7452:0209/144509.600:VERBOSE1:ads_service_impl.cc(1769)] Timeout ad notification with uuid e23dfbf7-22aa-4fdd-8229-2f9cf4c79f28 in 120 seconds [7108:7452:0209/144509.601:VERBOSE1:ad_notification_serving.cc(101)] Ad notification delivered: uuid: e23dfbf7-22aa-4fdd-8229-2f9cf4c79f28 creativeInstanceId: 979735a3-f779-49a2-aa91-59f06f83c1a2 creativeSetId: 6b869181-7330-4b9c-80b7-d33c50493b22 campaignId: 37d14e05-fa1f-4730-8fd0-fa44a5ea41b3 segment: personal finance title: Gala Games: Be the First! 🔥🔥 body: We are spawn-camping old-school gaming! Join us! targetUrl: https://gala.games/gala-games-and-brave-present-mirandus-shops/ [7108:7452:0209/144509.604:VERBOSE3:ad_notification_event_viewed.cc(24)] Viewed ad notification with uuid e23dfbf7-22aa-4fdd-8229-2f9cf4c79f28 and creative instance id 979735a3-f779-49a2-aa91-59f06f83c1a2 [7108:7452:0209/144509.611:VERBOSE1:confirmations.cc(90)] Confirming view ad for creative instance id 979735a3-f779-49a2-aa91-59f06f83c1a2 ``` - Ensured that `pulls` value changed to `1` and `value` changed to `0.0` for dismissing an AD 

Case 3.5: Viewed two Ads of same segment/category, click the 1st Ad and dismiss the 2nd AD

 **Viewed 1st Ad from Technology & Computing segment** ``` [7804:7348:0209/180827.447:VERBOSE2:epsilon_greedy_bandit_model.cc(149)] Exploring epsilon greedy bandit segments: [7804:7348:0209/180827.447:VERBOSE2:epsilon_greedy_bandit_model.cc(151)] technology & computing [7804:7348:0209/180827.457:VERBOSE2:epsilon_greedy_bandit_model.cc(151)] [7804:7348:0209/180827.458:VERBOSE2:epsilon_greedy_bandit_model.cc(151)] [7804:7348:0209/180827.467:INFO:notification_helper_win.cc(171)] Focus Assist is disabled [7804:7348:0209/180827.475:INFO:notification_helper_win.cc(208)] Notifications are enabled [7804:7348:0209/180827.478:VERBOSE1:ad_notification_serving.cc(165)] Serve ad for segments: [7804:7348:0209/180827.478:VERBOSE1:ad_notification_serving.cc(167)] technology & computing [7804:7348:0209/180827.486:VERBOSE1:ad_notification_serving.cc(167)] [7804:7348:0209/180827.495:VERBOSE1:ad_notification_serving.cc(167)] [7804:7348:0209/180827.498:VERBOSE2:ad_notification_serving.cc(273)] 2 eligible ads before pacing [7804:7348:0209/180827.506:VERBOSE2:eligible_ads_priority_filter.cc(81)] 2 eligible ads with a priority of 1 [7804:7348:0209/180827.507:VERBOSE2:ad_notification_serving.cc(278)] 2 eligible ads after pacing [7804:7348:0209/180827.507:VERBOSE1:ad_notification_serving.cc(264)] Found 2 eligible ads [7804:7348:0209/180827.517:VERBOSE1:ads_service_impl.cc(1769)] Timeout ad notification with uuid 4e956a71-b472-4873-a67f-4fbbd1fc3280 in 120 seconds [7804:7348:0209/180827.518:VERBOSE1:ad_notification_serving.cc(101)] Ad notification delivered: uuid: 4e956a71-b472-4873-a67f-4fbbd1fc3280 creativeInstanceId: 30ecd70f-f56d-4745-a457-4035fcab54b6 creativeSetId: 0a697a3b-233a-44bc-a337-e2441fb277a7 campaignId: d4d0474c-277a-4714-b25d-7b484cdb580a segment: technology & computing title: This is a test 2 body: This is a test 2 targetUrl: https://www.brave.com [7804:7348:0209/180827.522:VERBOSE3:ad_notification_event_viewed.cc(24)] Viewed ad notification with uuid 4e956a71-b472-4873-a67f-4fbbd1fc3280 and creative instance id 30ecd70f-f56d-4745-a457-4035fcab54b6 [7804:7348:0209/180827.529:VERBOSE1:confirmations.cc(90)] Confirming view ad for creative instance id 30ecd70f-f56d-4745-a457-4035fcab54b6 [7804:7348:0209/180827.530:VERBOSE1:redeem_unblinded_token.cc(49)] Redeem unblinded token ``` - Ensured `pull` value is `1` and `value` is `1.0`  **Dismiss 2nd Ad from the same segment** ``` [7804:7348:0209/182103.646:VERBOSE2:epsilon_greedy_bandit_model.cc(165)] Exploiting epsilon greedy bandit segments: [7804:7348:0209/182103.646:VERBOSE2:epsilon_greedy_bandit_model.cc(167)] technology & computing [7804:7348:0209/182103.655:INFO:notification_helper_win.cc(171)] Focus Assist is disabled [7804:7348:0209/182103.661:INFO:notification_helper_win.cc(208)] Notifications are enabled [7804:7348:0209/182103.663:VERBOSE1:ad_notification_serving.cc(165)] Serve ad for segments: [7804:7348:0209/182103.663:VERBOSE1:ad_notification_serving.cc(167)] technology & computing [7804:7348:0209/182103.670:VERBOSE1:eligible_ad_notifications.cc(70)] All advertisers have been shown, so round robin [7804:7348:0209/182103.670:VERBOSE1:client.cc(385)] Resetting seen advertisers [7804:7348:0209/182103.671:VERBOSE2:ad_notification_serving.cc(273)] 1 eligible ads before pacing [7804:7348:0209/182103.671:VERBOSE2:eligible_ads_priority_filter.cc(81)] 1 eligible ads with a priority of 1 [7804:7348:0209/182103.672:VERBOSE2:ad_notification_serving.cc(278)] 1 eligible ads after pacing [7804:7348:0209/182103.674:VERBOSE1:ad_notification_serving.cc(264)] Found 1 eligible ads [7804:7348:0209/182103.674:VERBOSE1:ads_service_impl.cc(1769)] Timeout ad notification with uuid 89924e1e-ff36-4b59-abc8-3f7e2e9071fc in 120 seconds [7804:7348:0209/182103.675:VERBOSE1:ad_notification_serving.cc(101)] Ad notification delivered: uuid: 89924e1e-ff36-4b59-abc8-3f7e2e9071fc creativeInstanceId: ed6c467a-8643-42eb-a7cd-32e878fa3cd4 creativeSetId: a9f0b49c-3298-4c1b-a8b5-76a71e8d7332 campaignId: 0e70c893-b0c0-4dab-860d-7e65446f0f0f segment: technology & computing title: This is a test 1 body: This is a test 1 targetUrl: https://www.brave.com [7804:7348:0209/182103.675:VERBOSE3:ad_notification_event_viewed.cc(24)] Viewed ad notification with uuid 89924e1e-ff36-4b59-abc8-3f7e2e9071fc and creative instance id ed6c467a-8643-42eb-a7cd-32e878fa3cd4 [7804:7348:0209/182103.676:VERBOSE1:confirmations.cc(90)] Confirming view ad for creative instance id ed6c467a-8643-42eb-a7cd-32e878fa3cd4 ``` - Ensured pull value is `2` and value becomes `0.5` 

Case 3.6: View an AD don't take any action on the notification wait until AD gets time out

**View an AD don't take any action on the Ads notification:** ``` [14444:3632:0209/170103.653:VERBOSE2:epsilon_greedy_bandit_model.cc(165)] Exploiting epsilon greedy bandit segments: [14444:3632:0209/170103.653:VERBOSE2:epsilon_greedy_bandit_model.cc(167)] folklore [14444:3632:0209/170103.661:VERBOSE2:epsilon_greedy_bandit_model.cc(167)] family & parenting [14444:3632:0209/170103.661:VERBOSE2:epsilon_greedy_bandit_model.cc(167)] personal finance [14444:3632:0209/170103.663:INFO:notification_helper_win.cc(171)] Focus Assist is disabled [14444:3632:0209/170103.668:INFO:notification_helper_win.cc(208)] Notifications are enabled [14444:3632:0209/170103.670:VERBOSE1:ad_notification_serving.cc(165)] Serve ad for segments: [14444:3632:0209/170103.670:VERBOSE1:ad_notification_serving.cc(167)] folklore [14444:3632:0209/170103.677:VERBOSE1:ad_notification_serving.cc(167)] family & parenting [14444:3632:0209/170103.677:VERBOSE1:ad_notification_serving.cc(167)] personal finance [14444:3632:0209/170103.686:VERBOSE2:ad_notification_serving.cc(273)] 23 eligible ads before pacing [14444:3632:0209/170103.686:VERBOSE2:eligible_ads_priority_filter.cc(81)] 23 eligible ads with a priority of 1 [14444:3632:0209/170103.691:VERBOSE2:ad_notification_serving.cc(278)] 23 eligible ads after pacing [14444:3632:0209/170103.692:VERBOSE1:ad_notification_serving.cc(264)] Found 23 eligible ads [14444:3632:0209/170103.695:VERBOSE1:ads_service_impl.cc(1769)] Timeout ad notification with uuid b8923d20-1d1b-4c56-9f94-58a11270d2c1 in 120 seconds [14444:3632:0209/170103.696:VERBOSE1:ad_notification_serving.cc(101)] Ad notification delivered: uuid: b8923d20-1d1b-4c56-9f94-58a11270d2c1 creativeInstanceId: e7706405-3ffb-48e9-8864-00aa3bf3dfbb creativeSetId: 6b869181-7330-4b9c-80b7-d33c50493b22 campaignId: 37d14e05-fa1f-4730-8fd0-fa44a5ea41b3 segment: folklore title: Gala Games: Revolution! body: Tired of the same repetitive gaming experiences? Join us! targetUrl: https://gala.games/gala-games-and-brave-present-mirandus-shops/ [14444:3632:0209/170103.704:VERBOSE3:ad_notification_event_viewed.cc(24)] Viewed ad notification with uuid b8923d20-1d1b-4c56-9f94-58a11270d2c1 and creative instance id e7706405-3ffb-48e9-8864-00aa3bf3dfbb [14444:3632:0209/170103.707:VERBOSE1:confirmations.cc(90)] Confirming view ad for creative instance id e7706405-3ffb-48e9-8864-00aa3bf3dfbb ``` - `pulls` value and `value` remained unchanged ( same as default value i.e `pull` 0 `value` 1.0) - logged https://github.com/brave/brave-browser/issues/14054 

Case 3.7: Click two Ads of same segment

 - Ensured that the value for pull becomes `2` and value is `1.0` 

Case 3.8: Dismiss two Ads of same segment

 - Ensured that the value for pull becomes `2` and value is `0.0` 

[srirambv]

Verification passed on OnePlus 6T with Android 10 running 1.20.102 x64 build

Case 1: Default models enabled (Text Classification and Purchase Intent)

Used --args --enable-logging=stderr --vmodule="*/bat-native-ledger/*"=6,"*/brave_rewards/*"=6,"*/bat-native-ads/*"=6,"*/bat-native-confirmations/*"=6,"*/brave_ads/*"=6,"*/brave_user_model/*"=6 --brave-ads-staging --brave-ads-debug --rewards=staging=true,reconcile-interval=5

• Verified Text Classification and Purchase Intent models are enabled and Epsilon Greedy Bandit is disabled
• Triggered ad notification and verified that ads are served as usual via text classification
• Verify that bandit is processing (despite the model being disabled, that is by design)

Verified an AD using the search keyword `coinbase earn` and ensured purchase intent for Keywords are working as expected

 ``` 2021-02-09 16:28:40.793 20644-20644/? V/chromium: [VERBOSE1:purchase_intent_processor.cc(101)] Extracted purchase intent signal from visited URL 2021-02-09 16:28:40.870 20644-20644/? V/chromium: [VERBOSE1:text_classification_processor.cc(62)] Classified text with the top segment as sports-sports 2021-02-09 16:28:40.900 20644-20644/? V/chromium: [VERBOSE1:conversions.cc(151)] No conversions found for visited URL 2021-02-09 16:29:04.992 1305-1408/? I/OHPD: isIsolated process: app.uid:99495,app.info.uid:11227, app.processName:com.brave.browser:sandboxed_process1:org.chromium.content.app.SandboxedProcessService1:15 2021-02-09 16:29:04.993 1305-1408/? I/OHPD: isIsolated process: app.uid:99481,app.info.uid:11227, app.processName:com.brave.browser:sandboxed_process1:org.chromium.content.app.SandboxedProcessService1:8 2021-02-09 16:29:04.994 1305-1408/? I/OHPD: isIsolated process: app.uid:99479,app.info.uid:11227, app.processName:com.brave.browser:sandboxed_process1:org.chromium.content.app.SandboxedProcessService1:7 2021-02-09 16:29:05.689 1305-1408/? I/OHPD: isIsolated process: app.uid:99495,app.info.uid:11227, app.processName:com.brave.browser:sandboxed_process1:org.chromium.content.app.SandboxedProcessService1:15 2021-02-09 16:29:05.690 1305-1408/? I/OHPD: isIsolated process: app.uid:99481,app.info.uid:11227, app.processName:com.brave.browser:sandboxed_process1:org.chromium.content.app.SandboxedProcessService1:8 2021-02-09 16:29:05.691 1305-1408/? I/OHPD: isIsolated process: app.uid:99479,app.info.uid:11227, app.processName:com.brave.browser:sandboxed_process1:org.chromium.content.app.SandboxedProcessService1:7 2021-02-09 16:29:17.872 20644-20644/? V/chromium: [VERBOSE1:subdivision_targeting.cc(168)] Fetch subdivision target 2021-02-09 16:29:17.872 20644-20644/? V/chromium: [VERBOSE2:subdivision_targeting.cc(169)] GET /v1/getstate 2021-02-09 16:29:17.873 20644-20644/? V/chromium: [VERBOSE5:subdivision_targeting.cc(173)] URL Request: URL: https://ads-serve.bravesoftware.com/v1/getstate Method: BraveAdsUrlRequestMethod::GET 2021-02-09 16:29:17.910 20644-20644/? V/chromium: [VERBOSE1:subdivision_targeting.cc(189)] Successfully fetched subdivision target 2021-02-09 16:29:17.920 20644-20644/? V/chromium: [VERBOSE1:subdivision_targeting.cc(267)] Fetch ads subdivision target in 0 hours, 3 minutes, 54 seconds at 4:33 PM 2021-02-09 16:29:26.019 20644-20644/? V/chromium: [VERBOSE1:ad_notification_serving.cc(165)] Serve ad for segments: 2021-02-09 16:29:26.019 20644-20644/? V/chromium: [VERBOSE1:ad_notification_serving.cc(167)] sports-sports 2021-02-09 16:29:26.019 20644-20644/? V/chromium: [VERBOSE1:ad_notification_serving.cc(167)] hobbies & interests-coins 2021-02-09 16:29:26.019 20644-20644/? V/chromium: [VERBOSE1:ad_notification_serving.cc(167)] education-education 2021-02-09 16:29:26.019 20644-20644/? V/chromium: [VERBOSE1:ad_notification_serving.cc(167)] Crypto Purchase Intent 2021-02-09 16:29:26.023 20644-20644/? V/chromium: [VERBOSE1:ad_notification_serving.cc(182)] No eligible ads found for segments 2021-02-09 16:29:26.023 20644-20644/? V/chromium: [VERBOSE1:ad_notification_serving.cc(197)] Serve ad for parent segments: 2021-02-09 16:29:26.023 20644-20644/? V/chromium: [VERBOSE1:ad_notification_serving.cc(199)] sports 2021-02-09 16:29:26.023 20644-20644/? V/chromium: [VERBOSE1:ad_notification_serving.cc(199)] hobbies & interests 2021-02-09 16:29:26.024 20644-20644/? V/chromium: [VERBOSE1:ad_notification_serving.cc(199)] education 2021-02-09 16:29:26.024 20644-20644/? V/chromium: [VERBOSE1:ad_notification_serving.cc(199)] Crypto Purchase Intent 2021-02-09 16:29:26.028 20644-20644/? V/chromium: [VERBOSE1:ad_notification_serving.cc(214)] No eligible ads found for parent segments 2021-02-09 16:29:26.028 20644-20644/? V/chromium: [VERBOSE1:ad_notification_serving.cc(226)] Serve untargeted ad 2021-02-09 16:29:26.033 20644-20644/? V/chromium: [VERBOSE1:eligible_ad_notifications.cc(70)] All advertisers have been shown, so round robin 2021-02-09 16:29:26.034 20644-20644/? V/chromium: [VERBOSE1:client.cc(385)] Resetting seen advertisers 2021-02-09 16:29:26.042 20644-20644/? V/chromium: [VERBOSE2:ad_notification_serving.cc(273)] 2 eligible ads before pacing 2021-02-09 16:29:26.042 20644-20644/? V/chromium: [VERBOSE2:eligible_ads_priority_filter.cc(81)] 2 eligible ads with a priority of 1 2021-02-09 16:29:26.043 20644-20644/? V/chromium: [VERBOSE2:ad_notification_serving.cc(278)] 2 eligible ads after pacing 2021-02-09 16:29:26.043 20644-20644/? V/chromium: [VERBOSE1:ad_notification_serving.cc(264)] Found 2 eligible ads 2021-02-09 16:29:26.065 20644-20644/? V/chromium: [VERBOSE1:ad_notification_serving.cc(101)] Ad notification delivered: uuid: c6d6cceb-c645-4e6e-a522-b6eb017d55da creativeInstanceId: 30ecd70f-f56d-4745-a457-4035fcab54b6 creativeSetId: 0a697a3b-233a-44bc-a337-e2441fb277a7 campaignId: d4d0474c-277a-4714-b25d-7b484cdb580a segment: untargeted title: This is a test body: This is a test targetUrl: https://www.brave.com 2021-02-09 16:29:26.085 20644-20644/? V/chromium: [VERBOSE3:ad_notification_event_viewed.cc(24)] Viewed ad notification with uuid c6d6cceb-c645-4e6e-a522-b6eb017d55da and creative instance id 30ecd70f-f56d-4745-a457-4035fcab54b6 2021-02-09 16:29:26.086 20644-20644/? V/chromium: [VERBOSE1:confirmations.cc(90)] Confirming view ad for creative instance id 30ecd70f-f56d-4745-a457-4035fcab54b6 ```

• Verified model is not suggesting segments to the ad serve pipeline and didn't see Exploring epsilon greedy bandit segments: ... or Exploiting epsilon greedy bandit segments: ...

Case 2: Variant models enabled

• Verified that Purchase Intent models and Epsilon Greedy Bandit are enabled and Text Classification is disabled when the brave is launched using cmd line: --enable-logging=stderr --vmodule="*/bat-native-ads/*"=9,"*/brave_ads/*"=9,"*/brave_user_model/*"=9,"*/bat_ads/*"=9 --brave-ads-staging --rewards=staging=true --disable-features="TextClassification" --enable-features="EpsilonGreedyBandit"
• Viewed an AD using the search keyword best bitcoin lending sites and ensured purchase intent for Keywords are working as expected
• Verified that bandit is processing
• Verified model is suggesting segments to the ad serve pipeline and didn't see Exploring epsilon greedy bandit segments: ... or Exploiting epsilon greedy bandit segments: ...