search

mozilla / blurts-addon

Mozilla Public License 2.0
5 stars 8 forks source link

Unable to set study variant from CLI using VARIATION_NAME env var #88

Closed pdehaan closed 6 years ago

pdehaan commented 6 years ago

Steps to reproduce:

  1. Launch Firefox w/ Firefox Monitor add-on using $ VARIATION_NAME=3 npm start -- --firefox=nightly
  2. Open about:config and search for "fxmonitor". Note that the extensions.fxmonitor_shield_mozilla_org.variationName config variable is correctly set to 3 (integer).
  3. Open "Tools > Web Developer > Browser Console" and search for "getVariation"

Actual results:

Even though I set the VARIATION_NAME to 3 via the CLI and confirmed in about:config, the Browser Console reports the variation as "2".

about-config

shield-study-utils: getVariation: {"name":"2","weight":1} api.js:3739

browser_console_fx

https://github.com/mozilla/blurts-addon/blob/bab629bcee9b386141646a9b12ed108c64aea716/web-ext-config.js#L20-L22

$ VARIATION_NAME=3 npm start -- --firefox=nightly

> blurts-addon@1.0.0 prestart /Users/pdehaan/dev/github/mozilla/blurts-addon
> npm run bundle-utils

> blurts-addon@1.0.0 bundle-utils /Users/pdehaan/dev/github/mozilla/blurts-addon
> copyStudyUtils src/privileged

> blurts-addon@1.0.0 start /Users/pdehaan/dev/github/mozilla/blurts-addon
> web-ext run --no-reload "--firefox=nightly"

Applying config files: ./package.json, ./web-ext-config.js
Running web extension from /Users/pdehaan/dev/github/mozilla/blurts-addon/src
Setting custom Firefox preferences: {
  "shieldStudy.logLevel": "All",
  "extensions.fxmonitor_shield_mozilla_org.variationName": 3
}
Use --verbose or open Tools > Web Developer > Browser Console to see logging
Installed /Users/pdehaan/dev/github/mozilla/blurts-addon/src as a temporary add-on
Automatic extension reloading has been disabled
nhnt11 commented 6 years ago

Interesting point. The purpose of that env var was to allow easy testing, so I didn't worry too much about doing it "correctly". The pref is used to OVERRIDE the variation that shield assigns, rather than compelling shield to assign a specific variation. Now that I think about it though, we should fix this in case someone "misuses" this env var for non-testing purposes, so that telemetry stays meaningful (after https://github.com/mozilla/blurts-addon/commit/c347c583bf5f6154960313f129d6dd9f0b0ff52d)

nhnt11 commented 6 years ago

No longer relevant :)