[Task]: Implement updated stashing strategy to stash soft as well as hard blocks

KevinMind commented 2 months ago

Description

After we decide how to stash soft/hard blocks we must apply these changes to our stashing generation logic to upload new soft add/remove entries.

Acceptance Criteria

  ### Milestones/checkpoints
  - [ ] existing commands/cron job account for soft/hard block updates
  - [ ] FX is aware and aligned on this updated data structure

Checks

[X] If I have identified that the work is specific to a repository, I have removed "repository:addons-server" or "repository:addons-frontend"

┆Issue is synchronized with this Jira Task

diox commented 2 months ago

https://mozilla-hub.atlassian.net/browse/AMOENG-1093?focusedCommentId=940513 documents the format of the stashes.

ioanarusiczki commented 2 weeks ago

https://remote-settings-dev.allizom.org/v1/buckets/blocklists/collections/addons-bloomfilters/records

right now:

dev stash

Blocked/restricted guids -> those with ❌ are not in stash but I did block them late before 3 pm (could be that , so I'll re-check on Monday those). Also I've unblocked a few but they're not in stash , same problem, perhaps it was late in the afternoon.

https://reviewers.addons-dev.allizom.org/en-US/reviewers/review/634128 ✅ https://addons-internal.dev.mozaws.net/en-US/admin/models/blocklist/block/26508/change/ shows up in stash correctly ✅ https://reviewers.addons-dev.allizom.org/en-US/reviewers/review-listed/634127 ✅ https://addons-internal.dev.mozaws.net/en-US/admin/models/blocklist/block/26509/change/ ✅ hard block on https://addons-internal.dev.mozaws.net/en-US/admin/models/blocklist/blocklistsubmission/807/change/ 5 guids ❌ soft block https://addons-internal.dev.mozaws.net/en-US/admin/models/blocklist/blocklistsubmission/808/change/ 7 guids ❌

deleted blocks with multiple blocks https://addons-internal.dev.mozaws.net/en-US/admin/models/blocklist/blocklistsubmission/809/change/ hard block {f5257268-e035-46ee-9e12-3414e62a3682} ❌ soft block {71edc960-6cce-4dbe-be16-f79dd4f50b92} ❌ soft block {eea536e1-5a7b-4e1b-b4f4-7eabfdfe6b53} ❌ hard block FirefoxColor@mozilla.com version 10.0.0 ❌

Restricted and installed: these look good

https://addons-dev.allizom.org/en-US/firefox/addon/good-vibrations_test/ ✅ https://addons-dev.allizom.org/en-US/firefox/addon/restricted-soft-block-test1/ ✅ https://addons-dev.allizom.org/en-US/developers/addon/restricted-soft-block-test2/edit ✅

In the browser

I see them restricted

after enabling them I've noticed a URL that redirects to prod (I might be missing a pref, I've tried to modify this that for dev but I've still the prod url instead of the correct env.) @willdurand perhaps halp here ...

url points to prod zip

ioanarusiczki commented 1 week ago

Good news 😊

all guids blocked, restricted or unblocked showed up. Unblocked guids (soft and hard blocks) are together into unblocked [...]

I'll try to do more checks over this week, and should also be verified for stage reason why I'm not marking verified yet.

soft and hard blocks deleted are together

I've added https://services.addons-dev.allizom.org/api/v4/addons/addon/ to extensions.abuseReport.amoDetailsURL and I've a correct redirect to dev.

correct url zip

willdurand commented 1 week ago

after enabling them I've noticed a URL that redirects to prod (I might be missing a pref, I've tried to modify this that for dev but I've still the prod url instead of the correct env.) @willdurand perhaps halp here ...

The pref is: extensions.blocklist.addonItemURL

willdurand commented 1 week ago

I've added services.addons-dev.allizom.org/api/v4/addons/addon to extensions.abuseReport.amoDetailsURL and I've a correct redirect to dev.

This pref (extensions.abuseReport.amoDetailsURL) doesn't exist AFAICT.

ioanarusiczki commented 1 week ago

@willdurand Looks like I've modified extensions.blocklist.addonItemURL too on Friday and kept having the prod url , this morning it worked as expected 🤷‍♀️
And extensions.abuseReport.amoDetailsURL is no longer in the list indeed, it's been used before abuse reporting was modified.

ioanarusiczki commented 3 days ago

I verified AMO stage with enable-soft-blocking disabled.

blocked multiple guids , were blocked in the browser (I've installed 10 out of 31) https://addons-internal.stage.mozaws.net/en-US/admin/models/blocklist/blocklistsubmission/485/change/
deleted multiple guids, those installed in Nighltly were unblocked https://addons-internal.stage.mozaws.net/en-US/admin/models/blocklist/blocklistsubmission/486/change/
a version with dual sign off blocked then unblocked https://reviewers.addons.allizom.org/en-US/reviewers/review/1004682 version 1.4.1.7
a theme and a dictionary blocked https://reviewers.addons.allizom.org/en-US/reviewers/review-listed/2243482#review-actions https://reviewers.addons.allizom.org/en-US/reviewers/review/2243481
addons blocked with delay: after generating mlbf they're still available in the browser which is expected since the delay did not expire https://addons-internal.stage.mozaws.net/en-US/admin/models/blocklist/blocklistsubmission/491/change/

bloomfilters

takes a while for https://firefox.settings.services.allizom.org/v1/buckets/blocklists/collections/addons-bloomfilters/records to update but I also used https://remote-settings.allizom.org/v1/buckets/blocklists/collections/addons-bloomfilters/records which does not seem to be cached.

I had the expected results in [blocked] and [unblocked] [softblocked] is present in the generation of the stash - always empty

ioanarusiczki commented 3 days ago

I also tried on stage with enable-soft-blocking disabled

to create a softblock , generate mlbf. Checking it, no guids in [softblocked] and the version continues to be enabled in about:addons
softened an existing hard block -> this guid {00294368-7d81-43ec-a16a-85c1400c2da4} shows up as [unblocked], checking the browser the version has been enabled (previously blocked).

I hope the latter is expected.

KevinMind commented 3 days ago

Yes, when soft blocking is disabled, softening a hard block is treated as an unblocking. It should be in the unblocked stash. If softblocking is enabled it instead would be in the softblocked stash.

You could verify this by flipping the waffle switch and rerunning the cron.

ioanarusiczki commented 3 days ago

Yeah, I saw that on -dev while testing , I just didn't know how it behaves with switch off.

willdurand commented 3 days ago

[softblocked] is present in the generation of the stash - always empty

Yes, that should be the only obvious difference.

mozilla / addons