combine uses of Addon.objects and Addon.with_unlisted

[eviljeff]

Addon.objects contains all listed, not deleted, add-ons (filter on .is_listed); Addon.with_unlisted contains both listed and unlisted (not deleted) add-ons. With mixed listed/unlisted versions Addon.objects will contain add-ons that contain unlisted versions; Addon.with_unlisted will be obsolete.

Tasks:

• check Addon.objects usage can handle unlisted versions and add extra filter as appropriate
  • [x] src/olympia/users/views.py
  • Uses Addon.objects.valid and Addon.objects.none(), both should be fine.
  • [x] src/olympia/users/models.py
  • False positive, uses CollectionAddon.objects.
  • [x] src/olympia/legacy_api/views.py
  • Has few instances that need to be fixed/double-checked.
  • [x] src/olympia/addons/forms.py
  • Has few instances that need to be fixed/double-checked.
  • [x] src/olympia/addons/management/commands/process_addons.py
  • Tasks in there should be able to deal with unlisted add-ons. Need to remove the --with-unlisted parameter.
  • [x] src/olympia/addons/utils.py
  • reverse_name_lookup need to be patched, which may be non-trivial. reverse_name_lookup is gone. get_featured_ids and get_creatured_ids should filter on status.
  • [x] src/olympia/addons/cron.py
  • _change_last_updated is weird, should be looked at, but it makes sense that it'd affect unlisted add-ons
  • It should be harmless for update_addon_appsupport
  • deliver_hotness should probably filter on status.
  • Rest of the file is simply doing a .get(pk={something}) and should be fine.
  • [x] src/olympia/addons/views.py
  • home is using .listed() which is already filtering on status. It's also using featured, which isn't, but it's less important since it's editorial stuff. Filtering on status there shouldn't hurt either, though. Need to be careful as there is a second home view for mobile.
  • persona_detail is fine since it's Personas and it's using .public()
  • AddonFeaturedView is using .valid() which filters on status already. Maybe it should use something more restrictive like .public() though.
  • Rest of the code is using addon_view_factory() + some querysets that should already filter things out correctly depending on status.
  • [x] src/olympia/addons/models.py
  • As in cron.py above, _last_updated_queries() could be updated but at the same time it might make sense for it to affect unlisted add-ons.
  • Rest of the code seems to be filtering or just getting single instances by id/guid.
  • [x] src/olympia/addons/tasks.py
  • Only used to get specific instances by id
  • [x] src/olympia/addons/decorators.py
  • addon_view decorator has a default qs argument Addon.objects.all but it's never used directly, views always use addon_view_factory and always pass a queryset.
  • [x] src/olympia/reviews/views.py
  • Should be fine, either filtering on status or just getting some add-ons by their id.
  • [x] src/olympia/reviews/feeds.py
  • Just getting an add-on by its id
  • [x] src/olympia/reviews/tasks.py
  • Some filtering by id, but also addon_bayesian_rating does some aggregating over the whole db which sounds suspicious, probably need to filter on status.
  • [x] src/olympia/devhub/forms.py
  • Only one instance and it's using .reviewed().
  • [x] src/olympia/devhub/tasks.py
  • Filtering on some specific ids
  • [x] src/olympia/bandwagon/views.py
  • Filtering on specific ids or with filtering on valid statuses
  • [x] src/olympia/bandwagon/models.py
  • False positives CollectionAddon.
  • [x] src/olympia/bandwagon/feeds.py
  • Uses .valid().
  • [x] src/olympia/bandwagon/tasks.py
  • False positives CollectionAddon.
  • [x] src/olympia/landfill/collection.py
  • False positive, uses CollectionAddon.objects.
  • [x] src/olympia/landfill/generators.py
  • Only uses Addon.objects.create
  • [x] src/olympia/versions/views.py
  • Uses Addon.objects.valid which filters on status already.
  • [x] src/olympia/versions/feeds.py
  • Uses .valid().
  • [x] src/olympia/zadmin/views.py
  • Uses Addon.objects to fetch langpacks and filtering on addonuser__user__email=settings.LANGPACK_OWNER_EMAIL, so that's OK.
  • Another instance is using status__in=amo.VALID_ADDON_STATUSES which is OK as well.
  • ValidationResultAffectedAddon.objects is a false positive
  • addon_search() gets the object by id or uses ES - in any case it should be fine for it to return unlisted add-ons, it's an admin tool.
  • [x] src/olympia/zadmin/models.py
  • False positive ValidationResultAffectedAddon.objects
  • [x] src/olympia/zadmin/tasks.py
  • add_validation_jobs() is filtering on specific pks.
  • notify_compatibility() is... a mess. Need to revisit this later, but it does not expect unlisted add-ons so it might break.
  • [x] src/olympia/stats/management/commands/theme_update_counts_from_file.py
  • Loads... all the add-ons. In a single set. Should at least filter on add-on type to only consider themes, possibly on status as well.
  • [x] src/olympia/stats/management/commands/update_counts_from_file.py
  • Also loads all the add-ons, filters on a couple things but would still need to be fixed. Maybe filter on _current_version existence, if not status.
  • [x] src/olympia/stats/management/commands/download_counts_from_file.py
  • Also loads all the add-ons... and all the files. No filtering in both cases, probably need to filter on _current_version existence, and/or status like the previous one.
  • [x] src/olympia/stats/cron.py
  • addon_total_contributions() is going through all the add-ons in chunks to update their total number of contributions. Should probably filter on _current_version or/and status, or be much smarter and filter on the list of add-ons which had contributions since the last time the cron was run... This cron is not called anywhere, removing it.
  • [x] src/olympia/stats/tasks.py
  • addon_total_contributions() is filtering on id
  • _get_daily_jobs computes the number of new add-ons (addon_count_new) with Addon.objects. We may want to keep it as-is to include purely unlisted add-ons, or filter on _current_version.
  • It also has a couple other querysets that filter on status already.
  • [x] src/olympia/editors/views.py
  • Only uses Addon.objects in `queue_version_notes() to get an add-on by its id.
  • [x] src/olympia/editors/helpers.py
  • get_position() filters on status already.
  • [x] src/olympia/editors/views_themes.py
  • Filters on status and type=amo.ADDON_PERSONA
  • [x] src/olympia/legacy_discovery/views.py
  • Uses Addon.objects.valid.
  • [x] src/olympia/legacy_discovery/modules.py
  • Uses Addon.objects.in_bulk() with specific ids.
  • [x] src/olympia/browse/views.py
  • addon_listing() filters with .public() or through .listed() + amo.REVIEWED_STATUSES.
  • language_tools(), creatured() and personas_listing() filter with .public().
  • category_landing() filters with .listed() (and therefore amo.STATUS_PUBLIC)
  • SearchToolsFilter filters with .valid()
  • search_tools() filters with .listed(), passing amo.STATUS_PUBLIC explicitly.
  • [x] src/olympia/browse/feeds.py
  • Uses Addon.objects.featured(). Like mentioned above it's not a deal breaker because it's editorial stuff but we should add filtering on status in get_featured_ids(). done.
  • [x] src/olympia/discovery/views.py
  • Uses Addon.objects.public().
  • [x] src/olympia/search/views.py
  • Uses Addon.objects.none() and Addon.objects.reviewed().
  • [x] src/olympia/compat/cron.py
  • Filters on specific ids.
• change Addon.with_unlisted to Addon.objects

[diox]

Added a gigantic list of files to check.

Random notes from IRC discussion with @eviljeff :

• Because pure unlisted add-ons will have STATUS_NULL status, most of the public-facing code should be relatively safe.
• It might be worth trying to remove the filtering and see what tests break (though we might only find very specific managers / models tests)
• One alternate approach could be to switch .objects to with_unlisted as we go along and fix code, and when everything is switched to with_unlisted, remove the filtering from objects manager and switch the code back to using .objects.

[diox]

All usage of Addon.objects should have been fixed now, so we can start working on changing the managers. I'll do that.