Add-on indexing (or other task) sometimes happen too early

[diox]

We can sometimes end up indexing add-ons too early before the transaction has been committed. I believe this is what's been happening in those cases:

https://github.com/mozilla/addons/issues/4127 https://github.com/mozilla/addons/issues/4415

We developed post-request-task to solve this, but never deployed it on AMO (it was deployed on Marketplace however) because of issues with validation never completing when it was enabled. In addition, we had at least one instance where we fired up a celery task synchronously, which was broken with post-request-task.

We need to find a solution for this to stop incoherent data like mentioned in the 2 issues above.

Note that recent django versions will bring on_commit but this will just allow us to write post-request-task in a cleaner way, the same problems we had with it will likely remain. Plus, we're not on the right django version yet anyway.

[EnTeQuAk]

Assigning to me to look into this in the coming weeks as part of my attempt to work on django 1.11 and potentially celery 4 upgrade.

[ValentinaPC]

Adding some examples of add-ons that no longer exist, but are still displayed in AMO public pages:

• Webext-+beta - https://addons-dev.allizom.org/en-US/firefox/addon/webext-beta/
• - https://addons-dev.allizom.org/en-US/firefox/addon/scriptalertproblem1script/
• Warrior Forum Formatter - https://addons-dev.allizom.org/en-US/firefox/addon/warrior-forum-formatter/
• Webext-to many reasons - https://addons-dev.allizom.org/en-US/firefox/addon/webext-to-many-reasons/

Also if we click the "View classic desktop site" from the Up & Coming page we will see first 3 incomplete add-ons in old pages too, clicking on them will redirect to a black search results page:

[muffinresearch]

Since this is more visible on the new frontend I'm going to move it up to p2.

[EnTeQuAk]

QA: Can you do additional testing around upload and validation of add-ons to verify this didn't break anything?

I just switched the flag on -stage and it's active on -dev as well.

cc @krupa @ValentinaPC

[ValentinaPC]

Sure! Will do this today and tomorrow as well.

[ValentinaPC]

Verified this on AMO-stage and -dev FF56 (Win 7) and I have a question: things where fixed only for the add-ons that will be deleted/disabled from now on, or for the already deleted add-ons also?

Because "Trending extension" listing still displays incomplete add-ons. See homepage https://addons-dev.allizom.org/en-US/firefox/ or "See more trending extensions" page https://addons-dev.allizom.org/en-US/firefox/search/?sort=hotness&type=extension

[EnTeQuAk]

Yes, only for new cases that would be broken. Existing ones weren't fixed, we primarily fixed the root case but didn't do anything (yet?) to fix broken data. A reindex might help here

[diox]

Yeah we'll need a reindex to fix existing add-ons.

[ValentinaPC]

Verified different scenarios using AMO-dev FF56(Win 7), like: add-on with its only version disabled, add-on with no version, deleted add-on, submission, new file and new version upload, auto approve, check for updates and everything seems to work as expected.

I suppose https://github.com/mozilla/addons/issues/4993 has nothing to do with this issue, yes?

Anyway, I think we can mark this issue verified.

[diox]

mozilla/addons#4993 has nothing to do with this, no.

[jzav]

Are there any news here, please? I have exactly same issue as described at https://github.com/mozilla/addons/issues/4920...

[diox]

@jzav it should have been fixed a couple weeks ago - but we haven't recalculated ratings for affected add-ons. Can you link your add-on?

[jzav]

Thank you for quick response. And here is the link: https://addons.mozilla.org/en-US/firefox/addon/at-your-command/

[jzav]

@diox Can I somehow help to resolve this issue? My addon is still affected.

[diox]

The issue is fixed. The ratings for your add-on will automatically be recalculated as soon as a new one is posted.