search

mozilla-mobile / perf-frontend-issues

A repository to hold issues related to front-end mobile application performance.
4 stars 0 forks source link

FNPRMS is missing results 4/9-4/12 #100

Closed mcomella closed 4 years ago

mcomella commented 4 years ago

The results are NA for these days (and hasn't yet run on the 13th).

After a brief investigation, it looks like we're unable to download the build:

Downloading apk.
Done downloading apk.
Trying again to download the nightly apk (error 22).
...

And the reason is because the fennec-nightly builds don't exist for those days: https://firefox-ci-tc.services.mozilla.com/tasks/index/project.mobile.fenix.v2.fennec-nightly.2020.04

Since the nightly builds are also missing, perhaps it's because there was a compilation error on master that prevented these builds from running.

mcomella commented 4 years ago

After reading the #fenix-team slack backscroll from late last week, I discovered Google Play is backlogged on our Nightly builds so we disabled the builds. This impacts performance testing because we can't measure the changes making it in every night. I suggested that we continue building Nightly but disable the upload to GP instead so we can resume performance testing.

Waiting on a response from the Fenix team.

liuche commented 4 years ago

@JohanLorenzo from the GPS email thread, it looks like one of the options is re-enabling the Nightly builds but publishing them to archive.mo without pushing to the Play Store. Is there a bug to track that work? That option helps solve some of our QA/perf problems in the short-term - and in the meantime, Relman is going to see how long it actually takes a Nightly build to get approved, but we want to continue building our Nightlies.

mcomella commented 4 years ago

I found https://bugzilla.mozilla.org/show_bug.cgi?id=1628413 as an upstream issue.

JohanLorenzo commented 4 years ago

Thanks for raising this issue! I'm going to make the right changes in bug 1628413

JohanLorenzo commented 4 years ago

Good news: fenix nightlies are back - on their regular schedule, they're just not shipped to Google Play anymore.

(I just landed a way to manually send them to GP. This way, we have control over what gets sent to GP).

Please let me know if you see any issue!

mcomella commented 4 years ago

I confirmed that our perf test results are back and it seems there haven't been any regressions (though it's within noise limits so we won't know for a few days). Thanks a bunch JohanLorenzo! Closing as there are no more actions to do here.

bdekoz commented 4 years ago

I see apk's again on 2020-04-14, but none for 10, 11, 12, 13? Will these be back-filled?

On Wed, Apr 15, 2020 at 1:31 PM Michael Comella notifications@github.com wrote:

I confirmed that our perf test results are back and it seems there haven't been any regressions (though it's within noise limits so we won't know for a few days). Thanks a bunch JohanLorenzo! Closing as there are no more actions to do here.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/mozilla-mobile/perf-frontend-issues/issues/100#issuecomment-614264465, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHTQP6HJNTC2CMZVF222HTRMYKQZANCNFSM4MHJDSOA .

mcomella commented 4 years ago

As we discussed during sync today, we don't think it's necessary to backfill because we haven't seen a regression.

mcomella commented 4 years ago

As we discussed during sync today,

Sorry, I hope this didn't come off passive aggressively :) – I wanted to reference the discussion we had in sync for extra context to my response.