FNPRMS perf tests are broken because fennecNightly builds are no longer built (results report N/A)

[mcomella]

It looks like fennec-nightly is unavailable: https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/mobile.v2.fenix.fennec-nightly.2020.07.24.latest.armeabi-v7a/artifacts/public/build/armeabi-v7a/geckoNightly/target.apk

nightly, on the other hand, is fine: https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/mobile.v2.fenix.nightly.2020.07.24.latest.armeabi-v7a/artifacts/public/build/armeabi-v7a/geckoNightly/target.apk

This may be related to the build variant changes.

[mcomella]

As of three days ago, this is intended behavior: https://groups.google.com/a/mozilla.com/g/fenix-team/c/r-eDkOr0hkQ/m/tlZRumsdCAAJ

I'll follow-up.

[mcomella]

This landed in https://github.com/mozilla-mobile/fenix/pull/12225. Given it's friday and we haven't had results for a few days, I think we should revert the change and figure out the long term solution next week. My preference is that the migration code moves into the nightly build configuration. Barring that, we should continue to build fennec-nightly builds, even if we don't build beta, so that the migration code is covered in on a nightly basis.

[mcomella]

I opened https://github.com/mozilla-mobile/fenix/pull/12918 to revert

[mcomella]

Given it's friday and we haven't had results for a few days, I think we should revert the change and figure out the long term solution next week.

It's difficult to switch to nightly temporarily because it'll create a jump in the results and we can't annotate the dashboards to explain it. Also, we'll eventually undo the change when the migration code is included so it's preferred not to add that confusion to the dashboards.

[mcomella]

csadilek raised a risk https://github.com/mozilla-mobile/fenix/pull/12918#issuecomment-663728284 so we can't land the revert.

Until we talk with JohanLorenzo, I think our options are:

• Do nothing. The FNPRMS performance tests will remain broken from mid-week until whenever this is resolved
• Switch to nightly builds. This will cause our dashboards to have inaccurate data until the long term solution is resolved (assuming it's resolved in the way I'm thinking of), at which point it'll revert back to the current trend. We're unable to annotate the dashboard so this temporary trend could be confusing

@MarcLeclair Do you have an opinion on the best course of action?

[JohanLorenzo]

Hi! I'm sorry for the breakage. I wasn't aware these tests existed.

For the record, an email thread was started. Let's continue the conversation there and get back here once we have a solution.

[mcomella]

I discovered a solution for our perf-tests that makes this less urgent (i.e. it's still urgent but not every-day-counts urgent): SF FNPRMS has been testing two builds – with and without on the migration code – so we can run tests on the current build variant (without the migration code) without skewing our data. It's not ideal but it's serviceable.

We'll continue discussing the long term solution in the email thread.

[mcomella]

Opened https://github.com/mozilla-mobile/FNPRMS/pull/30 for the short term solution.

[mcomella]

bdekoz merged the changes upstream and confirmed it addressed the issue. They're filling in the missed jobs as well.

Now we're waiting on upstream changes for the long term solution.

[mcomella]

The short term fix didn't work: I think parsing is still broken. Moving to In Progress.

[mcomella]

bdekoz made some changes that I believe address the short term issues we've been seeing. Moving to Waiting for long term solution.

[mcomella]

I think it's been too long and it'd be more disruptive to change things so I think we should leave it the way it is.