Investigate TypeError: list indices must be integers or slices, not str failure, with truncated fxa-homepage.json output file

[stephendonner]

I need to investigate why I'm seeing these failures[0] (along with truncated fxa-homepage.json files):

20:54:29 [wpt-api.fxa-perf] Running shell script
20:54:29 + python ./send_to_datadog.py
20:54:30 Traceback (most recent call last):
20:54:30   File "./send_to_datadog.py", line 21, in <module>
20:54:30     TTFB = loaded_json["data"]["median"]["firstView"]["TTFB"]
20:54:30 TypeError: list indices must be integers or slices, not str

Attached are the good/full fxa-homepage-passing.json and fxa-homepage-failed.json files from [0] and [1] below, respectively

[0] https://qa-preprod-master.fxtest.jenkins.stage.mozaws.net/job/wpt-api.fxa-perf/3355/ [1] https://qa-preprod-master.fxtest.jenkins.stage.mozaws.net/job/wpt-api.fxa-perf/3356/

fxa-homepage-passing-json.txt fxa-homepage-failed-json.txt

Just rename those files above to end in .json, of course ^^^

[stephendonner]

As I originally suspected, I'm pretty sure this is due to the fact that I had two perf-testing jobs running off the same Jenkinsfile (different branches), and thus, same cron.

This caused them to run in parallel, and I believe whichever one finished first, passed. The other suffered a truncated JSON file due to an incomplete run.

To bear this out, I've disabled https://qa-preprod-master.fxtest.jenkins.stage.mozaws.net/job/wpt-api.amo.dev-hp/, and left my original job of https://qa-preprod-master.fxtest.jenkins.stage.mozaws.net/job/wpt-api.fxa-perf/ intact.

I'll keep watching it for stability.

[stephendonner]

Notice that the delta in the passing job's time interval (i.e. the unlabeled "Last Success" column with 4 minutes 11 seconds) directly correlates with the failing time-since delta in the failing job ("Last Failure").

[stephendonner]

One approach, from chatting with @davehunt, is to add an executor label in Jenkins, and force the 4 or so branched builds to run sequentially/only when the labelled executor is free.