Are we retaining and amassing/accumulating all Telemetry json files from past runs?

stephendonner commented 5 years ago

Looking at https://qa-preprod-master.fxtest.jenkins.stage.mozaws.net/job/wpt/job/vimeo/601/console, are we really keeping all these files? (Found while investigating the dataloss over in https://github.com/mozilla/wpt-api/issues/264...)

23:24:20 [vimeo] Running shell script
23:24:20 + echo NICK fxtest-601
23:24:20 + openssl s_client -connect irc.mozilla.org:6697
23:24:20 + echo USER fxtest-601 8 CHANGELOG.md commands.txt Dockerfile docs Jenkinsfile LICENSE.txt metrics.json pipenv.txt Pipfile Pipfile.lock README.md send_to_datadog.py send_to_telemetry.py setup.cfg top50.json webpagetest-api wpt.json wpt.json.sample wpt-schema.json wpt-telemetry-190214_7P_12D.json wpt-telemetry-190214_G9_12H.json wpt-telemetry-190214_R8_12B.json wpt-telemetry-190214_W5_12C.json wpt-telemetry-190215_1V_87.json wpt-telemetry-190215_3N_N8.json wpt-telemetry-190215_40_1X.json wpt-telemetry-190215_4M_EW.json wpt-telemetry-190215_5P_20.json wpt-telemetry-190215_5S_VD.json wpt-telemetry-190215_69_VE.json wpt-telemetry-190215_6X_1Y.json wpt-telemetry-190215_CG_EV.json wpt-telemetry-190215_CQ_86.json wpt-telemetry-190215_ER_N7.json wpt-telemetry-190215_EZ_11Q.json wpt-telemetry-190215_FQ_EQ.json wpt-telemetry-190215_FV_N6.json wpt-telemetry-190215_FW_11M.json wpt-telemetry-190215_GF_85.json wpt-telemetry-190215_H4_1Z.json wpt-telemetry-190215_MM_11N.json wpt-telemetry-190215_QE_VF.json wpt-telemetry-190215_SC_VC.json wpt-telemetry-190215_TB_11P.json wpt-telemetry-190215_W4_NA.json wpt-telemetry-190215_WZ_EP.json wpt-telemetry-190215_Z3_8B.json wpt-telemetry-190216_25_V6.json wpt-telemetry-190216_2Y_88.json wpt-telemetry-190216_5V_V7.json wpt-telemetry-190216_60_V5.json wpt-telemetry-190216_73_89.json wpt-telemetry-190216_85_8C.json wpt-telemetry-190216_98_EN.json wpt-telemetry-190216_BG_EQ.json wpt-telemetry-190216_CV_MW.json wpt-telemetry-190216_FD_11D.json wpt-telemetry-190216_HB_22.json wpt-telemetry-190216_J2_EP.json wpt-telemetry-190216_JH_MX.json wpt-telemetry-190216_JP_11H.json wpt-telemetry-190216_KJ_N0.json wpt-telemetry-190216_M2_11C.json wpt-telemetry-190216_MB_11E.json wpt-telemetry-190216_P7_8B.json wpt-telemetry-190216_PM_ER.json wpt-telemetry-190216_QD_MY.json wpt-telemetry-190216_QJ_21.json wpt-telemetry-190216_WK_V8.json wpt-telemetry-190216_XE_23.json wpt-telemetry-190216_Z1_24.json wpt-telemetry-190217_1S_8A.json wpt-telemetry-190217_2F_EJ.json wpt-telemetry-190217_32_20.json wpt-telemetry-190217_65_1X.json wpt-telemetry-190217_7P_V2.json wpt-telemetry-190217_B0_88.json wpt-telemetry-190217_DH_EK.json wpt-telemetry-190217_DY_V4.json wpt-telemetry-190217_EJ_1Z.json wpt-telemetry-190217_ER_EH.json wpt-telemetry-190217_FP_11D.json wpt-telemetry-190217_G1_11B.json wpt-telemetry-190217_H6_MQ.json wpt-telemetry-190217_M2_89.json wpt-telemetry-190217_M4_EM.json wpt-telemetry-190217_NS_MP.json wpt-telemetry-190217_NT_MS.json wpt-telemetry-190217_P4_MT.json wpt-telemetry-190217_P6_V5.json wpt-telemetry-190217_R1_11C.json wpt-telemetry-190217_S7_V3.json wpt-telemetry-190217_W0_8B.json wpt-telemetry-190217_W1_1Y.json wpt-telemetry-190217_Y2_119.json wpt-telemetry-190218_02_121.json wpt-telemetry-190218_3X_11Z.json wpt-telemetry-190218_40_87.json wpt-telemetry-190218_64_EW.json wpt-telemetry-190218_6E_24.json wpt-telemetry-190218_70_YR.json wpt-telemetry-190218_7T_VD.json wpt-telemetry-190218_8P_YP.json wpt-telemetry-190218_8Y_EP.json wpt-telemetry-190218_AH_22.json wpt-telemetry-190218_B9_23.json wpt-telemetry-190218_BE_21.json wpt-telemetry-190218_BZ_N1.json wpt-telemetry-190218_EY_EN.json wpt-telemetry-190218_FK_11Y.json wpt-telemetry-190218_JA_88.json wpt-telemetry-190218_JB_N4.json wpt-telemetry-190218_KQ_YQ.json wpt-telemetry-190218_NE_EQ.json wpt-telemetry-190218_NZ_YS.json wpt-telemetry-190218_PG_N0.json wpt-telemetry-190218_S2_N3.json wpt-telemetry-190218_SG_85.json wpt-telemetry-190218_VQ_V6.json wpt-telemetry-190218_WM_V8.json wpt-telemetry-190218_XD_120.json wpt-telemetry-190218_Y2_V7.json wpt-telemetry-190218_Z8_86.json wpt-telemetry-190219_01_VH.json wpt-telemetry-190219_54_23.json wpt-telemetry-190219_62_22.json wpt-telemetry-190219_6J_N2.json wpt-telemetry-190219_A3_20.json wpt-telemetry-190219_B5_N3.json wpt-telemetry-190219_BR_EX.json wpt-telemetry-190219_C3_8J.json wpt-telemetry-190219_CK_11S.json wpt-telemetry-190219_CV_EZ.json wpt-telemetry-190219_DZ_11Q.json wpt-telemetry-190219_FJ_N4.json wpt-telemetry-190219_G1_VA.json wpt-telemetry-190219_GM_VC.json wpt-telemetry-190219_H5_11R.json wpt-telemetry-190219_KR_11P.json wpt-telemetry-190219_SB_24.json wpt-telemetry-190219_V3_8G.json wpt-telemetry-190219_WB_VB.json wpt-telemetry-190219_WV_N5.json wpt-telemetry-190219_X9_8H.json wpt-telemetry-190219_YA_EY.json wpt-telemetry-190219_YG_8F.json wpt-telemetry-190219_ZJ_F0.json wpt-telemetry-190220_0T_EE.json wpt-telemetry-190220_24_TY.json wpt-telemetry-190220_4V_8A.json wpt-telemetry-190220_69_119.json wpt-telemetry-190220_9T_EG.json wpt-telemetry-190220_AM_85.json wpt-telemetry-190220_CC_EF.json wpt-telemetry-190220_EE_MV.json wpt-telemetry-190220_ER_V0.json wpt-telemetry-190220_H2_1X.json wpt-telemetry-190220_JQ_1Y.json wpt-telemetry-190220_JS_V1.json wpt-telemetry-190220_MM_ED.json wpt-telemetry-190220_NA_MX.json wpt-telemetry-190220_R7_MW.json wpt-telemetry-190220_R9_118.json wpt-telemetry-190220_TJ_116.json wpt-telemetry-190220_VF_1Z.json wpt-telemetry-190220_VN_MT.json wpt-telemetry-190220_X2_TZ.json wpt-telemetry-190220_XE_87.json wpt-telemetry-190220_Z8_20.json wpt-telemetry-190220_ZT_86.json wpt-telemetry-190220_ZV_117.json wpt-telemetry-190221_36_11X.json wpt-telemetry-190221_3M_N8.json wpt-telemetry-190221_41_22.json wpt-telemetry-190221_4N_11W.json wpt-telemetry-190221_56_EZ.json wpt-telemetry-190221_8S_NE.json wpt-telemetry-190221_91_NA.json wpt-telemetry-190221_9H_23.json wpt-telemetry-190221_9Z_F3.json wpt-telemetry-190221_AE_VN.json wpt-telemetry-190221_B3_VP.json wpt-telemetry-190221_CY_VM.json wpt-telemetry-190221_DD_8B.json wpt-telemetry-190221_FT_8D.json wpt-telemetry-190221_G9_N9.json wpt-telemetry-190221_H6_24.json wpt-telemetry-190221_MD_EY.json wpt-telemetry-190221_MK_VQ.json wpt-telemetry-190221_MT_8H.json wpt-telemetry-190221_RR_11Y.json wpt-telemetry-190221_SZ_25.json wpt-telemetry-190221_WM_F1.json wpt-telemetry-190221_WT_8C.json wpt-telemetry-190221_XR_11V.json wpt-telemetry-190222_0K_MQ.json wpt-telemetry-190222_2T_86.json wpt-telemetry-190222_3G_21.json wpt-telemetry-190222_58_MS.json wpt-telemetry-190222_8E_22.json wpt-telemetry-190222_90_TZ.json wpt-telemetry-190222_9H_87.json wpt-telemetry-190222_9P_118.json wpt-telemetry-190222_A8_ED.json wpt-telemetry-190222_BJ_EE.json wpt-telemetry-190222_H7_11B.json wpt-telemetry-190222_J4_V0.json wpt-telemetry-190222_K1_TY.json wpt-telemetry-190222_PF_85.json wpt-telemetry-190222_PX_EG.json wpt-telemetry-190222_QB_MP.json wpt-telemetry-190222_S0_V1.json wpt-telemetry-190222_SW_11C.json wpt-telemetry-190222_SY_24.json wpt-telemetry-190222_TN_EF.json wpt-telemetry-190222_WY_23.json wpt-telemetry-190222_Y9_MT.json wpt-telemetry-190222_YS_88.json wpt-telemetry-190222_Z5_11D.json wpt-telemetry-190223_3P_MX.json wpt-telemetry-190223_43_ED.json wpt-telemetry-190223_5P_V3.json wpt-telemetry-190223_6S_24.json wpt-telemetry-190223_81_EJ.json wpt-telemetry-190223_86_EH.json wpt-telemetry-190223_8A_23.json wpt-telemetry-190223_8J_21.json wpt-telemetry-190223_9W_8A.json wpt-telemetry-190223_AM_119.json wpt-telemetry-190223_B2_22.json wpt-telemetry-190223_BA_116.json wpt-telemetry-190223_BP_86.json wpt-telemetry-190223_D0_MV.json wpt-telemetry-190223_DM_118.json wpt-telemetry-190223_K8_EE.json wpt-telemetry-190223_PE_V4.json wpt-telemetry-190223_RE_V5.json wpt-telemetry-190223_SJ_MW.json wpt-telemetry-190223_WF_V2.json wpt-telemetry-190223_XF_117.json wpt-telemetry-190223_YV_87.json wpt-telemetry-190223_ZD_MS.json wpt-telemetry-190223_ZR_85.json wpt-telemetry-190224_04_86.json wpt-telemetry-190224_0Z_87.json wpt-telemetry-190224_6B_11C.json wpt-telemetry-190224_6W_MR.json wpt-telemetry-190224_82_EM.json wpt-telemetry-190224_83_22.json wpt-telemetry-190224_BE_V0.json wpt-telemetry-190224_D9_21.json wpt-telemetry-190224_F6_TZ.json wpt-telemetry-190224_G1_23.json wpt-telemetry-190224_HE_MS.json wpt-telemetry-190224_KK_118.json wpt-telemetry-190224_M0_24.json wpt-telemetry-190224_M8_117.json wpt-telemetry-190224_MM_89.json wpt-telemetry-190224_NH_EJ.json wpt-telemetry-190224_PT_EH.json wpt-telemetry-190224_RK_88.json wpt-telemetry-190224_T7_MQ.json wpt-telemetry-190224_V2_MY.json wpt-telemetry-190224_VR_EN.json wpt-telemetry-190224_XW_119.json wpt-telemetry-190224_YX_V6.json wpt-telemetry-190224_ZR_V2.json wpt-telemetry-190225_09_MQ.json wpt-telemetry-190225_2R_EM.json wpt-telemetry-190225_57_MR.json wpt-telemetry-190225_7F_11C.json wpt-telemetry-190225_8F_EJ.json wpt-telemetry-190225_A7_23.json wpt-telemetry-190225_BS_V4.json wpt-telemetry-190225_CC_1Y.json wpt-telemetry-190225_D2_V3.json wpt-telemetry-190225_ED_11D.json wpt-telemetry-190225_GK_V5.json wpt-telemetry-190225_H2_89.json wpt-telemetry-190225_JA_8C.json wpt-telemetry-190225_JR_11B.json wpt-telemetry-190225_K9_V6.json wpt-telemetry-190225_KH_1X.json wpt-telemetry-190225_NH_8A.json wpt-telemetry-190225_NX_EK.json wpt-telemetry-190225_QZ_EN.json wpt-telemetry-190225_SH_8B.json wpt-telemetry-190225_TZ_1Z.json wpt-telemetry-190225_W5_MS.json wpt-telemetry-190225_WM_11E.json wpt-telemetry-190225_WY_MT.json wpt-telemetry-190226_03_V3.json wpt-telemetry-190226_0G_85.json wpt-telemetry-190226_3Q_11A.json wpt-telemetry-190226_48_EE.json wpt-telemetry-190226_4F_11C.json wpt-telemetry-190226_68_V1.json wpt-telemetry-190226_8P_EJ.json wpt-telemetry-190226_AR_11B.json wpt-telemetry-190226_B8_V4.json wpt-telemetry-190226_BZ_88.json wpt-telemetry-190226_CB_1Z.json wpt-telemetry-190226_E0_87.json wpt-telemetry-190226_G3_MX.json wpt-telemetry-190226_H2_21.json wpt-telemetry-190226_HR_EK.json wpt-telemetry-190226_HW_20.json wpt-telemetry-190226_JV_MS.json wpt-telemetry-190226_KW_86.json wpt-telemetry-190226_Q7_ED.json wpt-telemetry-190226_RR_MT.json wpt-telemetry-190226_VM_V2.json wpt-telemetry-190226_XJ_MW.json wpt-telemetry-190226_XM_116.json wpt-telemetry-190226_ZG_22.json wpt-telemetry-190227_24_21.json wpt-telemetry-190227_3A_23.json wpt-telemetry-190227_5P_22.json wpt-telemetry-190227_5S_11Y.json wpt-telemetry-190227_AG_VF.json wpt-telemetry-190227_AV_121.json wpt-telemetry-190227_BA_VB.json wpt-telemetry-190227_BE_EV.json wpt-telemetry-190227_BH_8G.json wpt-telemetry-190227_C5_120.json wpt-telemetry-190227_DC_EW.json wpt-telemetry-190227_DW_N2.json wpt-telemetry-190227_FA_11Z.json wpt-telemetry-190227_FV_N3.json wpt-telemetry-190227_GV_8E.json wpt-telemetry-190227_JM_24.json wpt-telemetry-190227_K4_VA.json wpt-telemetry-190227_PH_8D.json wpt-telemetry-190227_PQ_ES.json wpt-telemetry-190227_PQ_ET.json wpt-telemetry-190227_SH_8F.json wpt-telemetry-190227_TG_N5.json wpt-telemetry-190227_VR_N4.json wpt-telemetry-190227_XF_VE.json wpt-telemetry-190228_1M_24.json wpt-telemetry-190228_1P_21.json wpt-telemetry-190228_2J_1Z.json wpt-telemetry-190228_32_MS.json wpt-telemetry-190228_4C_8A.json wpt-telemetry-190228_5Y_TY.json wpt-telemetry-190228_DQ_EH.json wpt-telemetry-190228_DV_8C.json wpt-telemetry-190228_FZ_11C.json wpt-telemetry-190228_G8_MR.json wpt-telemetry-190228_HP_8B.json wpt-telemetry-190228_J9_TZ.json wpt-telemetry-190228_KF_MP.json wpt-telemetry-190228_N0_117.json wpt-telemetry-190228_P9_V0.json wpt-telemetry-190228_PX_116.json wpt-telemetry-190228_QV_89.json wpt-telemetry-190228_R5_119.json wpt-telemetry-190228_RW_EJ.json wpt-telemetry-190228_SW_EG.json wpt-telemetry-190228_TP_V1.json wpt-telemetry-190228_V5_EM.json wpt-telemetry-190228_VK_MQ.json wpt-telemetry-190228_ZW_20.json wpt-telemetry-190301_05_V4.json wpt-telemetry-190301_0G_11C.json wpt-telemetry-190301_23_MY.json wpt-telemetry-190301_53_88.json wpt-telemetry-190301_5P_11A.json wpt-telemetry-190301_7H_V1.json wpt-telemetry-190301_7P_1Z.json wpt-telemetry-190301_81_MX.json wpt-telemetry-190301_A1_11B.json wpt-telemetry-190301_B7_20.json wpt-telemetry-190301_CX_85.json wpt-telemetry-190301_GQ_MV.json wpt-telemetry-190301_KA_ED.json wpt-telemetry-190301_MA_MW.json wpt-telemetry-190301_ME_117.json wpt-telemetry-190301_MM_V5.json wpt-telemetry-190301_QF_86.json wpt-telemetry-190301_SE_EG.json wpt-telemetry-190301_V7_EF.json wpt-telemetry-190301_VK_8C.json wpt-telemetry-190301_VQ_1X.json wpt-telemetry-190301_YW_1Y.json wpt-telemetry-190301_Z2_EE.json wpt-telemetry-190301_Z9_V2.json wpt-telemetry-190302_0D_EF.json wpt-telemetry-190302_2C_V7.json wpt-telemetry-190302_2W_EN.json wpt-telemetry-190302_3A_V2.json wpt-telemetry-190302_AH_MW.json wpt-telemetry-190302_C3_MZ.json wpt-telemetry-190302_CH_11A.json wpt-telemetry-190302_E9_MT.json wpt-telemetry-190302_FP_EG.json wpt-telemetry-190302_HK_MY.json wpt-telemetry-190302_K2_8B.json wpt-telemetry-190302_KG_20.json wpt-telemetry-190302_M9_8C.json wpt-telemetry-190302_MW_88.json wpt-telemetry-190302_NC_22.json wpt-telemetry-190302_P5_V6.json wpt-telemetry-190302_Q1_21.json wpt-telemetry-190302_RJ_11C.json wpt-telemetry-190302_RY_24.json wpt-telemetry-190302_SJ_11E.json wpt-telemetry-190302_VX_V5.json wpt-telemetry-190302_WY_89.json wpt-telemetry-190302_X9_EP.json wpt-telemetry-190302_ZY_11D.json wpt-telemetry-190303_1Q_118.json wpt-telemetry-190303_1Q_8B.json wpt-telemetry-190303_1T_8A.json wpt-telemetry-190303_4B_TZ.json wpt-telemetry-190303_4X_1X.json wpt-telemetry-190303_5A_119.json wpt-telemetry-190303_6Z_V0.json wpt-telemetry-190303_7V_TY.json wpt-telemetry-190303_CC_117.json wpt-telemetry-190303_ES_V1.json wpt-telemetry-190303_ET_EG.json wpt-telemetry-190303_FD_MT.json wpt-telemetry-190303_HD_EH.json wpt-telemetry-190303_JC_1Z.json wpt-telemetry-190303_K2_MV.json wpt-telemetry-190303_KH_MQ.json wpt-telemetry-190303_KP_116.json wpt-telemetry-190303_MJ_EE.json wpt-telemetry-190303_QY_8C.json wpt-telemetry-190303_SJ_1Y.json wpt-telemetry-190303_X2_ED.json wpt-telemetry-190303_XH_89.json wpt-telemetry-190303_Y9_20.json wpt-telemetry-190303_Z5_MP.json wpt-telemetry-190304_06_MX.json wpt-telemetry-190304_0W_ED.json wpt-telemetry-190304_1Y_1X.json wpt-telemetry-190304_2M_V3.json wpt-telemetry-190304_5D_V5.json wpt-telemetry-190304_5T_118.json wpt-telemetry-190304_83_117.json wpt-telemetry-190304_AG_116.json wpt-telemetry-190304_B5_V2.json wpt-telemetry-190304_BA_119.json wpt-telemetry-190304_CS_8A.json wpt-telemetry-190304_JA_20.json wpt-telemetry-190304_K8_8B.json wpt-telemetry-190304_MZ_89.json wpt-telemetry-190304_NW_1Z.json wpt-telemetry-190304_T2_MQ.json wpt-telemetry-190304_V7_V4.json wpt-telemetry-190304_VB_EE.json wpt-telemetry-190304_VV_EG.json wpt-telemetry-190304_WY_MW.json wpt-telemetry-190304_YY_8C.json wpt-telemetry-190304_Z3_1Y.json wpt-telemetry-190304_Z7_MP.json wpt-telemetry-190304_ZK_EF.json wpt-telemetry-190305_04_11F.json wpt-telemetry-190305_1K_MR.json wpt-telemetry-190305_1S_11E.json wpt-telemetry-190305_2Y_EE.json wpt-telemetry-190305_5B_11G.json wpt-telemetry-190305_60_V5.json wpt-telemetry-190305_7J_87.json wpt-telemetry-190305_8A_V7.json wpt-telemetry-190305_8D_MX.json wpt-telemetry-190305_AQ_MZ.json wpt-telemetry-190305_B3_V4.json wpt-telemetry-190305_CW_1X.json wpt-telemetry-190305_DC_EF.json wpt-telemetry-190305_H4_8C.json wpt-telemetry-190305_HA_11D.json wpt-telemetry-190305_J6_8B.json wpt-telemetry-190305_K5_EH.json wpt-telemetry-190305_MC_22.json wpt-telemetry-190305_MK_V6.json wpt-telemetry-190305_NY_EG.json wpt-telemetry-190305_PY_1Y.json wpt-telemetry-190305_QR_1Z.json wpt-telemetry-190305_SQ_MS.json wpt-telemetry-190305_W6_88.json wpt-telemetry-190306_05_1X.json wpt-telemetry-190306_0M_EF.json wpt-telemetry-190306_3A_20.json wpt-telemetry-190306_4C_EE.json wpt-telemetry-190306_4Q_11B.json wpt-telemetry-190306_5T_MR.json wpt-telemetry-190306_7B_EG.json wpt-telemetry-190306_A9_MS.json wpt-telemetry-190306_AF_89.json wpt-telemetry-190306_AZ_8A.json wpt-telemetry-190306_EV_V0.json wpt-telemetry-190306_FE_MQ.json wpt-telemetry-190306_HY_V1.json wpt-telemetry-190306_JF_V2.json wpt-telemetry-190306_KM_11C.json wpt-telemetry-190306_KM_8B.json wpt-telemetry-190306_MG_11A.json wpt-telemetry-190306_NY_11D.json wpt-telemetry-190306_RM_V5.json wpt-telemetry-190306_SZ_88.json wpt-telemetry-190306_WX_1Y.json wpt-telemetry-190306_YA_MP.json wpt-telemetry-190306_YD_21.json wpt-telemetry-190306_YM_ED.json wpt-telemetry-190307_2M_86.json wpt-telemetry-190307_4B_EJ.json wpt-telemetry-190307_4Z_87.json wpt-telemetry-190307_6B_11G.json wpt-telemetry-190307_6Y_TY.json wpt-telemetry-190307_79_11C.json wpt-telemetry-190307_7M_11F.json wpt-telemetry-190307_CK_MQ.json wpt-telemetry-190307_DW_1Z.json wpt-telemetry-190307_FM_TZ.json wpt-telemetry-190307_FP_EK.json wpt-telemetry-190307_G3_EH.json wpt-telemetry-190307_HF_V0.json wpt-telemetry-190307_K2_MP.json wpt-telemetry-190307_K6_21.json wpt-telemetry-190307_KY_88.json wpt-telemetry-190307_ME_MS.json wpt-telemetry-190307_MS_85.json wpt-telemetry-190307_ND_MR.json wpt-telemetry-190307_QK_11B.json wpt-telemetry-190307_SZ_1Y.json wpt-telemetry-190307_XJ_1X.json wpt-telemetry-190307_YA_V1.json wpt-telemetry-190307_Z6_EM.json wpt-telemetry-190308_0S_88.json wpt-telemetry-190308_0X_11B.json wpt-telemetry-190308_1H_MP.json wpt-telemetry-190308_4B_EM.json wpt-telemetry-190308_4X_116.json wpt-telemetry-190308_52_23.json wpt-telemetry-190308_5A_1Y.json wpt-telemetry-190308_6F_11C.json wpt-telemetry-190308_6Y_1X.json wpt-telemetry-190308_BG_V5.json wpt-telemetry-190308_BX_TY.json wpt-telemetry-190308_FX_TZ.json wpt-telemetry-190308_GT_86.json wpt-telemetry-190308_M4_MQ.json wpt-telemetry-190308_PZ_21.json wpt-telemetry-190308_Q9_MS.json wpt-telemetry-190308_R0_EJ.json wpt-telemetry-190308_RX_V1.json wpt-telemetry-190308_T0_EK.json wpt-telemetry-190308_V7_85.json wpt-telemetry-190308_XV_11D.json wpt-telemetry-190308_Y7_EH.json wpt-telemetry-190308_YB_MR.json wpt-telemetry-190308_Z7_87.json wpt-telemetry-190309_3J_ED.json wpt-telemetry-190309_5F_V2.json wpt-telemetry-190309_5R_MS.json wpt-telemetry-190309_6P_MP.json wpt-telemetry-190309_9P_87.json wpt-telemetry-190309_AS_EJ.json wpt-telemetry-190309_C7_MR.json wpt-telemetry-190309_DH_EH.json wpt-telemetry-190309_F4_1Y.json wpt-telemetry-190309_HR_89.json wpt-telemetry-190309_KE_88.json wpt-telemetry-190309_NW_MQ.json wpt-telemetry-190309_PJ_85.json wpt-telemetry-190309_R8_1Z.json wpt-telemetry-190309_RP_11C.json wpt-telemetry-190309_X8_1X.json wpt-telemetry-190309_YH_EE.json wpt-telemetry-190309_ZE_20.json wpt-telemetry-190310_00_23.json wpt-telemetry-190310_0M_V3.json wpt-telemetry-190310_F2_88.json wpt-telemetry-190310_H2_EM.json wpt-telemetry-190310_MS_11D.json wpt-telemetry-190310_Y9_MX.json wpt-telemetry-190311_23_1X.json wpt-telemetry-190311_3W_ZR.json wpt-telemetry-190311_6M_127.json wpt-telemetry-190311_7M_15B.json wpt-telemetry-190311_AW_88.json wpt-telemetry-190311_B0_EH.json wpt-telemetry-190311_BP_WT.json wpt-telemetry-190311_CX_MX.json wpt-telemetry-190311_JD_ZC.json wpt-telemetry-190311_N9_V5.json wpt-telemetry-190312_3A_TK.json wpt-telemetry-190312_8S_DP.json wpt-telemetry-190312_8X_M1.json wpt-telemetry-190312_9W_85.json wpt-telemetry-190312_CZ_94.json wpt-telemetry-190312_NR_11M.json wpt-telemetry-190312_TV_T9.json wpt-telemetry-190312_XY_23.json wpt-telemetry-190313_3K_124.json wpt-telemetry-190313_3Q_MS.json wpt-telemetry-190313_AZ_V2.json wpt-telemetry-190313_CA_VJ.json wpt-telemetry-190313_EC_86.json wpt-telemetry-190313_PB_ED.json wpt-telemetry-190313_VE_23.json wpt-telemetry-190314_3B_14X.json wpt-telemetry-190314_45_8C.json wpt-telemetry-190314_7N_118.json wpt-telemetry-190314_8M_V5.json wpt-telemetry-190314_8N_EF.json wpt-telemetry-190314_HX_1Z.json wpt-telemetry-190314_WH_MR.json : fxtest-601

If indeed we're keeping these, I wonder/think that we might be a big cause of some qa-preprod out-of-disk-space (usually blamed on Docker, by me) issues.

/cc @davehunt

davehunt commented 5 years ago

Yes, it appears that we're not wiping out the workspace before every build. I can't locate our job DSL job in Jenkins, which we run to (re)generate the apt jobs... we should just need to update this, but as I said, I can't locate it...

davehunt commented 5 years ago

If it's been deleted then that would be unfortunate. I think it was named wpt.jobdsl or similar. Perhaps we can look on the file system and recover it?

davehunt commented 5 years ago

Another option would be to attempt to delete any telemetry JSON files before we run the tests. Without the job DSL job though we won't be able to tweak the top 50 and regenerate the jobs.

stephendonner commented 5 years ago

@davehunt thanks; I don't think I'd delete that job, but perhaps I (stupidly) did, or a forced restart of the server restored the jobs from an earlier backup? I dunno.

I looked tonight, but will need to look again tomorrow, perhaps with @oremj's help?

Stephens-MacBook-Pro:.ssh stephendonner$ nslookup qa-preprod-master.fxtest.jenkins.stage.mozaws.net
Server:     10.0.0.1
Address:    10.0.0.1#53

Non-authoritative answer:
qa-preprod-master.fxtest.jenkins.stage.mozaws.net   canonical name = ec2-34-197-24-237.compute-1.amazonaws.com.
Name:   ec2-34-197-24-237.compute-1.amazonaws.com
Address: 34.197.24.237

Stephens-MacBook-Pro:.ssh stephendonner$ ssh -i fx-test-sdonner-2016-09-23.pem sdonner@mozilla.com@34.197.24.237
ssh: connect to host 34.197.24.237 port 22: Operation timed out
Stephens-MacBook-Pro:.ssh stephendonner$ ssh -i fx-test-sdonner-2016-09-23.pem sdonner@mozilla.com@34.197.24.237:2222
ssh: Could not resolve hostname 34.197.24.237:2222: nodename nor servname provided, or not known
Stephens-MacBook-Pro:.ssh stephendonner$ ssh -i fx-test-sdonner-2016-09-23.pem sdonner@mozilla.com@34.197.24.237
^C
Stephens-MacBook-Pro:.ssh stephendonner$ ssh -i fx-test-sdonner-2016-09-23.pem sdonner@mozilla.com@34.197.24.237
ssh: connect to host 34.197.24.237 port 22: Operation timed out

stephendonner commented 5 years ago

If it's been deleted then that would be unfortunate. I think it was named wpt.jobdsl or similar. Perhaps we can look on the file system and recover it?

I've restored this using the Job Config History Plugin (!):

Started by user Stephen Donner
[EnvInject] - Loading node environment variables.
Building remotely on webpagetest (qatest linux centos7) in workspace /home/jenkins/slave/workspace/wpt-job-dsl
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/mozilla/wpt-api.git # timeout=10
Fetching upstream changes from https://github.com/mozilla/wpt-api.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/mozilla/wpt-api.git +refs/heads/*:refs/remotes/origin/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision e17038e0938152f754a0529c18e23f7574da4077 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e17038e0938152f754a0529c18e23f7574da4077
Commit message: "Bump decorator from 4.3.2 to 4.4.0 (#267)"
First time build. Skipping changelog.
Processing provided DSL script
Added items:
    GeneratedJob{name='wpt'}
    GeneratedJob{name='wpt/google'}
    GeneratedJob{name='wpt/google-mail'}
    GeneratedJob{name='wpt/google-docs'}
    GeneratedJob{name='wpt/youtube'}
    GeneratedJob{name='wpt/facebook'}
    GeneratedJob{name='wpt/amazon'}
    GeneratedJob{name='wpt/reddit'}
    GeneratedJob{name='wpt/wikipedia'}
    GeneratedJob{name='wpt/yahoo'}
    GeneratedJob{name='wpt/yahoo-mail'}
    GeneratedJob{name='wpt/twitter'}
    GeneratedJob{name='wpt/instagram'}
    GeneratedJob{name='wpt/linkedin'}
    GeneratedJob{name='wpt/ebay'}
    GeneratedJob{name='wpt/netflix'}
    GeneratedJob{name='wpt/twitch'}
    GeneratedJob{name='wpt/microsoft-outlook'}
    GeneratedJob{name='wpt/microsoft-office'}
    GeneratedJob{name='wpt/pornhub'}
    GeneratedJob{name='wpt/imgur'}
    GeneratedJob{name='wpt/paypal'}
    GeneratedJob{name='wpt/bing'}
    GeneratedJob{name='wpt/pinterest'}
    GeneratedJob{name='wpt/tumblr'}
    GeneratedJob{name='wpt/imdb'}
    GeneratedJob{name='wpt/wikia'}
    GeneratedJob{name='wpt/livejasmin'}
    GeneratedJob{name='wpt/apple'}
    GeneratedJob{name='wpt/microsoft-support'}
    GeneratedJob{name='wpt/github'}
    GeneratedJob{name='wpt/stack-overflow'}
    GeneratedJob{name='wpt/dropbox'}
    GeneratedJob{name='wpt/xvideos'}
    GeneratedJob{name='wpt/blogger'}
    GeneratedJob{name='wpt/adobe'}
    GeneratedJob{name='wpt/msn'}
    GeneratedJob{name='wpt/indeed'}
    GeneratedJob{name='wpt/wordpress'}
    GeneratedJob{name='wpt/spotify'}
    GeneratedJob{name='wpt/chaturbate'}
    GeneratedJob{name='wpt/xhamster'}
    GeneratedJob{name='wpt/soundcloud'}
    GeneratedJob{name='wpt/bbc'}
    GeneratedJob{name='wpt/discord'}
    GeneratedJob{name='wpt/vkontakte'}
    GeneratedJob{name='wpt/vimeo'}
    GeneratedJob{name='wpt/xnxx'}
    GeneratedJob{name='wpt/deviantart'}
    GeneratedJob{name='wpt/yandex'}
    GeneratedJob{name='wpt/whatsapp'}
Finished: SUCCESS

stephendonner commented 5 years ago

Sigh, thanks for the help so far, and I'll need further help in sorting this out.

Looks like calling deleteDir() in an always stage is a good idea (per https://stackoverflow.com/questions/37468455/jenkins-pipeline-wipe-out-workspace and https://jenkins.io/doc/pipeline/steps/workflow-basic-steps/#deletedir-recursively-delete-the-current-directory-from-the-workspace), but it's not doing what we want, which we can see over in:

https://qa-preprod-master.fxtest.jenkins.stage.mozaws.net/job/wpt/job/twitch/872/artifact/

Attaching post-merge screenshot of Twitch's latest run, with a bazillion wpt-telemetry*.json files around in its workspace :-(

The issue is likely: it doesn't look like we're cleaning past files up, but that might be due to Firefox crashing, and thus, the wpt build failing at a step where there's no cleanup -- or not the correct type/level. (So, look into failed(), success(), always(), Pipeline-stage robustness, etc., as well as selective before-SCM-checkout cleaning up.)

davehunt commented 5 years ago

It looks like there's a cleanup condition, which should help, but on trying to use it I've found our Jenkins instance is not up to date. I thought we had automatic updates... @oremj can you assist?

davehunt commented 5 years ago

We're using Jenkins 2.73.3.1, which was released in 2017-11-08(!)

davehunt commented 5 years ago

@stephendonner originally I thought we could modify the wpt-job-dsl job to wipe of the workspace and force clone, instead of using the deleteDir in our pipeline. It turns out this repeats the wipe for every stage, and so we wipe out the webpagetest-api repository that we clone before we're able to use it. I then discovered the cleanup post condition mentioned above and implemented that only to discover our Jenkins was out of date. I've restored your original solution, and our workspaces should be mostly clear now. Let's keep this open to see if we can upgrade and implement a cleanup post condition.

stephendonner commented 5 years ago

@davehunt yup; one of my runs (which deleted the workspace at the beginning, before SCM checkout) ended up not having -- as you've observed/noted -- the Dockerfile from that workspace, for starters.

I've had https://github.com/mozilla/wpt-api/issues/206 open and it seems like "soon" is a good time to smartly pay attention to the status codes we get returned from the API.

In the case of these crashes (Vimeo, Twitch, Yandex), we should, at the least:

Before doing any Telemetry and/or DataDog submissions (so, before entering their respective steps in our Declarative Pipeline), we should ensure we're getting a 200 status code back (codes 100 == "test started", 101 === "test in queue", etc.)

Note to self: we should reference both https://sites.google.com/a/webpagetest.org/docs/advanced-features/webpagetest-restful-apis and Marcel's NodeJS wrapper, to see what fits for us: https://github.com/marcelduran/webpagetest-api.

mozilla / wpt-api

Are we retaining and amassing/accumulating all Telemetry json files from past runs? #265