search

cloud-bulldozer / performance-dashboards

Performance dashboards from the Perf & Scale team
Apache License 2.0
16 stars 36 forks source link

could not find a service account token for service account "prometheus-k8s" #39

Closed qiliRedHat closed 2 years ago

qiliRedHat commented 2 years ago

dittybopper deploy was successfully, but dashboards had no data and were "Forbidden".

Deploy was successful

% ./deploy.sh 

    ____  _ __  __        __
   / __ \(_) /_/ /___  __/ /_  ____  ____  ____  ___  _____
  / / / / / __/ __/ / / / __ \/ __ \/ __ \/ __ \/ _ \/ ___/
 / /_/ / / /_/ /_/ /_/ / /_/ / /_/ / /_/ / /_/ /  __/ /
/_____/_/\__/\__/\__, /_.___/\____/ .___/ .___/\___/_/
                /____/           /_/   /_/

Using k8s command: oc
Using namespace: dittybopper
Using default grafana password: admin

Getting environment vars...
Command "get-token" is deprecated, and will be removed in the future version. Use oc create token instead.
error: could not find a service account token for service account "prometheus-k8s"
Prometheus URL is: https://prometheus-k8s-openshift-monitoring.apps.qili-preserve-az0604.qe.azure.devcluster.openshift.com
Prometheus bearer token collected.

Creating namespace...
Looks like the namespace dittybopper already exists, deleting it
namespace "dittybopper" deleted
namespace/dittybopper created

Deploying Grafana...
service/dittybopper created
route.route.openshift.io/dittybopper created
Warning: would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (containers "dittybopper", "dittybopper-syncer" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "dittybopper", "dittybopper-syncer" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "dittybopper", "dittybopper-syncer" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "dittybopper", "dittybopper-syncer" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
deployment.apps/dittybopper created
configmap/sc-ocp-prom created
configmap/sc-grafana-config created

Waiting for dittybopper deployment to be available...
deployment.apps/dittybopper condition met

You can access the Grafana instance at http://dittybopper-dittybopper.apps.qili-preserve-az0604.qe.azure.devcluster.openshift.com

Resources were running well

 % oc get all -n dittybopper
NAME                               READY   STATUS    RESTARTS   AGE
pod/dittybopper-5c65fccf66-hcdx5   2/2     Running   0          65m

NAME                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/dittybopper   ClusterIP   172.30.164.177   <none>        3000/TCP   65m

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/dittybopper   1/1     1            1           65m

NAME                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/dittybopper-5c65fccf66   1         1         1       65m

NAME                                   HOST/PORT                                                                             PATH   SERVICES      PORT   TERMINATION   WILDCARD
route.route.openshift.io/dittybopper   dittybopper-dittybopper.apps.qili-preserve-az0604.qe.azure.devcluster.openshift.com          dittybopper   3000                 None
qili@qili-mac dittybopper % ls
README.md           deploy.sh           dittybopper_screenshot.png  k8s-deploy.sh           syncer              templates

But dashboard graphs had no data and were "Forbidden"

GET
    http://dittybopper-dittybopper.apps.qili-preserve-bm.qe-lrc.devcluster.openshift.com/api/datasources/proxy/1/api/v1/series?match[]=kube_node_role{role="master"}&start=1654567794&end=1654571394
Status403
Forbidden
Grafana
Templating
Template variable service failed <!DOCTYPE html> <html lang="en" charset="utf-8"> <head> <title>Log In</title> <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no"> <style> @font-face { font-family: "Open Sans"; src: url(data:application/x-font-woff;charset=utf-8;base64,d09GRg.......AA) format('woff'); font-weight: 700; font-style: normal; } @font-face { font-family: "Open Sans"; src: url(data:application/x-font-woff;charset=utf-8;base64,d09GRg......AAA=) format('woff'); font-weight: 400; font-style: normal; } * { box-sizing: border-box; } body { font-family: "Open Sans", Helvetica, Arial, sans-serif; font-size: 12px; line-height: 1.66666667; color: #363636; background-color: #f2f2f2; margin: 0; display: flex; height: 100%; -webkit-flex-direction: column; -ms-flex-direction: column; flex-direction: column; border-top: 35px solid #000; } .btn { box-shadow: 0 2px 3px rgba(3,3,3,.1); display: inline-block; margin-bottom: 0; font-weight: 600; text-align: center; vertical-align: middle; -ms-touch-action: manipulation; touch-action: manipulation; cursor: pointer; background-image: none; border: 1px solid transparent; white-space: nowrap; padding: 2px 6px; font-size: 12px; line-height: 1.66666667; border-radius: 1px; -webkit-user-select: none; -moz-user-select: none; -ms-user-select: none; user-select: none; } .btn:active, .btn.active { box-shadow: inset 0 2px 8px rgba(3,3,3,.2); outline: 0; } .btn:focus, .btn:active:focus, .btn.active:focus { outline: thin dotted; outline: 5px auto -webkit-focus-ring-color; outline-offset: -2px; } .btn-lg { border-radius: 1px; font-size: 14px; line-height: 1.3333333; padding: 6px 10px; } .btn-default { background-color: #f1f1f1; background-image: linear-gradient(to bottom,#fafafa 0,#ededed 100%); background-repeat: repeat-x; border-color: #bbb; color: #4d5258; } .btn-default:focus, .btn-default:hover { background-color: #f1f1f1; background-image: none; border-color: #bbb; color: #4d5258; } .btn-primary { background-color: #0088ce; background-image: linear-gradient(to bottom,#39a5dc 0,#0088ce 100%); background-repeat: repeat-x; border-color: #00659c; color: #fff; } .btn-primary.active, .btn-primary:active, .btn-primary:focus, .btn-primary:hover { background-color: #0088ce; background-image: none; border-color: #00659c; color: #fff; text-decoration: none; } .btn-primary.active.focus, .btn-primary.active:focus, .btn-primary.active:hover, .btn-primary:active.focus, .btn-primary:active:focus, .btn-primary:active:hover { background-color: #0077b5; border-color: #004e78; } img { align-self: center; width: 170px; } footer { color: #999; display: block; font-size: 10px; margin-bottom: 10px; text-align: center; } footer a { color: #999; display: inline-block; height: 25px; line-height: 25px; text-decoration: underline; } footer a:hover { color: #777; } .form-control { display: block; width: 100%; height: 26px; padding: 2px 6px; font-size: 12px; line-height: 1.66666667; color: #333333; background-color: #ffffff; background-image: none; border: 1px solid #bababa; border-radius: 1px; -webkit-box-shadow: inset 0 1px 1px rgba(0, 0, 0, 0.075); box-shadow: inset 0 1px 1px rgba(0, 0, 0, 0.075); -webkit-transition: border-color ease-in-out .15s, box-shadow ease-in-out .15s; transition: border-color ease-in-out .15s, box-shadow ease-in-out .15s; } .form-control:focus { border-color: #66afe9; outline: 0; -webkit-box-shadow: inset 0 1px 1px rgba(0,0,0,.075), 0 0 8px rgba(102, 175, 233, 0.6); box-shadow: inset 0 1px 1px rgba(0,0,0,.075), 0 0 8px rgba(102, 175, 233, 0.6); } .form-control::-moz-placeholder { color: #999999; opacity: 1; } .form-control:-ms-input-placeholder { color: #999999; } .form-control::-webkit-input-placeholder { color: #999999; } .form-control:-moz-placeholder { color: #999999; font-style: italic; } .form-control::-moz-placeholder { color: #999999; font-style: italic; } .form-control:-ms-input-placeholder { color: #999999; font-style: italic; } .form-control::-webkit-input-placeholder { color: #999999; font-style: italic; } .form-control[disabled], .form-control[readonly], fieldset[disabled] .form-control { cursor: not-allowed; background-color: #f8f8f8; opacity: 1; } .form-group { margin-bottom: 15px; } label { display: inline-block; margin-bottom: 5px; font-weight: bold; color: #676767; } .panel-content { padding: 0 20px; } .panel-content p { margin: 0 0 2em; } .panel-content-centered { text-align: center; } .panel-login { background: #fff; box-shadow: 0 3px 1px -2px rgba(0, 0, 0, 0.15), 0 2px 2px 0 rgba(0, 0, 0, 0.1), 0 1px 5px 0 rgba(0, 0, 0, 0.09); display: flex; flex-direction: column; justify-content: center; margin: 20px auto; max-width: 500px; padding: 20px 0; width: 80%; } @media (min-width: 480px) { .panel-login { margin-top: 65px; } } @media (min-width: 768px) { .panel-login { margin-top: 150px; padding: 50px 0; } } .panel-login + .panel-login { margin-top: 50px; } .panel-login-internal { box-shadow: none; background: #ededed; border: 1px solid #e0e0e0; } @media (min-width: 768px) { .panel-login-internal { padding: 50px; } } .panel-logos { display: flex; justify-content: center; padding: 0 50px 2em 50px; } </style> </head> <body> <div class="panel-login"> <div class="panel-logos"> <img src="data:image/svg+xml, %3Csvg width='142px' height='37px' viewBox='0 0 142 37' version='1.1' xmlns='http://www.w3.org/2000/svg' xmlns:xlink='http://www.w3.org/1999/xlink'%3E%3Cdefs%3E%3C/defs%3E%3Cg id='Page-1' stroke='none' stroke-width='1' fill='none' fill-rule='evenodd'%3E%3Cg id='Logotype_RH_OpenShift_wLogo_RGB_Black'%3E%3Cpath d='M11.4906686,16.643211 L5.13975878,18.9582569 C5.22362007,19.9751801 5.39156205,20.9834008 5.64186063,21.9725688 L11.6772605,19.7729358 C11.4767181,18.7425779 11.4139711,17.6901164 11.4906686,16.643211' id='Shape' fill='%23A30000'%3E%3C/path%3E%3Cpath d='M39.5710537,9.6166055 C39.1270163,8.69627223 38.6061009,7.81509898 38.0138595,6.98247706 L31.6629496,9.29752294 C32.393731,10.0493934 33.0212164,10.8952647 33.5288687,11.812844 L39.5710537,9.6166055 Z' id='Shape' fill='%23A30000'%3E%3C/path%3E%3Cpath d='M25.5936239,7.81412844 C26.8751896,8.41611467 28.0372319,9.24529918 29.0235224,10.2615596 L35.3778248,7.94990826 C30.6393,1.29992412 22.0481954,-1.36891502 14.3787494,1.42651497 C6.70930337,4.22194497 1.84685009,11.7944602 2.49354634,19.9358716 L8.83767104,17.6072477 C9.12320758,13.740697 11.2921307,10.2623601 14.6375179,8.30593966 C17.9829052,6.34951924 22.0762678,6.16560047 25.5834462,7.81412844' id='Shape' fill='%23CC0000'%3E%3C/path%3E%3Cpath d='M6.0387925,21.8266055 L0.0169629003,24.0262385 C0.573491414,26.2356156 1.53519027,28.3223912 2.85315983,30.1804587 L9.17692904,27.8755963 C7.56226609,26.2105296 6.47016636,24.1088798 6.03539992,21.83' id='Shape' fill='%23CC0000'%3E%3C/path%3E%3Cpath d='M32.3686063,19.4300917 C32.2643314,20.841458 31.9033186,22.2219034 31.3033362,23.5034862 C28.5472776,29.4104497 21.5317403,31.9694836 15.6228311,29.223211 C14.3374913,28.6240179 13.1718208,27.7958556 12.182755,26.7791743 L5.83523769,29.1010092 C7.64240716,31.6466911 10.0799743,33.678559 12.9087671,34.9972477 C22.0050887,39.2270771 32.8068989,35.2929255 37.0571519,26.2021101 C38.3818383,23.3690381 38.9454473,20.2393588 38.6923755,17.1218349 L32.3686063,19.4300917 Z' id='Shape' fill='%23CC0000'%3E%3C/path%3E%3Cpath d='M33.9427634,11.6702752 L27.9073635,13.8699083 C29.0399306,15.8940137 29.5542168,18.2060678 29.3865284,20.5197248 L35.7272605,18.2148624 C35.5445721,15.9412435 34.9393778,13.7217122 33.9427634,11.6702752' id='Shape' fill='%23CC0000'%3E%3C/path%3E%3Cpath d='M11.4669206,17.9229358 C11.4514214,17.4973787 11.4593494,17.0712905 11.4906686,16.6466055 L5.13975878,18.9582569 C5.170292,19.3655963 5.22457328,19.7695413 5.28563972,20.1734862 L11.4669206,17.9229358 Z' id='Shape' fill='%23820000'%3E%3C/path%3E%3Cpath d='M38.648272,7.94311927 C38.4481097,7.60366972 38.2411624,7.29816514 38.0206447,6.98587156 L31.6697348,9.29752294 C31.9463089,9.58265476 32.2068094,9.88296619 32.4500282,10.1970642 L38.648272,7.94311927 Z' id='Shape' fill='%23820000'%3E%3C/path%3E%3Cpath d='M5.83523769,29.1010092 C6.32688347,29.7920917 6.86650454,30.4477158 7.4501058,31.0630275 L14.3506136,28.5443119 C13.5569444,28.0487617 12.8256869,27.4595506 12.1725772,26.7893578 L5.83523769,29.1010092 Z M38.6923755,17.1218349 L32.355036,19.4266972 C32.2850799,20.3500244 32.1051289,21.261683 31.8190083,22.1422936 L38.7195162,19.623578 C38.7704215,18.7844821 38.762486,17.942848 38.6957681,17.1048624 L38.6923755,17.1218349 Z' id='Shape' fill='%23A30000'%3E%3C/path%3E%3Cpath d='M51.3704472,14.9086239 L50.0337706,12.1488991 L49.1211666,12.1488991 L49.1211666,14.9086239 L46.9024192,14.9086239 L46.9024192,6.72788991 L50.5664057,6.72788991 C51.0049547,6.72370771 51.442394,6.77271946 51.8691564,6.87385321 C52.2362205,6.95700406 52.5825153,7.11407964 52.8869305,7.33550459 C53.1703323,7.55001939 53.3935388,7.83415441 53.5349132,8.16036697 C53.6963021,8.55057816 53.7737503,8.97042267 53.7622161,9.39256881 C53.785524,9.92358385 53.6471055,10.4491608 53.3652842,10.8997248 C53.0903921,11.2956461 52.7077154,11.6043165 52.2626957,11.7890826 L53.9013119,14.9120183 L51.3602694,14.9120183 L51.3704472,14.9086239 Z M51.2652772,8.81889908 C51.0532678,8.63460819 50.7745732,8.54616178 50.4951615,8.57449541 L49.1381295,8.57449541 L49.1381295,10.3633945 L50.4781986,10.3633945 C50.7642411,10.3916085 51.0500626,10.3076391 51.2754549,10.1291743 C51.4428842,9.94436454 51.5274641,9.69906702 51.509543,9.45027523 C51.5144976,9.21582541 51.4267072,8.98890554 51.2652772,8.81889908 L51.2652772,8.81889908 Z' id='Shape' fill='%23000000'%3E%3C/path%3E%3Cpolygon id='Shape' fill='%23000000' points='55.6179574 14.9086239 55.6179574 6.72788991 61.8806602 6.72788991 61.8806602 8.63220183 57.8604528 8.63220183 57.8604528 9.73201835 60.2793624 9.73201835 60.2793624 11.6023853 57.8604528 11.6023853 57.8604528 13.0043119 61.9620821 13.0043119 61.9620821 14.9086239 55.6179574 14.9086239'%3E%3C/polygon%3E%3Cpath d='M70.5520948,12.7191743 C70.3880958,13.2099857 70.0979314,13.64899 69.7107349,13.9921101 C69.3098113,14.3302067 68.8360219,14.5707208 68.3265623,14.6947706 C67.7017301,14.8482634 67.0598617,14.9212701 66.4165397,14.9120183 L63.7635421,14.9120183 L63.7635421,6.72788991 L66.6268797,6.72788991 C67.2130564,6.72237328 67.7976527,6.78964551 68.3672732,6.92816514 C68.8570395,7.04587705 69.3117486,7.27859613 69.693772,7.60706422 C70.0723856,7.94674356 70.3613806,8.37465098 70.5351319,8.85284404 C70.7510322,9.47201481 70.8522048,10.1254066 70.8336789,10.7809174 C70.8491792,11.4382351 70.753988,12.0934746 70.5520948,12.7191743 L70.5520948,12.7191743 Z M68.4079842,9.83724771 C68.3571389,9.58815447 68.2465309,9.35516404 68.0856891,9.15834862 C67.9291483,8.98112258 67.7274731,8.84972505 67.5021653,8.77816514 C67.2144139,8.68969682 66.9142446,8.6484289 66.6133094,8.6559633 L66.0535336,8.6559633 L66.0535336,12.970367 L66.5454578,12.970367 C66.8489365,12.976027 67.1516686,12.938307 67.4444915,12.8583486 C67.6778863,12.7942351 67.8898969,12.6688321 68.0585485,12.4951376 C68.2292233,12.3025509 68.3462289,12.0684075 68.3978065,11.8162385 C68.4738124,11.4822802 68.509126,11.1403424 68.5029764,10.7978899 C68.5087551,10.4750693 68.4768755,10.1526763 68.4079842,9.83724771 L68.4079842,9.83724771 Z' id='Shape' fill='%23000000'%3E%3C/path%3E%3Cpolygon id='Shape' fill='%23000000' points='81.6152983 14.9086239 81.6152983 11.6838532 79.0640781 11.6838532 79.0640781 14.9120183 76.7571237 14.9120183 76.7571237 6.72788991 79.0640781 6.72788991 79.0640781 9.70825688 81.6119058 9.70825688 81.6119058 6.72788991 83.9256454 6.72788991 83.9256454 14.9086239 81.6119058 14.9086239'%3E%3C/polygon%3E%3Cpath d='M91.2196925,14.9086239 L90.7786571,13.4863303 L88.3258217,13.4863303 L87.8847863,14.9086239 L85.455699,14.9086239 L88.4242065,6.72449541 L90.7141981,6.72449541 L93.6827056,14.9086239 L91.2298702,14.9086239 L91.2196925,14.9086239 Z M90.076393,11.133945 C90.0062797,10.8691743 89.9440824,10.6428746 89.8898011,10.4550459 L89.7439202,9.92889908 C89.7009475,9.77275229 89.6636291,9.63018349 89.631965,9.50119266 C89.6003009,9.37220183 89.5731603,9.22963303 89.5505431,9.07348624 C89.5279259,9.21379205 89.5007853,9.35070336 89.4691212,9.48422018 C89.4374571,9.617737 89.4001387,9.76143731 89.357166,9.9153211 L89.2112851,10.4414679 L89.0246932,11.120367 L88.8720271,11.6566972 L90.2290591,11.6566972 L90.076393,11.133945 Z' id='Shape' fill='%23000000'%3E%3C/path%3E%3Cpolygon id='Shape' fill='%23000000' points='98.3033996 8.70348624 98.3033996 14.9086239 96.0269784 14.9086239 96.0269784 8.70348624 93.7369869 8.70348624 93.7369869 6.72788991 100.583213 6.72788991 100.583213 8.70348624 98.2932219 8.70348624'%3E%3C/polygon%3E%3Cpath d='M103.62975,8.1433945 C103.526378,8.39608777 103.325918,8.59666125 103.073367,8.70009174 C102.805439,8.80701938 102.506721,8.80701938 102.238792,8.70009174 C101.986242,8.59666125 101.785781,8.39608777 101.682409,8.1433945 C101.575887,7.87524867 101.575887,7.57649445 101.682409,7.30834862 C101.785781,7.05565535 101.986242,6.85508187 102.238792,6.75165138 C102.506721,6.64472374 102.805439,6.64472374 103.073367,6.75165138 C103.325918,6.85508187 103.526378,7.05565535 103.62975,7.30834862 C103.736965,7.57636125 103.736965,7.87538187 103.62975,8.1433945 L103.62975,8.1433945 Z M103.473692,7.37623853 C103.387177,7.16329452 103.218339,6.99436165 103.005516,6.90779817 C102.781088,6.8185969 102.531072,6.8185969 102.306644,6.90779817 C102.094037,6.9946886 101.925309,7.16351162 101.838468,7.37623853 C101.748972,7.60072684 101.748972,7.85101628 101.838468,8.07550459 C101.925633,8.28801611 102.094252,8.45673073 102.306644,8.54394495 C102.531072,8.63314622 102.781088,8.63314622 103.005516,8.54394495 C103.218122,8.45705452 103.38685,8.2882315 103.473692,8.07550459 C103.563188,7.85101628 103.563188,7.60072684 103.473692,7.37623853 L103.473692,7.37623853 Z M103.066582,7.71568807 C103.020631,7.77252874 102.957568,7.8130072 102.886775,7.83110092 L103.124256,8.29614679 L102.89356,8.29614679 L102.66965,7.85146789 L102.449132,7.85146789 L102.449132,8.29614679 L102.252363,8.29614679 L102.252363,7.12844037 L102.754465,7.12844037 C102.802692,7.1283472 102.850681,7.13520665 102.896953,7.14880734 C102.940455,7.16212286 102.980835,7.1840562 103.015693,7.21330275 C103.05072,7.24286016 103.078564,7.28000617 103.097115,7.32192661 C103.118595,7.37111146 103.129018,7.4244156 103.127648,7.47807339 C103.133444,7.55647728 103.11186,7.63446295 103.066582,7.6987156 L103.066582,7.71568807 Z M102.886775,7.37623853 C102.848653,7.34786622 102.801934,7.33348309 102.754465,7.33550459 L102.449132,7.33550459 L102.449132,7.67495413 L102.754465,7.67495413 C102.801863,7.67643861 102.848409,7.66210866 102.886775,7.63422018 C102.924461,7.60197447 102.946158,7.55484402 102.946158,7.50522936 C102.946158,7.4556147 102.924461,7.40848424 102.886775,7.37623853 L102.886775,7.37623853 Z' id='Shape' fill='%23000000'%3E%3C/path%3E%3Cpath d='M56.1913034,25.920367 C55.9722556,26.6492759 55.6190206,27.3308172 55.1497814,27.9299083 C54.7114768,28.4809174 54.1579818,28.9292929 53.5281281,29.243578 C52.2110644,29.8694724 50.6824961,29.8694724 49.3654324,29.243578 C48.7395057,28.9336335 48.1911209,28.4870218 47.760742,27.9366972 C47.301045,27.3388195 46.9559698,26.6609393 46.742968,25.9373394 C46.495773,25.1128125 46.3745329,24.2556418 46.3833545,23.3948624 C46.3740037,22.5302148 46.4975804,21.669269 46.7497531,20.8422018 C46.9636936,20.1158577 47.3086291,19.4348006 47.7675272,18.8326606 C48.2011576,18.2810922 48.7514936,17.83246 49.3790027,17.5189908 C50.6961754,16.8935806 52.2245257,16.8935806 53.5416984,17.5189908 C54.1674369,17.8292281 54.7157615,18.2757908 55.1463888,18.8258716 C55.6036776,19.4252683 55.948517,20.1026852 56.1641628,20.8252294 C56.4165173,21.6487646 56.5401098,22.5063855 56.5305614,23.3677064 C56.5486055,24.2309455 56.4341887,25.0918456 56.1913034,25.920367 L56.1913034,25.920367 Z M54.902123,21.2936697 C54.7318724,20.7139995 54.4620954,20.1683975 54.1048667,19.6812844 C53.7881209,19.2578671 53.3814124,18.9100568 52.9140711,18.6629358 C52.4547117,18.4223791 51.9431825,18.2987966 51.4247285,18.3031193 C50.9149493,18.2990408 50.412263,18.4227411 49.9625264,18.6629358 C49.5025532,18.9111647 49.1038238,19.2590902 48.7954789,19.6812844 C48.4471384,20.1637179 48.1864504,20.7037733 48.0253632,21.2766972 C47.8304445,21.9574655 47.7355889,22.6629947 47.7437791,23.3711009 C47.7347634,24.0797312 47.8319666,24.7857104 48.0321484,25.4655046 C48.202521,26.0438618 48.4698106,26.5890731 48.8226196,27.0778899 C49.135128,27.5026684 49.539982,27.8508769 50.00663,28.0962385 C50.4659398,28.336927 50.9775026,28.4605176 51.4959726,28.456055 C52.008891,28.4592624 52.514661,28.3356636 52.9683524,28.0962385 C53.431446,27.849806 53.832758,27.5016512 54.1421851,27.0778899 C54.4901007,26.5952029 54.7507511,26.0552256 54.9123007,25.4824771 C55.1069837,24.8016595 55.2018346,24.0961649 55.1938849,23.3880734 C55.2017485,22.6792753 55.1034023,21.9733002 54.902123,21.2936697 L54.902123,21.2936697 Z' id='Shape' fill='%23000000'%3E%3C/path%3E%3Cpath d='M67.9771265,22.536055 C67.7849808,22.991414 67.494128,23.398374 67.1255889,23.7275229 C66.7440918,24.0596662 66.2966389,24.3072347 65.8126605,24.453945 C65.2634083,24.6233341 64.6910728,24.7057969 64.1163704,24.6983486 L61.1512555,24.6983486 L61.1512555,29.5219266 L59.7942234,29.5219266 L59.7942234,17.2474312 L64.3877768,17.2474312 C64.9127032,17.242933 65.4353303,17.3173092 65.9381859,17.4680734 C66.392768,17.6054075 66.8134734,17.8366948 67.1730851,18.1469725 C67.5315297,18.4616971 67.8134835,18.8541011 67.997482,19.2943119 C68.3933428,20.3399157 68.3860867,21.4955106 67.9771265,22.536055 L67.9771265,22.536055 Z M66.2503033,19.1415596 C65.7171413,18.6827366 65.0249359,18.4522776 64.3233178,18.5 L61.1512555,18.5 L61.1512555,23.4559633 L64.1367259,23.4559633 C65.9822895,23.4559633 66.9050712,22.6254434 66.9050712,20.9644037 C66.9514424,20.2898408 66.7140761,19.6265629 66.2503033,19.1347706 L66.2503033,19.1415596 Z' id='Shape' fill='%23000000'%3E%3C/path%3E%3Cpolygon id='Shape' fill='%23000000' points='71.1424037 29.5253211 71.1424037 17.2474312 78.5551911 17.2474312 78.5551911 18.493211 72.5096135 18.493211 72.5096135 22.3527523 76.0107561 22.3527523 76.0107561 23.5985321 72.5096135 23.5985321 72.5096135 28.2795413 78.8198124 28.2795413 78.8198124 29.5253211 71.1424037 29.5253211'%3E%3C/polygon%3E%3Cpath d='M89.5098321,29.5253211 L84.1529482,21.123945 C84.0828349,21.0198471 84.0070673,20.8942508 83.9256454,20.747156 L83.6813796,20.3092661 C83.5999577,20.1621713 83.5207975,20.0252599 83.443899,19.8985321 C83.3820381,19.7968309 83.3265201,19.6914004 83.2776626,19.582844 L83.2776626,29.5253211 L81.9443786,29.5253211 L81.9443786,17.2474312 L83.2776626,17.2474312 L88.5870504,25.8049541 C88.6571637,25.911315 88.7329313,26.0369113 88.8143532,26.1817431 L89.058619,26.619633 C89.1400409,26.7667278 89.2192011,26.9036391 89.2960996,27.030367 C89.358006,27.1320424 89.4135255,27.2374757 89.462336,27.346055 L89.462336,17.2474312 L90.79562,17.2474312 L90.79562,29.5219266 L89.4996544,29.5219266 L89.5098321,29.5253211 Z' id='Shape' fill='%23000000'%3E%3C/path%3E%3Cpath d='M101.794365,27.6481651 C101.614779,28.0662826 101.342088,28.4378167 100.997108,28.7344037 C100.620385,29.0534717 100.186294,29.2977138 99.7181055,29.4540367 C99.1647693,29.6379509 98.5844631,29.7274558 98.00146,29.7188073 C97.2011368,29.7277732 96.4083533,29.563337 95.6775427,29.236789 C95.0107525,28.9422166 94.4027336,28.5293584 93.8828678,28.0181651 L94.7954719,26.9998165 C95.2416145,27.4292311 95.7493081,27.7896225 96.3017774,28.0690826 C96.8464614,28.3401103 97.4474455,28.4785194 98.0557413,28.4730275 C98.7462214,28.511211 99.4292056,28.3137399 99.9929045,27.9129358 C100.464701,27.5429103 100.728797,26.9676407 100.701954,26.3684404 C100.70524,26.1050555 100.658022,25.843483 100.562858,25.5978899 C100.456194,25.3464576 100.291025,25.1241684 100.081112,24.9495413 C99.8045716,24.7187732 99.5016459,24.5216464 99.1786853,24.3622936 C98.7251099,24.135808 98.2570403,23.9396294 97.7775497,23.7750459 C97.2032068,23.5804344 96.6441467,23.3432955 96.1050078,23.0655963 C95.7073087,22.8634309 95.3415136,22.6037985 95.0193821,22.2950459 C94.7546141,22.0365242 94.5506542,21.722295 94.4222881,21.3751376 C94.2936751,20.9972158 94.2316862,20.5998038 94.2390887,20.2006422 C94.2316769,19.7470041 94.3279908,19.2976722 94.5206729,18.8869725 C94.7055399,18.5036705 94.9718426,18.1653909 95.3009663,17.8957798 C95.6586019,17.6089841 96.0706082,17.3976683 96.5121174,17.2745872 C97.0217222,17.1292923 97.5496298,17.0584075 98.0794893,17.0641284 C98.7924833,17.0426363 99.5030012,17.1578582 100.172711,17.403578 C100.762472,17.644279 101.314379,17.9689791 101.811327,18.3676147 L100.936042,19.4368807 C100.528306,19.0872794 100.070982,18.8001466 99.5790097,18.5848624 C99.0690297,18.3814042 98.5231887,18.283259 97.9743194,18.2963303 C97.6068653,18.2890171 97.2405386,18.3394154 96.8886938,18.4456881 C96.6167299,18.5272515 96.3632288,18.6609912 96.1423261,18.8394495 C95.9575027,18.9921672 95.8130367,19.1880058 95.7216462,19.4097248 C95.6346426,19.6255014 95.5897307,19.8559491 95.5893356,20.0886239 C95.5872966,20.3191376 95.6228189,20.5484431 95.6945056,20.7675229 C95.782356,20.998859 95.9270341,21.2043246 96.1151855,21.3649541 C96.3863254,21.6100558 96.6914121,21.8147051 97.0210044,21.9725688 C97.509194,22.2047553 98.009792,22.409867 98.5205248,22.5869725 C99.1028479,22.7915642 99.6697358,23.0377325 100.216815,23.323578 C100.623019,23.5323344 100.993014,23.8051161 101.312618,24.1314679 C101.576275,24.4060688 101.77661,24.7351273 101.899534,25.0955046 C102.022879,25.4873767 102.082444,25.8965444 102.075949,26.3073394 C102.074721,26.7688462 101.978884,27.2251948 101.794365,27.6481651 L101.794365,27.6481651 Z' id='Shape' fill='%23000000'%3E%3C/path%3E%3Cpolygon id='Shape' fill='%23000000' points='112.908457 29.5253211 112.908457 23.7546789 106.801813 23.7546789 106.801813 29.5253211 105.444781 29.5253211 105.444781 17.2474312 106.801813 17.2474312 106.801813 22.4919266 112.908457 22.4919266 112.908457 17.2474312 114.265489 17.2474312 114.265489 29.5219266 112.908457 29.5219266'%3E%3C/polygon%3E%3Cpolygon id='Shape' fill='%23000000' points='118.112675 29.5253211 118.112675 17.2474312 119.469707 17.2474312 119.469707 29.5219266 118.112675 29.5219266'%3E%3C/polygon%3E%3Cpolygon id='Shape' fill='%23000000' points='124.687495 18.493211 124.687495 22.5666055 128.052934 22.5666055 128.052934 23.8123853 124.687495 23.8123853 124.687495 29.5287156 123.330463 29.5287156 123.330463 17.2474312 130.53291 17.2474312 130.53291 18.493211 124.697672 18.493211'%3E%3C/polygon%3E%3Cpolygon id='Shape' fill='%23000000' points='137.674291 18.493211 137.674291 29.5253211 136.317259 29.5253211 136.317259 18.493211 132.697376 18.493211 132.697376 17.2474312 141.283996 17.2474312 141.283996 18.493211'%3E%3C/polygon%3E%3C/g%3E%3C/g%3E%3C/svg%3E " alt="OpenShift logo"> </div> <div class="panel-content panel-content-centered"> <form method="GET" action="/oauth/start"> <input type="hidden" name="rd" value="/api/v1/label/code/values?start=1654519724&amp;end=1654523324"> <button type="submit" class="btn btn-lg btn-primary">Log in with OpenShift</button> </form> </div> </div> <footer> Secured with <a href="https://github.com/openshift/oauth-proxy#oauth2_proxy">OpenShift oauth-proxy</a> </footer> <script> if (window.location.hash) { (function() { var inputs = document.getElementsByName('rd'); for (var i = 0; i < inputs.length; i++) { inputs[i].value += window.location.hash; } })(); } </script> </body> </html>
API Performance
30s
....

Directly curling got 403 Forbidden too

% curl http://dittybopper-dittybopper.apps.qili-preserve-bm.qe-lrc.devcluster.openshift.com/api/datasources/proxy/1/api/v1/ -vvv
*   Trying 18.116.16.2:80...
* Connected to dittybopper-dittybopper.apps.qili-preserve-bm.qe-lrc.devcluster.openshift.com (18.116.16.2) port 80 (#0)
> GET /api/datasources/proxy/1/api/v1/ HTTP/1.1
> Host: dittybopper-dittybopper.apps.qili-preserve-bm.qe-lrc.devcluster.openshift.com
> User-Agent: curl/7.79.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< content-type: text/html; charset=utf-8
< date: Tue, 07 Jun 2022 03:17:08 GMT
< x-content-type-options: nosniff
< x-frame-options: deny
< x-xss-protection: 1; mode=block
< transfer-encoding: chunked
< set-cookie: 2ef43dab3ff5816969202745658a11e1=39dcf00a3bc9d975a786f43d955e0fec; path=/; HttpOnly
<
qiliRedHat commented 2 years ago

Found this difference, but not sure if this is related. Server Version: 4.11.0-0.nightly-2022-06-04-014713 Kubernetes Version: v1.24.0+bb9c2f1

% oc get ns dittybopper --show-labels
NAME          STATUS   AGE     LABELS
dittybopper   Active   5m17s   kubernetes.io/metadata.name=dittybopper,pod-security.kubernetes.io/audit-version=v1.24,pod-security.kubernetes.io/audit=restricted,pod-security.kubernetes.io/warn-version=v1.24,pod-security.kubernetes.io/warn=restricted

Server Version: 4.10.0 Kubernetes Version: v1.23.3+e419edf

% oc get ns dittybopper --show-labels                         
NAME          STATUS   AGE     LABELS
dittybopper   Active   4d12h   kubernetes.io/metadata.name=dittybopper

I manually deleted the extra labels of dittybopper on 4.11 cluster, or updated "restricted" to "privileged", later they were overwritten by the origin values again. Same after I directly edited the ns's yaml.

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    openshift.io/sa.scc.mcs: s0:c26,c5
    openshift.io/sa.scc.supplemental-groups: 1000660000/10000
    openshift.io/sa.scc.uid-range: 1000660000/10000
  creationTimestamp: "2022-06-07T01:45:19Z"
  labels:
    kubernetes.io/metadata.name: dittybopper
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: v1.24
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: v1.24
  name: dittybopper
  resourceVersion: "421958"
  uid: 0e7bbab7-f543-4cfe-9712-a02e85cc7c6c
spec:
  finalizers:
  - kubernetes
status:
  phase: Active
qiliRedHat commented 2 years ago

Ok, I got this diffence: Server Version: 4.11.0-0.nightly-2022-06-04-014713 Kubernetes Version: v1.24.0+bb9c2f1

% oc sa get-token prometheus-k8s -n openshift-monitoring
Command "get-token" is deprecated, and will be removed in the future version. Use oc create token instead.
error: could not find a service account token for service account "prometheus-k8s"

Server Version: 4.10.0 Kubernetes Version: v1.23.3+e419edf

% oc sa get-token prometheus-k8s -n openshift-monitoring
Command "get-token" is deprecated, and will be removed in the future version. Use oc create token instead.
eyJhbGciOiJSUzI1NiIsImtpZCI6ImNvZ2JIUTZ....
qiliRedHat commented 2 years ago

Got a workaround by replacing this line in deploy.sh

#export PROMETHEUS_BEARER=$($k8s_cmd sa get-token prometheus-k8s -n openshift-monitoring)
export PROMETHEUS_BEARER=$(oc create token prometheus-k8s -n openshift-monitoring --duration 240h)
rsevilla87 commented 2 years ago

Hi @qiliRedHat, we recently hit a similar but in some of our tools after k8s 1.24 rebase, the problem with the command create token is that is not available in "old" oc clients and it's not backwards compatible.

The following expression works fine and is backwards compatible export PROMETHEUS_BEARER=$(oc sa get-token -n openshift-monitoring prometheus-k8s || oc sa new-token -n openshift-monitoring prometheus-k8s)

rsevilla87 commented 2 years ago

Fixed with https://github.com/cloud-bulldozer/performance-dashboards/pull/40