Closed pavolloffay closed 1 month ago
Attention: Patch coverage is 73.39450%
with 29 lines
in your changes missing coverage. Please review.
Project coverage is 73.23%. Comparing base (
75de22c
) to head (3ac2873
). Report is 1 commits behind head on main.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
@pavolloffay With the route enabled in TempoStack query frontend, we create a SA for the query frontend along with the default SA for the stack. The query frontend SA doesn't have the required annotations for STS which causes the frontend pod to fail.
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
name: tmstack
spec:
storage:
secret:
name: aws-sts
type: s3
storageSize: 20Gi
resources:
total:
limits:
memory: 4Gi
cpu: 2000m
template:
queryFrontend:
jaegerQuery:
enabled: true
ingress:
type: route
oc get sa
NAME SECRETS AGE
builder 1 38m
default 1 38m
deployer 1 38m
tempo-tmstack 1 29m
tempo-tmstack-query-frontend 1 29m
% oc get sa tempo-tmstack-query-frontend -o yaml
apiVersion: v1
imagePullSecrets:
- name: tempo-tmstack-query-frontend-dockercfg-7j4cm
kind: ServiceAccount
metadata:
annotations:
serviceaccounts.openshift.io/oauth-redirectreference.primary: '{"kind":"OAuthRedirectReference","apiVersion":"v1","reference":{"kind":"Route","name":"tempo-tmstack-query-frontend"}}'
creationTimestamp: "2024-07-12T04:39:44Z"
labels:
app.kubernetes.io/component: query-frontend
app.kubernetes.io/instance: tmstack
app.kubernetes.io/managed-by: tempo-operator
app.kubernetes.io/name: tempo
name: tempo-tmstack-query-frontend
namespace: test-tempostack
ownerReferences:
- apiVersion: tempo.grafana.com/v1alpha1
blockOwnerDeletion: true
controller: true
kind: TempoStack
name: tmstack
uid: e98d32a4-b942-4e66-b624-db57574bc78f
resourceVersion: "128365"
uid: 95134fd8-0f67-41ac-baa0-88475240973c
secrets:
- name: tempo-tmstack-query-frontend-dockercfg-7j4cm
% oc get pods
NAME READY STATUS RESTARTS AGE
tempo-tmstack-compactor-67776478c9-h7rs5 1/1 Running 0 32m
tempo-tmstack-distributor-5f8d5f9655-x8qph 1/1 Running 0 32m
tempo-tmstack-ingester-0 1/1 Running 0 32m
tempo-tmstack-querier-859d965f78-d2p44 1/1 Running 0 32m
tempo-tmstack-query-frontend-599769fdd-xlpwz 2/3 CrashLoopBackOff 11 (69s ago) 32m
The issue is that the query-frontend uses a different service account when the route for the Jaeger UI is enabled.
tempo-tmstack 1 6m25s
tempo-tmstack-query-frontend 1 6m24s
@IshwarKanse could we require associating role with two SAs? Lokistack does it https://loki-operator.dev/docs/short_lived_tokens_authentication.md/#aws-secure-token-service
# Create a trust relationship file
cat > "$trust_rel_file" <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::${aws_account_id}:oidc-provider/${oidc_provider}"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"${oidc_provider}:sub": [
"system:serviceaccount:${tempostack_ns}:tempo-${tempostack_name}",
"system:serviceaccount:${tempostack_ns}:tempo-${tempostack_name}-query-frontend"
]
}
}
}
]
}
EOF
@pavolloffay Tested with the fix commit. The stack is running now. I have also added the AWS IAM policy script to our git repo. https://github.com/openshift/distributed-tracing-qe/blob/main/scripts/aws-sts-s3-access.sh
Resolves #553
https://issues.redhat.com/browse/TRACING-4227
Test instructions: