Open iblancasa opened 5 months ago
It seems something changed recently in the image and that is breaking the operator integration
same issues here, we use jaeger-operator v1.49.0
last successful spark job is 20 days ago
NAME COMPLETIONS DURATION AGE
jaeger-operator-jaeger-cassandra-schema-job 1/1 37s 601d
jaeger-operator-jaeger-spark-dependencies-28491835 1/1 32s 22d
jaeger-operator-jaeger-spark-dependencies-28493275 1/1 36s 21d
jaeger-operator-jaeger-spark-dependencies-28494715 1/1 37s 20d
jaeger-operator-jaeger-spark-dependencies-28523515 0/1 12h 12h
operator's spark job has no tag on the image, so "latest" is used as a fallback.
Containers:
jaeger-operator-jaeger-spark-dependencies:
Image: ghcr.io/jaegertracing/spark-dependencies/spark-dependencies
Port: <none>
Host Port: <none>
the image that is used can be found here.
sadly there is only 1 tag which was overwritten 20 days ago, that makes any Workaround impossible 😔 any ideas ? In my opinion at least the old image should be provided..
We might open an issue in that repository.
as a workaround, we pin the spark-dependencies to the old untaggt Image sha256:683963b95bafb0721f3261a49c368c7bdce4ddcb04a23116c45068d254c5ec11
we use the Helm Values of the jaeger-operator to override the DockerImage of dependencies
in storage
section:
jaeger:
create: true
spec:
strategy: production
storage:
type: cassandra
options:
cassandra:
servers: xxx
keyspace: jaeger
username: xxx
password: xxx
dependencies:
image: ghcr.io/jaegertracing/spark-dependencies/spark-dependencies@sha256:683963b95bafb0721f3261a49c368c7bdce4ddcb04a23116c45068d254c5ec11
However, the current image is broken and should be fixed. In my opinion, the jaeger-operator itself should also pin its own dependencies to avoid this kind of Errors in Production.
@rriverak would you like to send a PR?
@iblancasa I'm not sure. We can solve this on several levels... which solution are we looking for? then i cloud provide a PR accordingly.
Fix Spark dependencies The 'latest' image should work again with the operator and a better versioning should take place. After that adapt the new spark-dependency versioning to the operator-logic.
Adjust jaeger-operator The Image could be set strictly in the Flags as default. btw, the problem with 'latest' on Spark dependencies seems to be known
Adjust Helm Charts The Image could be set in the jaeger-operator-chart Values. https://github.com/jaegertracing/helm-charts/blob/d4e163e1311df2596aabb7db1aedc05169a71396/charts/jaeger-operator/values.yaml#L40 and in the jaeger-chart Values https://github.com/jaegertracing/helm-charts/blob/d4e163e1311df2596aabb7db1aedc05169a71396/charts/jaeger/values.yaml#L737
What is our path?
I would be happy if spark-dependency shows initiative here and fixes the problems with the Image and switches to a proper Versioning. If this does not happen, then one of the remaining two solutions must do the job.
I would prefer the Fix Spark dependencies
option. After that one, set the version in the Jaeger operator. The third one is not a real solution since a lot of people are not using Helm to install the operator.
What happened?
When running the
cassandra-spark
E2E test, the pod from the spark job fails:Steps to reproduce
Run the test
Expected behavior
.
Relevant log output
No response
Screenshot
No response
Additional context
No response
Jaeger backend version
No response
SDK
No response
Pipeline
No response
Stogage backend
No response
Operating system
No response
Deployment model
No response
Deployment configs
No response