Cassandra pod is not pulling backrest-sidecar container image

[petocervik]

I am trying to deploy demo cassandra cluster, but cassandra-demo-dc1-rack1-0 pod has problem to run its backrest-sidecar container. There is problem to pull image from google container repository with following pod log:

Events:
  Type     Reason     Age              From               Message
  ----     ------     ----             ----               -------
  Normal   Scheduled  17s              default-scheduler  Successfully assigned cassandra/cassandra-demo-dc1-rack1-0 to vm-k8s-master.vmnet.local
  Normal   Pulled     16s              kubelet            Container image "cassandra:4.0" already present on machine
  Normal   Created    16s              kubelet            Created container base-config-builder
  Normal   Started    16s              kubelet            Started container base-config-builder
  Normal   Pulled     15s              kubelet            Container image "datastax/cass-config-builder:1.0.4" already present on machine
  Normal   Created    15s              kubelet            Created container config-builder
  Normal   Started    15s              kubelet            Started container config-builder
  Normal   Created    8s               kubelet            Created container bootstrap
  Normal   Pulled     8s               kubelet            Container image "ghcr.io/cscetbon/casskop-bootstrap:0.1.10" already present on machine
  Normal   Started    8s               kubelet            Started container bootstrap
  Normal   Pulling    7s               kubelet            Pulling image "gcr.io/cassandra-operator/instaclustr-icarus:1.1.0"
  Warning  Failed     7s               kubelet            Failed to pull image "gcr.io/cassandra-operator/instaclustr-icarus:1.1.0": rpc error: code = Unknown desc = Error response from daemon: Head "https://gcr.io/v2/cassandra-operator/instaclustr-icarus/manifests/1.1.0": denied: Project cassandra-operator has been deleted.
  Warning  Failed     7s               kubelet            Error: ErrImagePull
  Normal   Pulled     6s (x2 over 7s)  kubelet            Container image "cassandra:4.0" already present on machine
  Normal   Created    6s (x2 over 7s)  kubelet            Created container cassandra
  Normal   Started    6s (x2 over 7s)  kubelet            Started container cassandra
  Normal   BackOff    5s (x2 over 6s)  kubelet            Back-off pulling image "gcr.io/cassandra-operator/instaclustr-icarus:1.1.0"
  Warning  Failed     5s (x2 over 6s)  kubelet            Error: ImagePullBackOff
  Warning  BackOff    5s               kubelet            Back-off restarting failed container

I also tried to pull the docker image manually, but I got similar response:

# docker pull gcr.io/cassandra-operator/instaclustr-icarus:1.1.0
Error response from daemon: Head "https://gcr.io/v2/cassandra-operator/instaclustr-icarus/manifests/1.1.0": denied: Project cassandra-operator has been deleted.

For me it looks like instaclustr-icarus image is not available anymore. I am not sure whether I missed some configuration or icarus docker image was removed from repository.

[cscetbon]

Hey, @smiklosovic any idea what's going on ?

[smiklosovic]

I think that is similar to this

https://github.com/instaclustr/cassandra-operator/issues/413

I am not sure what happened with gcr but you are welcome to build the operator and sidecar yourself and you can self-manage it. I just updated the scripts to be buildable.

We, as a company, are not actively maintaining that operator.

[cscetbon]

Yeah so I would need to do https://github.com/instaclustr/cassandra-operator/issues/413#issuecomment-1386885381 and store those images in my registry as well :/ Okay will see when I got sometimes or if someone else does. Thanks

[cscetbon]

@AKamyshnikova is it something you wanna work on ?

[AKamyshnikova]

@cscetbon Like creating a job that will build and store this image in your registry? I can do something like that, but I will have to find time for this as well.

[cscetbon]

yes @AKamyshnikova a simple Github workflow that does clone the repo, runs the command and pushes the image to casskop registry. Then we'll have to do another that rename the docker image used by default in the code as it's no longer available

[cscetbon]

Actually @AKamyshnikova I'm working on it 😉

[cscetbon]

I got everything building but end to end tests are failing and it's due to the fact that the jar built for icarus is seen as corrupted. I asked Štefan to help on this as I used the available scripts in their repo