Incorrect templating of shim container

[surskitt]

I am seeing the error below when trying to run launchpad with a private repo:

unable to run install step \"Deploy Kubelet\": unable to reconcile state of Kubelet component: unable to reconcile persistent state: image \"docker.io/repo.shanedabes.com/mirantis/ucp-containerd-shim-process:3.3.8\": not found

It looks like the templating of the image isn't working properly, the above should be repo.shanedabes.com/mirantis... instead of docker.io/repo.shanedabes.com/mirantis. This starts happening when trying to install MKE versions 3.3.8 and up, it's not seen on versions lower than that.

I am on the newest stable release of launchpad.

Thanks!

[surskitt]

Any update on whether this can be looked at? It's preventing us using launchpad to upgrade to versions 3.3.8 and up like mentioned.

[kke]

What do you have in imageRepo: in config? I have a hunch it may have some problems if it doesn't include an "organization" part (eg. just repo.example.com instead of repo.example.com/mirantis).

Could you find lines containing retag in the debug log (can be found in ~/.mirantis-launchpad/cluster/<cluster_name>/apply.log) or viewed during apply with --debug and paste a couple of them for example? (something like DEBU 00:00:00 retag mirantis/foofoo --> customrepo/foofoo)

[surskitt]

The imagerepo line is like the below:

"imageRepo": "nexus-rm.private.co.uk:9026/mirantis"

[surskitt]

Here's some examples of the retag happening, including the shim process container:

time="10 Jun 21 14:13 BST" level=debug msg="[ssh] cmpwkr302.devtech.private.co.uk:22: retag nexus-rm.private.co.uk:9026/mirantis/ucp-calico-cni:3.3.9 --> nexus-rm.private.co.uk:9026/mirantis/ucp-calico-cni:3.3.9"
time="10 Jun 21 14:13 BST" level=debug msg="[ssh] cmpwkr301.devtech.private.co.uk:22: retag nexus-rm.private.co.uk:9026/mirantis/ucp-compose:3.3.9 --> nexus-rm.private.co.uk:9026/mirantis/ucp-compose:3.3.9"
time="10 Jun 21 14:13 BST" level=debug msg="[ssh] cmpwkr300.devtech.private.co.uk:22: retag nexus-rm.private.co.uk:9026/mirantis/ucp-containerd-shim-process:3.3.9 --> nexus-rm.private.co.uk:9026/mirantis/ucp-containerd-shim-process:3.3.9"
time="10 Jun 21 14:13 BST" level=debug msg="[ssh] cmpwkr302.devtech.private.co.uk:22: retag nexus-rm.private.co.uk:9026/mirantis/ucp-calico-kube-controllers:3.3.9 --> nexus-rm.private.co.uk:9026/mirantis/ucp-calico-kube-controllers:3.3.9"
time="10 Jun 21 14:13 BST" level=debug msg="[ssh] cmpwkr301.devtech.private.co.uk:22: retag nexus-rm.private.co.uk:9026/mirantis/ucp-containerd-shim-process:3.3.9 --> nexus-rm.private.co.uk:9026/mirantis/ucp-containerd-shim-process:3.3.9"
time="10 Jun 21 14:13 BST" level=debug msg="[ssh] cmpwkr300.devtech.private.co.uk:22: retag nexus-rm.private.co.uk:9026/mirantis/ucp-controller:3.3.9 --> nexus-rm.private.co.uk:9026/mirantis/ucp-controller:3.3.9"
time="10 Jun 21 14:13 BST" level=debug msg="[ssh] cmpwkr302.devtech.private.co.uk:22: retag nexus-rm.private.co.uk:9026/mirantis/ucp-calico-node:3.3.9 --> nexus-rm.private.co.uk:9026/mirantis/ucp-calico-node:3.3.9"

[kke]

Ok, that looks correct. I think this may be a problem in the MKE itself. Investigation on-going, reporting back asap.

[kke]

@vikramhh / @ankolesnikov any follow-up on this?

[vikramhh]

PR - https://github.com/Mirantis/orca/pull/19238

@AlexShu88 ^^

[AlexShu88]

@vikramhh , not sure why after running this code https://github.com/Mirantis/orca/blob/7d07ecd9aa0a3d6a3e6b2cef0092810e61067f69/agent2/reconciler/image_util.go#L60 the docker-io prefix was automatically added before the imag name.

[AlexShu88]

@byDimasik, would you please confirm if this function should be modified to check if this hostname is included in the Org. https://github.com/Mirantis/orca/blob/7d07ecd9aa0a3d6a3e6b2cef0092810e61067f69/utils/imageref.go#L22

func LocalPlatformImageRef(repoName, version, org string) string { var ( tag = version repoNamespace = config.ImageOrg ) if org != "" {
judge if org has host name, if ture replacing the host name with docker.io otherwise repoNamespace = org

    }
if runtime.GOOS == constants.OSWindows {        windowsRepoName, ok := windowsRepoNames[repoName]       if ok {         repoName = windowsRepoName      }   }
return fmt.Sprintf("%s/%s:%s", repoNamespace, repoName, tag)}

[byDimasik]

Yeah. Seems it should work. Probably, just need one more check for repoNamespace before the return