istio-pilot goes to blocked from ingress relation version not found for app

[kimwnasptd]

Bug Description

I tried to deploy CKF 1.8. After waiting a bit the jupyter-ui and the istio-pilot Charms went into error state.

The interesting error logs in the istio-pilot charm were the following:

2024-06-22T09:49:44.735Z [container-agent] 2024-06-22 09:49:44 INFO juju-log Execution handled 1 errors.  See logs for details.
2024-06-22T09:49:44.740Z [container-agent] 2024-06-22 09:49:44 INFO juju-log Handled error 0/1: WaitingStatus('List of <ops.model.Relation ingress:12> versions not fo
und for apps: jupyter-ui')

To Reproduce

Deploy CKF 1.8 from the bundle

Although this looks like a transient issue.

Environment

CKF 1.8

Relevant Log Output

2024-06-22T09:49:44.735Z [container-agent] 2024-06-22 09:49:44 INFO juju-log Execution handled 1 errors.  See logs for details.
2024-06-22T09:49:44.740Z [container-agent] 2024-06-22 09:49:44 INFO juju-log Handled error 0/1: WaitingStatus('List of <ops.model.Relation ingress:12> versions not fo
und for apps: jupyter-ui')
2024-06-22T09:49:45.062Z [container-agent] 2024-06-22 09:49:45 INFO juju.worker.uniter.operation runhook.go:186 ran "config-changed" hook (via hook dispatching script
: dispatch)
2024-06-22T09:49:45.071Z [container-agent] 2024-06-22 09:49:45 INFO juju.worker.uniter resolver.go:76 reboot detected; triggering implicit start hook to notify charm
2024-06-22T09:49:45.547Z [container-agent] 2024-06-22 09:49:45 INFO juju-log Running legacy hooks/start.

Additional Context

The jupyter-ui failed because of an unrelated error with JuJu 3.5 https://bugs.launchpad.net/juju/+bug/2067636, so there's not much to debug on that side. But might be interesting for understanding the execution path.

2024-06-22T09:50:22.641Z [container-agent] 2024-06-22 09:50:22 INFO juju-log Reconcile completed successfully
2024-06-22T09:50:22.676Z [container-agent] 2024-06-22 09:50:22 INFO juju-log Pebble plan updated with new configuration, replaning
2024-06-22T09:50:22.714Z [container-agent] 2024-06-22 09:50:22 ERROR juju-log Uncaught exception while in charm code:
2024-06-22T09:50:22.714Z [container-agent] Traceback (most recent call last):
2024-06-22T09:50:22.714Z [container-agent]   File "./src/charm.py", line 484, in <module>
2024-06-22T09:50:22.714Z [container-agent]     main(JupyterUI)
2024-06-22T09:50:22.714Z [container-agent]   File "/var/lib/juju/agents/unit-jupyter-ui-0/charm/venv/ops/main.py", line 439, in main
2024-06-22T09:50:22.714Z [container-agent]     framework.reemit()
2024-06-22T09:50:22.714Z [container-agent]   File "/var/lib/juju/agents/unit-jupyter-ui-0/charm/venv/ops/framework.py", line 851, in reemit
2024-06-22T09:50:22.714Z [container-agent]     self._reemit()
2024-06-22T09:50:22.714Z [container-agent]   File "/var/lib/juju/agents/unit-jupyter-ui-0/charm/venv/ops/framework.py", line 930, in _reemit
2024-06-22T09:50:22.714Z [container-agent]     custom_handler(event)
2024-06-22T09:50:22.714Z [container-agent]   File "./src/charm.py", line 465, in main
2024-06-22T09:50:22.714Z [container-agent]     self._update_spawner_ui_config()
2024-06-22T09:50:22.714Z [container-agent]   File "./src/charm.py", line 387, in _update_spawner_ui_config
2024-06-22T09:50:22.714Z [container-agent]     self._upload_jwa_file_to_container(jwa_content)
2024-06-22T09:50:22.714Z [container-agent]   File "./src/charm.py", line 358, in _upload_jwa_file_to_container
2024-06-22T09:50:22.714Z [container-agent]     self.container.push(
2024-06-22T09:50:22.714Z [container-agent]   File "/var/lib/juju/agents/unit-jupyter-ui-0/charm/venv/ops/model.py", line 2073, in push
2024-06-22T09:50:22.714Z [container-agent]     self._pebble.push(str(path), source, encoding=encoding,
2024-06-22T09:50:22.714Z [container-agent]   File "/var/lib/juju/agents/unit-jupyter-ui-0/charm/venv/ops/pebble.py", line 2012, in push
2024-06-22T09:50:22.714Z [container-agent]     self._raise_on_path_error(typing.cast('_FilesResponse', resp), path)
2024-06-22T09:50:22.714Z [container-agent]   File "/var/lib/juju/agents/unit-jupyter-ui-0/charm/venv/ops/pebble.py", line 1964, in _raise_on_path_error
2024-06-22T09:50:22.714Z [container-agent]     raise PathError(error['kind'], error['message'])
2024-06-22T09:50:22.714Z [container-agent] ops.pebble.PathError: permission-denied - cannot create directory: mkdir /etc/config: permission denied
2024-06-22T09:50:22.983Z [container-agent] 2024-06-22 09:50:22 ERROR juju.worker.uniter.operation runhook.go:180 hook "leader-elected" (via hook dispatching script: d
ispatch) failed: exit status 1
2024-06-22T09:50:22.984Z [container-agent] 2024-06-22 09:50:22 INFO juju.worker.uniter resolver.go:180 awaiting error resolution for "leader-elected" hook

[syncronize-issues-to-jira[bot]]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5893.

This message was autogenerated

[kimwnasptd]

In that case also, the Istio charm ended up not creating any Gateway or VirtualService CRs, which resulted to 404 access when trying to connect to the Istio IngressGateway (to see the CentralDashboard)

[DnPlas]

@kimwnasptd could you also please share your k8s (version and type) and the juju version with revision included?