search

canonical / pipelines-rocks

ROCKs for Kubeflow Pipelines
Apache License 2.0
1 stars 7 forks source link

pebble cannot start the service of the api-server container image #22

Closed DnPlas closed 1 year ago

DnPlas commented 1 year ago

It seems like this issue is replicated on the container image generated from the api-server rock. When attempting to deploy a charm that uses this image, we get the following:

unit-kfp-api-0: 07:31:50 ERROR unit.kfp-api/0.juju-log kfp-viz:3: Failed to handle <RelationChangedEvent via KfpApiOperator/on/kfp_viz_relation_changed[49]> with error: Waiting for kfp-viz relation data
unit-kfp-api-0: 07:31:54 ERROR unit.kfp-api/0.juju-log kfp-viz:3: Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-kfp-api-0/charm/venv/charmed_kubeflow_chisme/pebble/_update_layer.py", line 31, in update_layer
    container.replan()
  File "/var/lib/juju/agents/unit-kfp-api-0/charm/venv/ops/model.py", line 1894, in replan
    self._pebble.replan_services()
  File "/var/lib/juju/agents/unit-kfp-api-0/charm/venv/ops/pebble.py", line 1658, in replan_services
    return self._services_action('replan', [], timeout, delay)
  File "/var/lib/juju/agents/unit-kfp-api-0/charm/venv/ops/pebble.py", line 1739, in _services_action
    raise ChangeError(change.err, change)
ops.pebble.ChangeError: cannot perform the following tasks:
- Start service "apiserver" (cannot start service: exited quickly with code 255)
----- Logs from task 0 -----
2023-07-13T07:31:54Z INFO Most recent service output:
    I0713 07:31:54.227121      15 client_manager.go:160] Initializing client manager
    I0713 07:31:54.227997      15 config.go:57] Config DBConfig.ExtraParams not specified, skipping

    (Error 1364: Field 'Description' doesn't have a default value) 
    [2023-07-13 07:31:54]  
    F0713 07:31:54.638256      15 client_manager.go:412] Failed to check if Minio bucket exists. Error: Get "http://minio.test-charm-ju8v:9000/mlpipeline/?location=": dial tcp 10.152.183.165:9000: connect: connection refused
2023-07-13T07:31:54Z ERROR cannot start service: exited quickly with code 255
-----

unit-kfp-api-0: 07:31:54 ERROR unit.kfp-api/0.juju-log kfp-viz:3: Failed to handle <RelationChangedEvent via KfpApiOperator/on/kfp_viz_relation_changed[52]> with error: Failed to replan
unit-kfp-api-0: 07:34:08 ERROR unit.kfp-api/0.juju-log Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-kfp-api-0/charm/venv/charmed_kubeflow_chisme/pebble/_update_layer.py", line 31, in update_layer
    container.replan()
  File "/var/lib/juju/agents/unit-kfp-api-0/charm/venv/ops/model.py", line 1894, in replan
    self._pebble.replan_services()
  File "/var/lib/juju/agents/unit-kfp-api-0/charm/venv/ops/pebble.py", line 1658, in replan_services
    return self._services_action('replan', [], timeout, delay)
  File "/var/lib/juju/agents/unit-kfp-api-0/charm/venv/ops/pebble.py", line 1739, in _services_action
    raise ChangeError(change.err, change)
ops.pebble.ChangeError: cannot perform the following tasks:
- Start service "apiserver" (cannot start service: exited quickly with code 255)
----- Logs from task 0 -----
2023-07-13T07:34:08Z INFO Most recent service output:
    I0713 07:34:08.309697      21 client_manager.go:160] Initializing client manager
    I0713 07:34:08.309780      21 config.go:57] Config DBConfig.ExtraParams not specified, skipping
    F0713 07:34:08.398224      21 client_manager.go:412] Failed to check if Minio bucket exists. Error: Get "http://minio.test-charm-ju8v:9000/mlpipeline/?location=": dial tcp 10.152.183.165:9000: connect: connection refused
2023-07-13T07:34:08Z ERROR cannot start service: exited quickly with code 255
-----

See here for a complete CI run.

DnPlas commented 1 year ago

It seems like the charm that used the container image generated from this Rockcraft project has a workaround for this issue. Keeping the fix in the charm code and not in the Rockcraft project as it is a workaround.