search

canonical / kfp-operators

Kubeflow Pipelines Operators
Apache License 2.0
2 stars 12 forks source link

[kfp-persistence] config-change hook failed with ValueError #574

Open rgildein opened 1 month ago

rgildein commented 1 month ago

Bug Description

After deploying kfp-persistence from 2.2/stable channel, the config-changed hook failed with ValueError.

To Reproduce

  1. juju add-model kubeflow
  2. juju deploy kubeflow --channel 1.9/stable --trust

Environment

Juju 3.5.3 MicroK8s v1.30.5 revision 7235

Relevant Log Output

unit-kfp-persistence-0: 12:55:12 INFO juju.worker.uniter awaiting error resolution for "config-changed" hook
unit-kfp-persistence-0: 12:56:28 INFO juju.worker.uniter awaiting error resolution for "config-changed" hook
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed Traceback (most recent call last):
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed   File "./src/charm.py", line 31, in <module>
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed     from components.sa_token_component import SaTokenComponent
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed   File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/src/components/sa_token_component.py", line 11, in <module>
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed     import kubernetes
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed   File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/venv/kubernetes/__init__.py", line 21, in <module>
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed     from . import dynamic
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed   File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/venv/kubernetes/dynamic/__init__.py", line 15, in <module>
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed     from .client import *  # NOQA
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed   File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/venv/kubernetes/dynamic/client.py", line 18, in <module>
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed     from kubernetes import watch
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed   File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/venv/kubernetes/watch/__init__.py", line 15, in <module>
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed     from .watch import Watch
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed   File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/venv/kubernetes/watch/watch.py", line 16, in <module>
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed     import pydoc
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed   File "/usr/lib/python3.8/pydoc.py", line 73, in <module>
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed     import sysconfig
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed   File "<frozen importlib._bootstrap_external>", line 844, in exec_module
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed   File "<frozen importlib._bootstrap_external>", line 976, in get_code
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed   File "<frozen importlib._bootstrap_external>", line 640, in _compile_bytecode
unit-kfp-persistence-0: 12:56:28 WARNING unit.kfp-persistence/0.config-changed ValueError: bad marshal data (invalid reference)
unit-kfp-persistence-0: 12:56:29 ERROR juju.worker.uniter.operation hook "config-changed" (via hook dispatching script: dispatch) failed: exit status 1
unit-kfp-persistence-0: 12:56:29 INFO juju.worker.uniter awaiting error resolution for "config-changed" hook
unit-kfp-persistence-0: 12:59:12 INFO juju.worker.uniter awaiting error resolution for "config-changed" hook
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed Traceback (most recent call last):
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed   File "./src/charm.py", line 31, in <module>
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed     from components.sa_token_component import SaTokenComponent
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed   File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/src/components/sa_token_component.py", line 11, in <module>
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed     import kubernetes
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed   File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/venv/kubernetes/__init__.py", line 21, in <module>
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed     from . import dynamic
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed   File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/venv/kubernetes/dynamic/__init__.py", line 15, in <module>
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed     from .client import *  # NOQA
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed   File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/venv/kubernetes/dynamic/client.py", line 18, in <module>
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed     from kubernetes import watch
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed   File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/venv/kubernetes/watch/__init__.py", line 15, in <module>
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed     from .watch import Watch
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed   File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/venv/kubernetes/watch/watch.py", line 16, in <module>
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed     import pydoc
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed   File "/usr/lib/python3.8/pydoc.py", line 73, in <module>
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed     import sysconfig
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed   File "<frozen importlib._bootstrap_external>", line 844, in exec_module
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed   File "<frozen importlib._bootstrap_external>", line 976, in get_code
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed   File "<frozen importlib._bootstrap_external>", line 640, in _compile_bytecode
unit-kfp-persistence-0: 12:59:12 WARNING unit.kfp-persistence/0.config-changed ValueError: bad marshal data (invalid reference)
unit-kfp-persistence-0: 12:59:13 ERROR juju.worker.uniter.operation hook "config-changed" (via hook dispatching script: dispatch) failed: exit status 1
unit-kfp-persistence-0: 12:59:13 INFO juju.worker.uniter awaiting error resolution for "config-changed" hook

Additional Context

No response

syncronize-issues-to-jira[bot] commented 1 month ago

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-6384.

This message was autogenerated

orfeas-k commented 1 month ago

@rgildein Thank you for reporting this. We 'll need more information in order to assess the issue. Some questions that would help us with that:

rgildein commented 1 month ago

Did you make a specific configuration or does this refer to the config-changed hook run during deployment?

No, I did not.

Was this a one time thing or have you observed it consisently?

It happened two times, but not I could not reproduce it.

Have you been able to reproduce this with Juju 3.4? We do not support 3.5.x right now, as we haven't tested our charms with it.

I did not try to use Juju 3.4, on the third try it worked fine with 3.5. I deployed it again now (doing some testing) and it worked fine again.