Open nishant-dash opened 5 days ago
syncronize-issues-to-jira[bot]
commented
5 days ago Thank you for reporting us your feedback!
The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5957.
This message was autogenerated
NohaIhab
commented
17 hours ago Hi @nishant-dash ,
In the meantime, we are prioritizing this issue and will try to reproduce it.
nishant-dash
commented
16 hours ago for
---------------------------------------------------------------------------
AssertionError Traceback (most recent call last)
Cell In[14], line 4
1 # fetch KFP experiment to ensure it exists
2 client.get_experiment(experiment_name=EXPERIMENT_NAME)
----> 4 assert_run_succeeded(client, run.run_id)File /opt/conda/lib/python3.11/site-packages/tenacity/init.py:336, in BaseRetrying.wraps.
File /opt/conda/lib/python3.11/site-packages/tenacity/init.py:475, in Retrying.call(self, fn, *args, **kwargs) 473 retry_state = RetryCallState(retry_object=self, fn=fn, args=args, kwargs=kwargs) 474 while True: --> 475 do = self.iter(retry_state=retry_state) 476 if isinstance(do, DoAttempt): 477 try:
File /opt/conda/lib/python3.11/site-packages/tenacity/init.py:376, in BaseRetrying.iter(self, retry_state) 374 result = None 375 for action in self.iter_state.actions: --> 376 result = action(retry_state) 377 return result
File /opt/conda/lib/python3.11/site-packages/tenacity/init.py:418, in BaseRetrying._post_stop_check_actions.
File /opt/conda/lib/python3.11/site-packages/tenacity/init.py:185, in RetryError.reraise(self) 183 def reraise(self) -> t.NoReturn: 184 if self.last_attempt.failed: --> 185 raise self.last_attempt.result() 186 raise self
File /opt/conda/lib/python3.11/concurrent/futures/_base.py:449, in Future.result(self, timeout) 447 raise CancelledError() 448 elif self._state == FINISHED: --> 449 return self.__get_result() 451 self._condition.wait(timeout) 453 if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:
File /opt/conda/lib/python3.11/concurrent/futures/_base.py:401, in Future.__get_result(self) 399 if self._exception: 400 try: --> 401 raise self._exception 402 finally: 403 # Break a reference cycle with the exception in self._exception 404 self = None
File /opt/conda/lib/python3.11/site-packages/tenacity/init.py:478, in Retrying.call(self, fn, *args, *kwargs) 476 if isinstance(do, DoAttempt): 477 try: --> 478 result = fn(args, **kwargs) 479 except BaseException: # noqa: B902 480 retry_state.set_exception(sys.exc_info()) # type: ignore[arg-type]
Cell In[13], line 9, in assert_run_succeeded(client, run_id) 7 """Wait for the run to complete successfully.""" 8 status = client.get_run(run_id).state ----> 9 assert status == "SUCCEEDED", f"KFP run in {status} state."
AssertionError: KFP run in RUNNING state.
2. but isn't this for a cluster internal service ? maybe kfp? (perhaps the no proxy needs tweaking at the containerd level?)
```console
Network is unreachable')': /simple/kfp/
DnPlas
commented
14 hours ago @nishant-dash also for reproducing the issue, could you please tell us which method for running the UATs are you using? It was not clear from the issue description.
Bug Description
running kfp_v2 integration test from https://github.com/canonical/charmed-kubeflow-uats/tree/main/tests, commit [0] experiment fails on
To Reproduce
In a kf 1.8 env behind proxy, run the kfp_v2 uats
Environment
Relevant Log Output
Additional Context
No response