Closed NohaIhab closed 2 months ago
Thank you for reporting us your feedback!
The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5946.
This message was autogenerated
In #78 I did the exploration for Pipelines, it is the odd one out in our UATs because we cannot modify the container specs of pipelines runners directly, it has to be done through the kfp
sdk API. This is not the case for the rest of the UATs, where we can edit the container spec. Therefore my porposal for the Pipelines UATs is different from the rest.
I checked issue #27 and did some ad-hoc test runs in a proxy environment What we need to achieve is to make all the workloads' containers have the following proxy environment variables set:
HTTP_PROXY
HTTPS_PROXY
NO_PROXY
http_proxy
https_proxy
no_proxy
Here are my proposals:
To run the UATs behind proxy, we need to set the proxy environment variables listed above in 2 places:
This can be achieved by:
apiVersion: kubeflow.org/v1alpha1
kind: PodDefault
metadata:
name: add-proxy-envs
namespace: <test_namespace>
spec:
desc: Add proxy #settings
env:
- name: HTTP_PROXY
value: x.x.x.x:port
- name: http_proxy
value: x.x.x.x:port
- name: HTTPS_PROXY
value: x.x.x.x:port
- name: https_proxy
value: x.x.x.x:port
- name: NO_PROXY
value: <no_proxy values>
- name: no_proxy
value: <no_proxy values>
selector:
matchLabels:
add-proxy-envs: "true"
where test_namespace
is the same namespace in which the tests create the notebook/job, as well as the workloads
Configurations
section in the Notebook creation page.labels
of the job template here the following
add-proxy-envs: "true"
@@ -160,6 +160,9 @@
" \"metadata\": {\n",
" \"annotations\": {\n",
" \"sidecar.istio.io/inject\": \"false\"\n",
+ " },\n",
+ " \"labels\": {\n",
+ " \"add-proxy-envs\": \"true\"\n",
" }\n",
" },\n",
" \"spec\": {\n",
it is similar to the way we are disabling istio sidecar injection in the workloads run by the UATs.
It is what Phoevos proposed in #27 and there's a POC implementation in this branch for only passing the proxy envs to the driver job, but not to the workloads.
The ConfigMap is to be created from an .env
file that is passed to the tests with tox:
tox -e uats -- --env ./params.env
The missing part is how to consume the ConfigMap from the workloads. This can be done in a similar way to Proposal 1, by editing the container spec to use the ConfigMap:
envFrom:
- configMapRef:
name: use-proxy
optional: true
The drawback of this proposal is that it cannot be used for both driver and notebooks method of running the UATs. I prefer the PodDefaults because when running from notebook, we can easily select it from Configurations dropdown menu. In this case, we can create a PodDefault that consumes the ConfigMap and apply it to Notebook.
Apply the PodDefault in proposal 1 to the user namespace and have the Notebook/Job use it.
Use the kfp sdk to inject the proxy environment variables into the pipelines runner pods. Here is how we can do this by modifying the existing notebook:
PROXY_URL=os.environ['HTTP_PROXY']
NO_PROXY_URLS=os.environ['NO_PROXY']
PipelineTask
. For more details see the kfp sdk docs. The helper function is as follows:
def add_proxy(obj, proxy=PROXY_URL, no_proxy=NO_PROXY_URLS):
"""Adds the proxy env vars to the PipelineTask object."""
return (
obj.set_env_variable(name='http_proxy', value=proxy)
.set_env_variable(name='https_proxy', value=proxy)
.set_env_variable(name='HTTP_PROXY', value=proxy)
.set_env_variable(name='HTTPS_PROXY', value=proxy)
.set_env_variable(name='no_proxy', value=no_proxy)
.set_env_variable(name='NO_PROXY', value=no_proxy)
)
Modify the cell where the pipeline is defined to use the add_proxy
helper so that it adds the proxy env vars to all components:
@dsl.pipeline(name='condition-v2')
def condition_pipeline(text: str = 'condition test', force_flip_result: str = ''):
flip1 = add_proxy(flip_coin(force_flip_result=force_flip_result))
add_proxy(print_msg(msg=flip1.output))
with dsl.Condition(flip1.output == 'heads'):
flip2 = add_proxy(flip_coin())
add_proxy(print_msg(msg=flip2.output))
add_proxy(print_msg(msg=text))
As mentioned in #54, we need to have an option to run the proxy tests with tox. This can be tricky because based on my above proposals, the notebooks will need to be slightly modified to be able to run behind proxy.
We can have a different set of Notebooks with a _proxy
suffix for every UAT.
We can have .patch
files containing the diff needed to be done to the notebooks and if the proxy option is chosen with tox then the .patch
files are used to edit the notebooks before they are run.
??
Really great work @NohaIhab! I have some questions, although they 're not a blocker, probably more things we 'll need to consider during implementation.
Another option would be to always keep the label there and apply the podDefault only when tox has the proper configuration. But for pipelines, we 'd still need to edit the notebook with the proxy code
.patch
approach (which I prefer), will the user need to apply manually those changes? On that note, also for pipelines, will they need to uncomment the code this configuration will add?If yes, we 'll need instructions for those.
thanks @orfeas-k for the review
Your suggestion is a great idea actually, we can always have the label, and conditionally create the PodDefault if we are running the tests behind proxy. If the PodDefault is not created, the label will still be there but no proxy envs will be added to the workloads.
.patch
will be needed for pipelines only, the .patch
removes the need to have the commented code. In that case, the user will apply the .patch
and then the notebook is ready to run. -> instructions for running from the UI will be added for thatclosing the issue as now we have enough information to start the implementation thanks @orfeas-k for the review
Afterthoughts:
After discussing with the team during implementation, one suggestion was to have in the Notebooks an if condition to check whether the proxy envs are set. and based on that use the PodDefault for the workload or not. This can then eliminate the need for .patch
files and make it easier to automate the tests using the driver.
Context
This task to explore and estimate the effort to be able to run the UATs behind the proxy. We need to look at issues #27 and #54 for initial pointers.
What needs to get done
Definition of Done
the effort for running UATs behind proxy is well understood and broken down