Istio Upgrade 1.10 Tracking and Testing

[wg102]

This issue will be to track the Istio Upgrade tasks that need to be done (either proactive or reactive) to ensure the Prod Ugrade goes smoothly.

List of tests:

Kubeflow

• [x] Menu displays as expected
• [x] Changing a namespace
• [x] Create a new notebook server with the configuration Mounted Minio Storage
• [x] Create a new protected-b notebook
• [x] Try to connect non-prob data to prob notebook
• [x] Try to connect prob data to non-prob notebook
• [x] Stop and restart a notebook
• [ ] Create a new namespace (when in dev)
• [x] Add a contributor to the namespace
• [x] Delete a contributor to the namespace
• [x] Create a protected B notebook of RStudio. Try to download the content

  Pipelines

• [x] Run example pipeline notebook (aaw-contrib-jupyter-notebooks/kfp-basics/average_with_docker_components.ipynb)
• [x] Is available in French

Minio

• [x] Creation of a minio Bucket
• [x] Test MC connectivity in notebooks
• [x] #1289

List of things we had to fix in dev:

1. https://github.com/StatCan/daaas/issues/1229
2. https://github.com/StatCan/daaas/issues/1230

[chuckbelisle]

Thank you @wg102 .

@zachomedia & @sylus , do you have any feedback or anything to add to this list?

[chuckbelisle]

Being pushed to a later date as the prod cluster upgrade of Istio did not happen last weekend.

[rohank07]

• I've noticed there is a slight delay for the notebook servers and volumes to show up when switching in between namespaces and reloading the page.
• Vault pod in the vault-system namespace Readiness Probe is failing. Snippet of the logs: 2022-07-22T03:39:46.607Z [ERROR] expiration: shutting down, not attempting further revocation of lease: lease_id=minio_gateway_standard_ro/keys/profile-sean-stilwell/FTTkDN2ymbqgfthTLm0VSAbd 2022-07-22T03:39:46.607Z [ERROR] expiration: core context canceled, not attempting further revocation of lease: lease_id=minio_gateway_standard_ro/keys/profile-ssc-poc-cfia-nrcan/Jcw5XlQvTWr1RfQTUJQhTNIR Mounting minio seems intermittent as well.

[rohank07]

The Vault pod is throwing err: Access Denied." 2022-07-22T18:18:15.144Z [ERROR] expiration: failed to revoke lease: lease_id=minio_gateway_standard_ro/keys/profile-agzero-poc/ppowm545UuB79xmrQCoMrOmB error="failed to revoke entry: resp: Some of the vault-agent containers in user namespaces throw: (ex. joseph-kuchar/api-testing-0) [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=4m55.16s │ 2022-07-25T14:56:00.014Z [INFO] auth.handler: authenticating The aaw-sedar/sedar-prod-0 vault agent container does not any errors.