search

sassoftware / viya4-deployment

This project contains Ansible code that creates a baseline in an existing Kubernetes environment for use with the SAS Viya Platform, generates the manifest for an order, and then can also deploy that order into the Kubernetes environment specified.
Apache License 2.0
71 stars 64 forks source link

SAS Viya 4 on AKS: Pods Crashing/Waiting to start after successful Installation #308

Closed vaibhav90 closed 2 years ago

vaibhav90 commented 2 years ago

Hello,

TLDR: I have followed the procedures provided on the SAS Github repositories & documentation pages to provision Azure based infrastructure and install SAS Viya 4. All these steps complete successfully, however after installation the Kubernetes pods start crashing & I need help to understand why this occurs.

Context

  1. I have followed the steps provided in viya4-iac-azure to provision the infrastructure on Azure. All the steps here are successfully executed. (I have attached the terraform.tfvars below)
  2. I have followed SAS-Viya-ARK to validate the infrastructure. We have as well received the official sizing from SAS to choose the machines based on our requirements (attached the report below).
  3. I mounted the NetApp volume to the JUMP host in the specified path and its successfully mounted. I then proceeded to install baseline, viya. The installation as well successfully completes. 0 failures, 0 unreachable. (I have attached the ansible vars file below)
  4. However, after installation, a lot of pods crash. I have attached a screenshot below for the command - kubectl get pods.

How did I try to debug the issue ?

  1. Went through all the issues in this repository and the viya4-iac-azure repository to check if someone else encountered the same - found issue #231 - but was not helpful.
  2. Made sure that all resources are reachable (for instance netapp -> jump). To the best of my knowledge all resources are reachable.
  3. Tried to view the logs of the pods that fail, however I received this message -> <Error from server (BadRequest): container "sas-import-data-loader" in pod "sas-import-data-loader-27754440-h54kv" is waiting to start: CreateContainerConfigError>
  4. So essentially I cannot see the pod logs to understand why they fail
  5. Went through the documentation to check if I am missing some configuration step. Perhaps I am missing something here but I could not find any step I overlooked.
  6. I am provisioning the infrastructure and installation of baseline,viya through Docker, so there are no issues with versions of the dependencies

Attachments:

  1. Terraform.tfvars
  2. viya_pre_install_report_2022-10-11T11_49_13
  3. ansible-vars.yaml
  4. Screenshots of Pods
  5. Screenshot of - kubectl get logs {pod} -all containers

I am attaching the files as .txt as github does not allow to attach (.tfvars, .YAML, .HTML)

Please let me know if you need any further information to help me debug this issue, such as the terraform.tfstate files or any other logs. Thanks a lot for your help.

ansible-vars.txt terraform.txt viya_pre_install_report_2022-10-11T11_49_13.txt Screenshot 2022-10-11 at 11 38 17 Screenshot 2022-10-11 at 11 38 28 Screenshot 2022-10-11 at 11 38 39 Screenshot 2022-10-11 at 12 28 42

ThrashBaidon commented 1 year ago

Hi, It seems that I'm having a similar error, would you mind to share the steps you made for debug this issue?