K8s setup for reNgine

[0xtejas]

This pull request introduces several Kubernetes configurations for deploying various services, including Celery, Nginx, PostgreSQL, Redis, and a web application. The changes include deployment configurations, persistent volume claims, services, and secrets.

Deployment Configurations:

• k8s/celery-beat/deployment.yml: Added deployment configuration for celery-beat with environment variables and volume mounts.
• k8s/celery/deployment.yml: Added deployment configuration for celery with environment variables, resource requests, and volume mounts.
• k8s/nginx/deployment.yml: Added deployment configuration for nginx with volume mounts for configuration, certificates, and static files.
• k8s/ollama/deployment.yml: Added deployment configuration for ollama with volume mounts.
• k8s/web/deployment.yml: Added deployment configuration for web with environment variables, ports, and init containers for database migration and static file collection.

Persistent Volume Claims:

• k8s/celery-beat/pvc.yml: Added persistent volume claims for celery-beat including github-repos-pvc, wordlist-pvc, scan-results-pvc, gf-patterns-pvc, nuclei-templates-pvc, and tool-config-pvc.
• k8s/pvc.yml: Added persistent volume claims for shared storage and static files with nfs-rwx-storage.
• k8s/ollama/pvc.yml: Added persistent volume claim for ollama data storage.

Services:

• k8s/celery-beat/service.yml: Added service configuration for celery-beat with TCP port 5672.
• k8s/nginx/service.yml: Added service configuration for nginx with LoadBalancer type and ports for HTTP and HTTPS.
• k8s/ollama/service.yml: Added service configuration for ollama with ClusterIP type and port 11434.
• k8s/redis/service.yml: Added service configuration for redis with port 6379.
• k8s/postgres/service.yml: Added service configuration for postgres with port 5432.

Secrets:

• k8s/postgres/secret.yml: Added secret configuration for PostgreSQL credentials and domain name.

ConfigMaps:

• k8s/nginx/configmap.yml: Added ConfigMap for nginx configuration, including SSL settings and proxy settings.

StatefulSets:

• k8s/postgres/statefulset.yml: Added StatefulSet configuration for postgres with environment variables and volume mounts for data storage.

These changes collectively set up the necessary infrastructure for deploying and managing the services in a Kubernetes environment.

[github-actions[bot]]

Woohoo @0xtejas! 🎉 You've just dropped some hot new code! 🔥

Hang tight while we review this! You rock! 🤘

[0xtejas]

I need additional hands on this to improvise a few more things:

• [x] Add Ingress
• [x] Support SSL from LetsEncrypt or Cert Manager services provided by Cloud
• [ ] Current setup works for Digital Ocean. Native block storage provided by the PVCs doesn't support ReadWriteMany. Will have to test if the current configuration works on other cloud providers that use OpenEBS NFS Provisioner. I had to install OpenEBS NFS Provisioner on the Digital Ocean's k8s marketplace.
• [x] Fix the issue when routing with a domain name, I assume this will be resolved by solving the first two issues. Currently, I can view reNgine using https://IP, not the domain name.
• [ ] Additionally, make some changes such that the DB works in master-slave or another concept to make use of the replicas that can be created using stateful sets.
• [x] Support user account creation upon web pod deployment.

[0xtejas]

The boot order for the services are: postgres, redis, web, celery and celery-beat.

[0xtejas]

@yogeshojha you'll also have to release packages (docker images) for celery and celery-beat. In the current setup, it is using the ones that I built and pushed to GHCR.

[yogeshojha]

@0xtejas I am reviewing this, is this ready?

[0xtejas]

No @yogeshojha I need some more help on the other points that I have listed out above. I'll try accomplishing if possible and update the checkboxes.

[0xtejas]

There seems to be an issue with the current PR, the celery pod gets evicted after an hour or so. Also, the current resource requests and limits will get the pod killed with OOM. If we remove it, the issue should be resolved. However, the former issue is not yet addressed.

[github-actions[bot]]

Hey, thanks for your contribution! 🙏

We appreciate the time and effort you put into this PR. Sadly this is not the right fit for reNgine at the moment.

While we couldn't merge it this time, we value your interest in improving reNgine.

Feel free to reach out if you have any questions. Thanks again!

[0xtejas]

Sorry closed the PR by mistake. I have a few more changes left and an investigation to conduct into why the pod celery gets evicted once it reaches the end.

[0xtejas]

Almost, everything is done. I still cannot figure out why the celery pod gets OOMKilled. I'd appreciate it if you could check. Overall it works. I did not implement a replication method for DBs. This means if a user has more than 1 replica then they'll have to figure out how to setup data replication. We can revisit this advanced stuff in later issues/pr.

[0xtejas]

I've confirmed that the celery pod used to get killed cuz of less resource in the node. We will have to figure out a proper resource request and limitation for it to work in the 2 nodes - 4 GB / 2 vCPU environment. I confirmed that it is working without eviction/termination in memory-optimized nodes m-4vcpu-32gb x 2

[0xtejas]

@yogeshojha, can you please take a look when you get time? I've added all the manifest required. Some say the correct way to distribute K8s is using the HELM Chart, but I'm not very familiar with doing it.