Self hosted deployment

[ThorodanBrom]

UGI self hosting of the data plane

• OGC server
• Postgresql with PostGIS
• MinIO

Ideal outcome: Simplified containerized deployment with minimal dependencies on weird container tech.

[ThorodanBrom]

• Since we only plan to have containerized deployment, virtual hosting access for S3 is OK and no need to support path based access
• @karun-singh suggested to try passing S3 bucket and S3 endpoint in the config instead of S3 bucket and S3 region - the S3 URL can still be formed by doing S3 bucket + S3 endpoint. If this is possible and minio virtual hosting access works with this, then it will be able to use the service name (in Docker Compose and potentially K8s as well) to refer to the minio container in the internal network
  • This will avoid using the network alias for the container
  • this is something that I'll have to test with minio
• HTTPS access
  • Ways to get around this:
    • Make self-signed certificates and inject into OGC container and Minio - difficult to manage
    • Use real certificate - too much work to add the certificate to the docker container. If users want external HTTPS access to Minio, better to use NGINX as a reverse proxy.
  • May be simpler to support HTTP - have config value

[ThorodanBrom]

One major restriction that virtual hosting based access introduces is that network aliases need to be added per bucket. The Docker network does not do subdomain resolution automatically. https://github.com/datakaveri/ogc-resource-server/blob/0efa06c0e5fb30f6c0bc5fb6a4c21898337bfe14/docker-compose.test.yml#L92-L93 Here, we added an alias for the bucket bucket1.

So if the server can support access to more than 1 bucket, the compose file or deployment files may need to be updated for every new bucket.

We can sort of do this for an existing deployment, because the server is designed to work with only 1 bucket. But tomorrow if the server supports multiple buckets, we have to update the aliases with every new bucket that is used, which is not feasible.

Some options around this:

• Use something like this - https://github.com/lucaslorentz/caddy-docker-proxy - as a reverse proxy in docker-compose. which may allow redirecting all subdomains (like a wildcard) of a URL to the correct container (this is a docker solution, cannot be extended to k8s)
• Modify code to support path-based access

[mahimatics]

Heterogeneous sources for buckets would be a good feature to support. However, that complicates the deployment configuration.

Also the codebase needs to support multiple buckets during runtime but the implementation of this is a big open question.

[mahimatics]

Open question: What happens to the auditing and metering flows?

[ThorodanBrom]

• @karun-singh said Going ahead w/ docker swarm
• For self-hosted, may be best option to have path based access
  • Difficult to keep adding subdomains for every new bucket if we do not control DNS
  • When someone hosts MinIO, will they ever want multiple buckets? Should not one bucket be enough?

[mahimatics]

MinIO integration is done. Configurations have to be updated and then the PR can be merged.

Bare metal deployment requires inputs from Karun and Swarup.

[ThorodanBrom]

• Need to come up with min compute and storage requirements for any on-prem deployment to have high availability on server and also avoiding data loss for MinIO
  • microk8s can work on a single machine, but there's no point because if that machine goes down, everything goes down
  • MinIO recommends MNMD for production workloads. SNMD may look OK since it has some failover/redundancy. But main thing to note is it's multi-disk and not just one disk for both of these. SNSD is not advised at all.
• Office cluster using microk8s
• Also using Longhorn for distributed storage management
• A fresh on-prem deployment will always start with setting up k8s/microk8s on a set of machines and then Longhorn also (if it makes managing storage easier than doing it manually)
• Once these are installed, then rest of components - server, postgres, minio, monitoring stack (loki, prometheus, grafana) can be installed

[ThorodanBrom]

MinIO Recommended requirements for MNMD - here

Notable points

• Use local storage (NFS slow)
• Use XFS formatted drives (other filesystems are not recommended or tested)
• All same kind of drives - all NVMe or all SSD or all HDD . They don't recommend HDD for performance reasons
• All consistent size
  • Lower limit of disk size is considered when different disks of different sizes are added
• Exclusive access to drives - nothing else should be on the drives or using the drives
• They recommend 32 GB of RAM on every node - but the hardware checklist has this table which I think implies you can get away w/ less

Other things

• When doing actual deployment, it will be good to follow the Hardware Checklist for K8S
• The disks need to be empty with no RAID or any configuration apart from formatted as XFS, so we do not need to do anything w.r.t. redundancy. MinIO in MNMD mode will take care of it.
• MNMD mode supports Erasure Codes by default.
• Adding new disks as is does not seem to be supported. You have to add a new MinIO node w/ disks to the pool of in MNMD or a new MinIO pod with new disks attached to the pod in k8s

Also

• Office cluster is using a NAS

[ThorodanBrom]

• Consider it as a prod thing
• Talk to Sunil to see if UI can handle multiple RSs