MINIO RELEASE

[Jean-Baptiste-Lasselle]

• This is a planned task
• consist in making a new release adding definition of an AWS AMI image that gives more thant 100 Gb on /var, CentsOS based, with LVM setup, two physical disk devices
• with ansible playbook that installs git docker , docker-compose, and executes the headless minikube provisioning recipe in release 0.0.3
• then deploy minio to the headless minikube (tested, ends up after a minute or two, to a fully operationnal all green lights minio service) :

Standalone deployment

# latest minor bug fix
export MINIO_VERSION=RELEASE.2020-03-25T07-03-04Z
# --- #
# execute this where you havev installed and configured your kubectl
git clone https://github.com/minio/minio
cd minio

git checkout ${MINIO_VERSION}

kubectl create -f docs/orchestration/kubernetes/minio-standalone-pvc.yaml
kubectl create -f docs/orchestration/kubernetes/minio-standalone-deployment.yaml
kubectl create -f docs/orchestration/kubernetes/minio-standalone-service.yaml

# very simple deployment : 
# - first, we claim volume for storage, to k8S cluster
# - then we proceed with minio deployment
# - and we finally create the service, that exposes minio to other services, or the outside world

• delete everything :

kubectl delete --all all
kubectl delete --all pvc

And here are the Persistent Volume Claims that you get in the distributed case (one less PVC) :

distributed deployment

# latest minor bug fix
export MINIO_VERSION=RELEASE.2020-03-25T07-03-04Z
# --- #
# execute this where you havev installed and configured your kubectl
git clone https://github.com/minio/minio
cd minio

git checkout ${MINIO_VERSION}

kubectl create -f docs/orchestration/kubernetes/minio-distributed-headless-service.yaml
kubectl create -f docs/orchestration/kubernetes/minio-distributed-statefulset.yaml
kubectl create -f docs/orchestration/kubernetes/minio-distributed-service.yaml

# very simple deployment : 
# - first, we claim volume for storage, to k8S cluster
# - then we proceed with minio deployment
# - and we finally create the service, that exposes minio to other services, or the outside world

I personnally tested it, the throughput is incredible, much faster than with the docker-compose starter deployment .

And here are the Persistent Volume Claims that you get in the distributed case (one less PVC) :

• The service exposes a port, to load balance the minio pods :

jbl@poste-devops-jbl-16gbram:~/minio$ kubectl get pods,svc
NAME          READY   STATUS    RESTARTS   AGE
pod/minio-0   1/1     Running   0          3m15s
pod/minio-1   1/1     Running   0          3m15s
pod/minio-2   1/1     Running   0          3m15s
pod/minio-3   1/1     Running   0          3m15s

NAME                    TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
service/kubernetes      ClusterIP      10.96.0.1        <none>        443/TCP          4m21s
service/minio           ClusterIP      None             <none>        9000/TCP         3m15s
service/minio-service   LoadBalancer   10.102.197.151   <pending>     9000:30138/TCP   3m12s
jbl@poste-devops-jbl-16gbram:~/minio$ 

Scale out minio

kubectl scale statefulsets minio --replicas=10

• below screenshots shows you that immeditely after scaling out from 4 to 10, persistent volume claims are automatically created :

• and after a few minutes , all 10 pods are up n running :

Minio Web UI Login

Using minio username, and password minio123 (see minio-distributed-deployment.yaml )

[Jean-Baptiste-Lasselle]

Integration Minio HAshiCorp Vault

reprendre https://docs.min.io/docs/minio-kms-quickstart-guide.html

[Jean-Baptiste-Lasselle]

Minio strorage performances

What I loved about Minio, is that I stored 54 Gb of video files in there, and I get only 14 Gb actually in use, on the filesystem on the single Cluster Node :

jibl@poste-devops-jbl-16gbram:~/minio$ kubectl get pods,svc
NAME          READY   STATUS    RESTARTS   AGE
pod/minio-0   1/1     Running   0          4h35m
pod/minio-1   1/1     Running   0          4h35m
pod/minio-2   1/1     Running   0          4h35m
pod/minio-3   1/1     Running   0          4h35m
pod/minio-4   1/1     Running   0          65m
pod/minio-5   1/1     Running   0          65m
pod/minio-6   1/1     Running   0          65m
pod/minio-7   1/1     Running   0          65m
pod/minio-8   1/1     Running   0          65m
pod/minio-9   1/1     Running   0          65m

NAME                    TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
service/kubernetes      ClusterIP      10.96.0.1        <none>        443/TCP          4h36m
service/minio           ClusterIP      None             <none>        9000/TCP         4h35m
service/minio-service   LoadBalancer   10.102.197.151   <pending>     9000:30138/TCP   4h35m
jibl@poste-devops-jbl-16gbram:~/minio$ 

jbl@pegasusio:~/minikube$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                  STORAGECLASS   REASON   AGE
pvc-133c4bc0-2afc-4427-8700-683088b00fe5   10Gi       RWO            Delete           Bound    default/data-minio-1   standard                4h18m
pvc-4aaef52a-fe97-48fd-81d5-ad00f6dcae70   10Gi       RWO            Delete           Bound    default/data-minio-8   standard                48m
pvc-508a4983-8120-4b9e-a9ee-ab5ab24cb800   10Gi       RWO            Delete           Bound    default/data-minio-3   standard                4h18m
pvc-67072be7-82ed-4484-8685-dce680566e73   10Gi       RWO            Delete           Bound    default/data-minio-9   standard                48m
pvc-80445ec1-2f58-4bf2-9feb-834abb423ac6   10Gi       RWO            Delete           Bound    default/data-minio-5   standard                48m
pvc-8e74b771-234d-42a6-8116-500cc6e2cf91   10Gi       RWO            Delete           Bound    default/data-minio-4   standard                48m
pvc-aa1a1e62-ae07-491c-a220-4e67fc4bc15e   10Gi       RWO            Delete           Bound    default/data-minio-6   standard                48m
pvc-cafbd409-f915-43aa-a7b8-393b139cb395   10Gi       RWO            Delete           Bound    default/data-minio-2   standard                4h18m
pvc-de2e7b5b-54f1-4a44-8591-c7f2d80ec958   10Gi       RWO            Delete           Bound    default/data-minio-0   standard                4h18m
pvc-ebd7f575-f46c-4486-bae3-6fc211a80e2a   10Gi       RWO            Delete           Bound    default/data-minio-7   standard                48m
jbl@pegasusio:~/minikube$ kubectl describe pv pvc-133c4bc0-2afc-4427-8700-683088b00fe5
Name:            pvc-133c4bc0-2afc-4427-8700-683088b00fe5
Labels:          <none>
Annotations:     hostPathProvisionerIdentity: a87fdd8d-7503-11ea-a376-080027d1dd23
                 pv.kubernetes.io/provisioned-by: k8s.io/minikube-hostpath
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    standard
Status:          Bound
Claim:           default/data-minio-1
Reclaim Policy:  Delete
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        10Gi
Node Affinity:   <none>
Message:         
Source:
    Type:          HostPath (bare host directory volume)
    Path:          /tmp/hostpath-provisioner/pvc-133c4bc0-2afc-4427-8700-683088b00fe5
    HostPathType:  
Events:            <none>
jbl@pegasusio:~/minikube$ sudo df -Th /tmp
Sys. de fichiers               Type Taille Utilisé Dispo Uti% Monté sur
/dev/mapper/pegasusio--vg-root ext4   110G     14G   91G  14% /
jbl@pegasusio:~/minikube$ sudo df -Th /tmp
Sys. de fichiers               Type Taille Utilisé Dispo Uti% Monté sur
/dev/mapper/pegasusio--vg-root ext4   110G     14G   91G  14% /
jbl@pegasusio:~/minikube$ ls -allh /tmp/hostpath-provisioner/
total 48K
drwxr-xr-x 12 root root 4,0K avril  2 23:13 .
drwxrwxrwt 15 root root 4,0K avril  3 00:07 ..
drwxrwxrwx  4 root root 4,0K avril  3 00:05 pvc-133c4bc0-2afc-4427-8700-683088b00fe5
drwxrwxrwx  2 root root 4,0K avril  2 23:13 pvc-4aaef52a-fe97-48fd-81d5-ad00f6dcae70
drwxrwxrwx  4 root root 4,0K avril  3 00:05 pvc-508a4983-8120-4b9e-a9ee-ab5ab24cb800
drwxrwxrwx  2 root root 4,0K avril  2 23:13 pvc-67072be7-82ed-4484-8685-dce680566e73
drwxrwxrwx  2 root root 4,0K avril  2 23:13 pvc-80445ec1-2f58-4bf2-9feb-834abb423ac6
drwxrwxrwx  2 root root 4,0K avril  2 23:13 pvc-8e74b771-234d-42a6-8116-500cc6e2cf91
drwxrwxrwx  2 root root 4,0K avril  2 23:13 pvc-aa1a1e62-ae07-491c-a220-4e67fc4bc15e
drwxrwxrwx  4 root root 4,0K avril  3 00:05 pvc-cafbd409-f915-43aa-a7b8-393b139cb395
drwxrwxrwx  4 root root 4,0K avril  3 00:05 pvc-de2e7b5b-54f1-4a44-8591-c7f2d80ec958
drwxrwxrwx  2 root root 4,0K avril  2 23:13 pvc-ebd7f575-f46c-4486-bae3-6fc211a80e2a
jbl@pegasusio:~/minikube$ sudo df -Th /tmp/hostpath-provisioner/
Sys. de fichiers               Type Taille Utilisé Dispo Uti% Monté sur
/dev/mapper/pegasusio--vg-root

[Jean-Baptiste-Lasselle]

External Name instead of nodeport

https://kubernetes.io/docs/concepts/services-networking/service/#externalname