high availability & ci/cd for biovalidator

[amnonkhen]

• [x] ci/cd for biovalidator - using k8s
  • [x] docker file
  • [x] gitlab-ci file
  • [x] all k8s files: deployment, service, image pull secret etc.
  • [x] push to branch triggers a build and deployment to k8s
  • [x] setup external access to service on k8s. See catalogue ticket: ebi-ait/data-catalogue-frontend#11
• [ ] clean up duplicate CI/CD from gitlab(biosamples group has a biovalidator being mirrored)
• [ ] If all the deployment looks good , remove VM based deployments, change production context path to match with existing deployment(i.e., with ait path) - see #63
• [x] update rules in wp-traffic-manager to match correct context path
• [x] use gitlab environments
• [ ] diagram showing the architecture
• [ ] merge to main branch

[snathanvj]

What I need to do:

Task-3(high availability):

• understand existing infrastructure availability(check with Amnon, refer one of the existing project, probably data-catalogue-frontend)
• how to deploy to k8s(self learning or knwolege from Amnon)
• meantime learn k8s basics

Task 1 & 2:

• gitlab CI/CD - I know it already

Task-1: Biovalidator:

• already running as a service in biosample vm
• not running as docker , just the project cloned and started using npm
• not clear about how is is being deployed currently(means any script or ci/cd or directly logging to server and starting it)

TODO:

• get the VM/k8s cluster details to deploy it
• configure deployment through gitlab (CI/CD)

Task-2 :CI/CD for schema store(checklist editor and schema ):

• schema store : biosample has schema-store for storing biosample checklist schemas in json format in mongodb. It can be accessed using java/spring based REST API (https://www.ebi.ac.uk/biosamples/schema/store/api/v2/schemas/search?text=&page=0&size=5).

Question: Is our checklist editor is going to use this schema-store service to store sample checklists and fields.

if the answer to above question is NO, then the second task CI/CD for schema-store is about CI/CD for checklist-editor(we are going to develop), is this correct?

[Wkt8]

Deploy it on k8s cluster as Docker can't run on VMs for some reason

[amnonkhen]

@snathanvj it appears this needs to be split to 3 tickets

[amnonkhen]

Question: Is our checklist editor is going to use this schema-store service to store sample checklists and fields.

if the answer to above question is NO, then the second task CI/CD for schema-store is about CI/CD for checklist-editor(we are going to develop), is this correct?

For the first stage we will use the schema store.

[snathanvj]

Manually created namespace: bio-val-dev created deployment and service and accessed using Node host name and port.

used existing docker image for this TODO:

• [ ] create and publish(to docker registry) new docker image using gitlab job. Pull it manually and verify it runs properly
• [ ] add k8s deployment scripts to gitlab.yml , configure deployment trigger as manual

[snathanvj]

@theisuru @amnonkhen I could not create a branch and push to biovalidator repo

16:14:10.552: [biovalidator] git -c credential.helper= -c core.quotepath=false -c log.showSignature=false push --progress --porcelain origin refs/heads/ck-49-k8s-deployment:refs/heads/ck-49-k8s-deployment --set-upstream remote: Permission to elixir-europe/biovalidator.git denied to snathanvj.

Could someone give me access please

[snathanvj]

Current changes are here https://github.com/elixir-europe/biovalidator/tree/ck-49-k8s-deployment

[amnonkhen]

@snathanvj thanks for saving your code to a branch. Can you create a draft pull request?

[snathanvj]

External URL: https://www.ebi.ac.uk/biovalidator

@amnonkhen I did not use the ait/biovalidator context path as it is already in use by another biovalidator hosted in vm. If we are happy with our new k8s deployment , we could remove the exiting vm deployment and use ait/biovalidator context path.

[snathanvj]

@theisuru Dev pod: we can use this for stress testing http://hx-rke-wp-webadmin-17-worker-3.caas.ebi.ac.uk:31990/

https://wwwdev.ebi.ac.uk/ait/biovalidator this is not yet working, will check it.

UPDATE: @theisuru now you can use the dev service at this endpoint https://wwwdev.ebi.ac.uk/biovalidator

[Wkt8]

Senthil to update the diagram and remove the previous CI/CD as it is now being replaced by the new k8s deployment

[snathanvj]

https://wwwdev.ebi.ac.uk/ait/biovalidator This works now.