Step 1: Setup basic AKS cluster

[ggorman]

• Assume 16 nodes and 32 shots.
• ssh access
• Devito install
• Dask configured; one worker/task per socket!
• GCC compiler.
• No MPI. No extras. Configure DEVITO_OPENMP. Configure pinning.

[navjotk]

Reproducing here all the steps required to do this:

1. Install az client, login.
2. Create resource group: az group create --name Nav_DAKS --location uksouth
3. Create container registry: az acr create --resource-group Nav_DAKS --name devitoaks --sku Basic
4. Login to container registry: az acr login --name devitoaks
5. Get container registry login server: az acr list --resource-group Nav_DAKS --query "[].{acrLoginServer:loginServer}" --output table
6. Locally, create the docker image for Devito using: docker build -t devito_base . -f docker/Dockerfile in the Devito home directory.
7. Locally, tag the Devito docker image: docker tag devito-base devitoaks.azurecr.io/devito-base:v1
8. Upload the Devito docker image: docker push devitoaks.azurecr.io/devito-base
9. Confirm the upload succeeded: az acr repository list --name devitoaks --output table
10. Create kubernetes cluster:

    az aks create \                                                               
     --resource-group Nav_DAKS \
     --name devitocluster1 \
     --node-count 2 \
     --generate-ssh-keys \
     --attach-acr devitoaks

11. Install kubernetes CLI: az aks install-cli
12. Setup kubernetes CLI authentication: az aks get-credentials --resource-group Nav_DAKS --name devitocluster1
13. Setup Dask-cluster.yaml as follows:

    apiVersion: apps/v1beta1
    kind: Deployment
    metadata:
    name: devito-server
    spec:
    replicas: 1
    template:
    metadata:
      labels:
        app: devito-server
    spec:
      nodeSelector:
        "beta.kubernetes.io/os": linux
      containers:
      - name: devito-server
        image: devitoaks.azurecr.io/devito-base:v2
        command: ['/venv/bin/dask-scheduler']
        ports:
        - containerPort: 8786
          name: devito-server
    ---
    apiVersion: v1
    kind: Service
    metadata:
    name: devito-server
    spec:
    type: LoadBalancer
    ports:
    - port: 8786
    selector:
    app: devito-server
    ---
    apiVersion: apps/v1beta1
    kind: Deployment
    metadata:
    name: devito-worker
    spec:
    replicas: 10
    strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    minReadySeconds: 5 
    template:
    metadata:
      labels:
        app: devito-worker
    spec:
      nodeSelector:
        "beta.kubernetes.io/os": linux
      containers:
      - name: devito-worker
        env:
        - name: PYTHONPATH
          value: /app
        - name: DEVITO_OPENMP
          value: "1"
        - name: OMP_PROC_BIND
          value: "TRUE"
        image: devitoaks.azurecr.io/devito-base:v2
        command: ['/venv/bin/dask-worker', 'tcp://devito-server:8786']
        ports:
        - containerPort: 80

    This will setup a kubernetes cluster with one Dask scheduler node with an open port 8786 to the world, and 10 workers that will connect to this scheduler automatically.

14. Apply the kubernetes configuration: kubectl apply -f dask-cluster.yaml
15. Find the IP address of the scheduler service: kubectl get services (the one we are interested in is the public IP of the LoadBalancer)
16. Fire up another container from the Devito-base image we created earlier (I use docker-compose up devito but this container could be fired in Azure or anywhere).
17. Open up the Devito Dask tutorial in a Jupyter notebook inside this container (the docker-compose setup in Devito makes this easy).
18. In cell 5 of the notebook, replace the line cluster = LocalCluster(n_workers=nsources, death_timeout=600) with client = Client('51.11.43.137:8786'), where 51.11.43.137 is the IP of the scheduler we found in step 8.
19. Run all cells in the notebook. This notebook is now running its heavy functions on the kubernetes cluster we created.

[navjotk]

Closing as this is now long complete. The instructions have been moved to the wiki.