The correctness of the Ansible playbook is not covered by the CI tests

[julienrf]

Our tests do not cover the Ansible playbook. The requirements to run the playbook are not clear, and any modification to the playbook may silently break it.

[julienrf]

For the record, here is how I locally tested the Ansilbe playbook. The information below could be used to implement an automated workflow.

1. Install Ansible on my machine

2. Create Docker containers: a Spark master node, a worker node, and DynamoDB, using the following docker-compose.yaml:

   services:
    master:
      build: dockerfiles/ansible
    worker:
      build: dockerfiles/ansible
   
    dynamodb:
      command: "-jar DynamoDBLocal.jar -sharedDb -inMemory"
      image: "amazon/dynamodb-local:latest"
      expose:
        - 8000
      ports:
        - "8000:8000"
      working_dir: /home/dynamodblocal

   It uses the following Dockerfile (at ./dockerfiles/ansible/Dockerfile), which sets up an SSH server in an Ubuntu OS:

   FROM ubuntu
   
   RUN apt-get update && apt-get install -y openssh-server sudo software-properties-common iproute2
   
   RUN mkdir /var/run/sshd
   
   RUN useradd -ms /bin/bash ubuntu \
      && echo 'ubuntu:aaaaaa' | chpasswd \
      && sudo adduser ubuntu sudo \
      && echo "ubuntu ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/ubuntu
   
   RUN echo "PasswordAuthentication yes" >> /etc/ssh/sshd_config \
      && echo "PermitRootLogin yes" >> /etc/ssh/sshd_config
   
   EXPOSE 22
   
   CMD ["/usr/sbin/sshd", "-D"]

3. Start the containers with docker compose up
4. Find the IP addresses of the master and worker services with a command like:

   docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' <container-name>

5. Edit ansible/inventory/hosts.ini and set the IP address of the master and worker hosts. Disable the spark_worker2 host.
6. Comment out private_key_file in ansible/ansible.cfg
7. Run ansible-playbook scylla-migrator.yml
8. When the process completes, open a shell in the master node and start Spark:

   docker compose exec master /bin/bash
   cd scylla-migrator
   ./start-spark.sh

9. Start the Spark worker on the worker node:

   docker compose exec worker /bin/bash
   ./start-slave.sh

10. From the master node, create a DynamoDB source table and put an item in it:

    aws configure set region us-west-1
    aws configure set aws_access_key_id dummy
    aws configure set aws_secret_access_key dummy
    aws \
     --endpoint-url http://dynamodb:8000 \
     dynamodb create-table \
       --table-name Source \
       --attribute-definitions AttributeName=id,AttributeType=S \
       --key-schema AttributeName=id,KeyType=HASH \
       --provisioned-throughput ReadCapacityUnits=100,WriteCapacityUnits=100
    aws \
     --endpoint-url http://dynamodb:8000 \
     dynamodb put-item \
     --table-name Source \
     --item '{ "id": { "S": "foo" } }'

11. From the master node, edit the file scylla-migrator/config.dynamodb.yml:

    source:
     type: dynamodb
     table: Source
     endpoint:
       host: http://dynamodb
       port: 8000
     credentials:
       accessKey: empty
       secretKey: empty
    target:
     type: dynamodb
     table: Target
     endpoint:
       host: http://dynamodb
       port: 8000
     credentials:
       accessKey: empty
       secretKey: empty
     streamChanges: false
    savepoints:
     path: /app/savepoints
     intervalSeconds: 300

12. Start the migration from the master node:

    ./submit-alternator-job.sh