atsign-foundation / at_server

The software implementation of Atsign's core technology
https://docs.atsign.com
BSD 3-Clause "New" or "Revised" License
40 stars 12 forks source link

Define correct volume mounts for a secondary #296

Closed cpswan closed 2 years ago

cpswan commented 3 years ago

Is your feature request related to a problem? Please describe. In trying to understand why the issues seen in #290 were passing end to end tests I looked for differences between the environments.

Here's what we do in dess (used for cicd1 & cicd2 instances in end to end tests):

  volumes:
    - /home/atsign/atsign/etc/live/cicd1.atsign.wtf:/atsign/certs:rw
    - /home/atsign/atsign/etc/archive/cicd1.atsign.wtf:/archive/cicd1.atsign.wtf:rw
    - /home/atsign/atsign/@cicd1:/atsign/storage:rw

If I break it down line by line:

  1. /atsign/certs is mapped to where the certs are held on the host. FWIW I don't see any good reason for this being writable from the container, so it should probably now be :rw
  2. /archive/$FQDN is mapped to the cert archive. WHY?
  3. /atsign/storage is mapped to a place on the host to keep the state of the secondary.

Turning to the staging environment (or production) we have something like:

    volumes:
      - "/gluster/@/secondaries/ea/b6/59/c3/eab659c3530555fbd08b9da7d17991a1/f49ebec6-3ed8-59a5-96ba-6cfe743e762e:/atsign"
      - "/etc/letsencrypt/live/f49ebec6-3ed8-59a5-96ba-6cfe743e762e.buzz.do-sf2.atsign.zone:/atsign/certs"
      - "/atsign/config"

Again breaking it down line by line:

  1. /atsign is mounted to the place on the host to keep the state of the secondary. Why not /atsign/storage?
  2. /atsign/certs is mapped to where the certs are held on the host.
  3. /atsign/config unknown why this is here?

NB this time there's no attempt to map archived certs.

Describe the solution you'd like

I think we can just mount:

This should be uniform across dess, staging and production.

Describe alternatives you've considered We could put the certs into the state area, and reduce down to one mount point. But that might create extra complexity for cert management.

cconstab commented 3 years ago

/archive/$FQDN is mapped to the cert archive. WHY?

Because of the joy of certbot... Certbot uses symbolic links in the live/ directory to the ../../archive directory so it also has to be mounted as well. Much simplier to do this with two point mounts than play with certbot

cpswan commented 3 years ago

Thanks @cconstab. So we need to keep /archive/$FQDN for dess, but it's not needed for staging/production.

Speaking to @kumarnarendra701 earlier he said that the /atsign/config entry is intended to allow config within the container to be reachable, but I'm not sure that's working.

cpswan commented 3 years ago

@kumarnarendra701 (CC @athandle) - this is high priority for PR25, but should be straightforward.

First we need to modify the script that creates the docker-compose.yml files for each secondary so that they're mounting:

    volumes:
      - "/gluster/@/secondaries/$MOUNTSTUB/$UUID/storage:/atsign/storage"
      - "/etc/letsencrypt/live/$UUID.hornet.atsign.zone:/atsign/certs"

NB there should no longer be a need for a volume mount - "/atsign/config" as the config dir inside the container is now safely outside the other mount points.

Then I can take care of updating the running secondaries.

I'll do something like this:

#!/bin/bash
while IFS=, read -r SECONDARY ATSIGN PORT MOUNT
do
        sudo docker service scale "$SECONDARY"=0
        sudo docker service update --mount-rm /atsign "$SECONDARY"
        sudo docker service update --mount-rm /atsign/config "$SECONDARY"
        sudo docker service update --mount-add type=bind,source="$MOUNT"/storage,target=/atsign/storage "$SECONDARY"
        sudo docker service scale "$SECONDARY"=1
done < secondaries.csv

Here's how I generate secondaries.csv:

#!/bin/bash
for SECONDARY in $(sudo docker service ls |grep secondary |awk '{print $2}')
  do
    INSPECT=$(sudo docker inspect "$SECONDARY")
    ATSIGN=$(echo "$INSPECT" | jq -r ' .[0].Spec.TaskTemplate.ContainerSpec.Args[1]')
    PORT=$(echo "$INSPECT" | jq -r ' .[0].Spec.TaskTemplate.ContainerSpec.Args[3]')
    MOUNT=$(echo "$INSPECT" | jq -r ' .[0].Spec.TaskTemplate.ContainerSpec.Mounts[]
          | select(.Target | IN ("/atsign","/atsign/storage")) | .Source')
    echo "$SECONDARY,$ATSIGN,$PORT,$MOUNT"
  done
kumarnarendra701 commented 3 years ago

@cpswan - I have created the script that will look for secondary "docker-compose.yml" and check for "/atsign/config" if exist delete it and update the stack.

Script -

#!/bin/bash
#for UUID in $(docker stack ls | grep -Ev "portainer|traefik|NAME" | awk '{print $1}')
for UUID in 419ebfe0-86df-55b9-b733-8156b1f3defa
do 
    log='secoundaries-logs/temp/'$UUID'-create_log.txt'
    echo "=============================" >> $log 2>&1
    echo "Secondarie migration started" >> $log 2>&1
    secoundary_full_path=$(find /gluster/@/secondaries/ -name "$UUID" -print)
    echo $secoundary_full_path  >> $log 2>&1
    grep -R '/atsign/config' $secoundary_full_path/docker-compose.yml
    if [ $? -eq 0 ]; then
        echo "Mount point(/atsign/config) exist"
        sed -i '/\/atsign\/config/d' $secoundary_full_path/docker-compose.yml
        docker stack deploy  --compose-file $secoundary_full_path/docker-compose.yml $UUID  --with-registry-auth
        echo "Secondarie migration completed" >> $log 2>&1
        echo "===============================" >> $log 2>&1
    else
            echo "Secoundary Not exist"
    fi
done

After upgrade -

version: "3.8"

services:
  secondary: 
    image: reg.buzz.do-sf2.atsign.zone/atsigncompany/secondary:dev_env
    command: "-a @parliamentary1taekwondo -p 2351 -s 854812a8948bc4c94149dfa296d46d86249a634df719d46fd78609f6cee09f15862d8ccee56f981a6f975427786e281855f51f750aa0ab51f2a6da292f1f168a"
    deploy:
      placement:
        constraints: [node.role == worker]
      resources:
        limits:
          cpus: '0.25'
          memory: '50M'
        reservations:
                #cpus: '0.01'
          memory: '6M'
    ports:
      - target: 2351
        published: 2351
    volumes:
      - "/gluster/@/secondaries/14/8d/5b/57/148d5b573c88ed387208ce420c6706bb/8d7de2eb-2aa3-54f9-bb39-453745ebb9a4:/atsign"
      - "/etc/letsencrypt/live/8d7de2eb-2aa3-54f9-bb39-453745ebb9a4.buzz.do-sf2.atsign.zone:/atsign/certs"
    networks:
      second: {}

networks:
  second:
    external: true
    name: secondaries

Please review and share your feedback.

kumarnarendra701 commented 3 years ago

Performed testing on below secondaries. 583wrong parliamentary1taekwondo correspondingshaokao5

cpswan commented 3 years ago

@kumarnarendra701 the script is missing the most important piece, which is to mount /atsign/storage rather than /atsign

kumarnarendra701 commented 3 years ago

@cpswan - Please find updated script.

#!/bin/bash
#for UUID in $(docker stack ls | grep -Ev "portainer|traefik|NAME" | awk '{print $1}')
for UUID in 9eda0f61-afe5-593f-bcf7-c884a7d9709b
do 
    log='secoundaries-logs/temp/'$UUID'-create_log.txt'
    echo "=============================" >> $log 2>&1
    echo "Secondarie migration started" >> $log 2>&1
    secoundary_full_path=$(find /gluster/@/secondaries/ -name "$UUID" -print)
    echo $secoundary_full_path  >> $log 2>&1
    grep -R '/atsign/config' $secoundary_full_path/docker-compose.yml
    if [ $? -eq 0 ]; then
        echo "Mount point(/atsign/config) exist"
        cp -rpf $secoundary_full_path/docker-compose.yml $secoundary_full_path/docker-compose.yml-$(date '+%Y-%m-%d')
        sed -i '/\/atsign\/config/d' $secoundary_full_path/docker-compose.yml
        sed -i "s/$UUID\:\/atsign/$UUID\/storage\:\/atsign\/storage/g" $secoundary_full_path/docker-compose.yml
        docker stack deploy  --compose-file $secoundary_full_path/docker-compose.yml $UUID  --with-registry-auth
        echo "Secondarie migration completed" >> $log 2>&1
        echo "===============================" >> $log 2>&1
    else
            echo "Secoundary Not exist"
    fi
done

docker-compose.yml

version: "3.8"

services:
  secondary: 
    image: reg.buzz.do-sf2.atsign.zone/atsigncompany/secondary:dev_env
    command: "-a @acutesaudathedeal3 -p 2459 -s c4857ae1ea8eac9e40a08e8857d8389564167745020fd64333ce2b9ace9a37416a4aa0193d55b075af651f936b6da82000435770c355e65aef23881408cd3547"
    deploy:
      placement:
        constraints: [node.role == worker]
      resources:
        limits:
          cpus: '0.25'
          memory: '50M'
        reservations:
                #cpus: '0.01'
          memory: '6M'
    ports:
      - target: 2459
        published: 2459
    volumes:
      - "/gluster/@/secondaries/80/dd/08/96/80dd089641a34c95fdf950e9c66c60fd/a6582237-a0d2-5987-89f3-290f119d7e7f/storage:/atsign/storage"
      - "/etc/letsencrypt/live/a6582237-a0d2-5987-89f3-290f119d7e7f.buzz.do-sf2.atsign.zone:/atsign/certs"
    networks:
      second: {}

networks:
  second:
    external: true
    name: secondaries
cpswan commented 3 years ago

@kumarnarendra701 LGTM, though you might want to s/ecound/econd/ (which is right in all the places it needs to be, but wrong everywhere else).

kumarnarendra701 commented 3 years ago

@cpswan - Updated script

#!/bin/bash
#for UUID in $(docker stack ls | grep -Ev "portainer|traefik|NAME" | awk '{print $1}')
for UUID in 9eda0f61-afe5-593f-bcf7-c884a7d9709b
do 
    log='secondaries-logs/temp/'$UUID'-create_log.txt'
    echo "=============================" >> $log 2>&1
    echo "Secondary migration started" >> $log 2>&1
    secondary_full_path=$(find /gluster/@/secondaries/ -name "$UUID" -print)
    echo $secondary_full_path  >> $log 2>&1
    grep -R '/atsign/config' $secondary_full_path/docker-compose.yml
    if [ $? -eq 0 ]; then
        echo "Mount point(/atsign/config) exist"
        cp -rpf $secondary_full_path/docker-compose.yml $secondary_full_path/docker-compose.yml-$(date '+%Y-%m-%d')
        sed -i '/\/atsign\/config/d' $secondary_full_path/docker-compose.yml
        sed -i "s/$UUID\:\/atsign/$UUID\/storage\:\/atsign\/storage/g" $secondary_full_path/docker-compose.yml
        docker stack deploy  --compose-file $secondary_full_path/docker-compose.yml $UUID  --with-registry-auth
        echo "Secondary migration completed" >> $log 2>&1
        echo "===============================" >> $log 2>&1
    else
            echo "Secondary Not exist"
    fi
done
cpswan commented 3 years ago

Thanks @kumarnarendra701 LGTM

Have you also updated the secondary creation script?

kumarnarendra701 commented 3 years ago

@cpswan - I was waiting for the final review, now I'll make changes to the creation script, and also let me should I run the script for stacks migration.

cpswan commented 3 years ago

The first priority is to get the creation script changed. So that we're not creating new secondaries with mounts to /atsign

Once that's done we can remount the existing atsigns, and change their docker-compose files so that they won't get reverted.

As usual, let's do staging first, and once that's done move on to production.

NB there are some secondaries that have already been moved by hand (but only their live config, not their compose files).

kumarnarendra701 commented 3 years ago

@cpswan - I have updated the base docker-compose file and create bash script.

version: "3.8"

services:
  secondary: 
    image: reg.buzz.do-sf2.atsign.zone/atsigncompany/secondary:dev_env
    command: "-a @ATSIGN -p PORT -s ATSECRET"
    environment:
      - rootServerUrl=${root_server_url}
    deploy:
      placement:
        constraints: [node.role == worker]
      resources:
        limits:
          cpus: '0.25'
          memory: '50M'
        reservations:
                #cpus: '0.01'
          memory: '6M'
    ports:
      - target: PORT
        published: PORT
    volumes:
      - "/gluster/@/secondaries/DIR/NAME/storage:/atsign/storage"
      - "/etc/letsencrypt/live/UUID.buzz.do-sf2.atsign.zone:/atsign/certs"
    networks:
      second: {}

networks:
  second:
    external: true
    name: secondaries

Also, added the below line in the creation script, because we need a storage directory for the mount. mkdir -p ../secondaries/$DIR/$UUID/storage

cpswan commented 3 years ago

Thanks @kumarnarendra701

Can you now please update all the existing secondaries in staging

cpswan commented 2 years ago

@kumarnarendra701 reports that this has now been done for staging, and I've confirmed with some spot checks.

Changes haven't been applied to prod yet, so needs to be carried over to PR26

kumarnarendra701 commented 2 years ago

@cpswan - Updated creation script on prod and migrated 5 secondaries for monitoring if all good for some time then I'll trigger for all stacks.

cpswan commented 2 years ago

@kumarnarendra701 please go ahead with the rest of the secondaries in prod. We know that this change works, and so if we hit issues it will be because of some peculiarity in a given secondary rather than because the overall approach is wrong.

kumarnarendra701 commented 2 years ago

@cpswan - Migration script started on prod Tmux Session - "tmux attach -t 5".

cpswan commented 2 years ago

@kumarnarendra701 has confirmed that the prod remount script has now completed