Closed cpswan closed 2 years ago
/archive/$FQDN is mapped to the cert archive. WHY?
Because of the joy of certbot... Certbot uses symbolic links in the live/ directory to the ../../archive directory so it also has to be mounted as well. Much simplier to do this with two point mounts than play with certbot
Thanks @cconstab. So we need to keep /archive/$FQDN
for dess, but it's not needed for staging/production.
Speaking to @kumarnarendra701 earlier he said that the /atsign/config
entry is intended to allow config within the container to be reachable, but I'm not sure that's working.
@kumarnarendra701 (CC @athandle) - this is high priority for PR25, but should be straightforward.
First we need to modify the script that creates the docker-compose.yml files for each secondary so that they're mounting:
volumes:
- "/gluster/@/secondaries/$MOUNTSTUB/$UUID/storage:/atsign/storage"
- "/etc/letsencrypt/live/$UUID.hornet.atsign.zone:/atsign/certs"
NB there should no longer be a need for a volume mount - "/atsign/config"
as the config dir inside the container is now safely outside the other mount points.
Then I can take care of updating the running secondaries.
I'll do something like this:
#!/bin/bash
while IFS=, read -r SECONDARY ATSIGN PORT MOUNT
do
sudo docker service scale "$SECONDARY"=0
sudo docker service update --mount-rm /atsign "$SECONDARY"
sudo docker service update --mount-rm /atsign/config "$SECONDARY"
sudo docker service update --mount-add type=bind,source="$MOUNT"/storage,target=/atsign/storage "$SECONDARY"
sudo docker service scale "$SECONDARY"=1
done < secondaries.csv
Here's how I generate secondaries.csv:
#!/bin/bash
for SECONDARY in $(sudo docker service ls |grep secondary |awk '{print $2}')
do
INSPECT=$(sudo docker inspect "$SECONDARY")
ATSIGN=$(echo "$INSPECT" | jq -r ' .[0].Spec.TaskTemplate.ContainerSpec.Args[1]')
PORT=$(echo "$INSPECT" | jq -r ' .[0].Spec.TaskTemplate.ContainerSpec.Args[3]')
MOUNT=$(echo "$INSPECT" | jq -r ' .[0].Spec.TaskTemplate.ContainerSpec.Mounts[]
| select(.Target | IN ("/atsign","/atsign/storage")) | .Source')
echo "$SECONDARY,$ATSIGN,$PORT,$MOUNT"
done
@cpswan - I have created the script that will look for secondary "docker-compose.yml" and check for "/atsign/config" if exist delete it and update the stack.
Script -
#!/bin/bash
#for UUID in $(docker stack ls | grep -Ev "portainer|traefik|NAME" | awk '{print $1}')
for UUID in 419ebfe0-86df-55b9-b733-8156b1f3defa
do
log='secoundaries-logs/temp/'$UUID'-create_log.txt'
echo "=============================" >> $log 2>&1
echo "Secondarie migration started" >> $log 2>&1
secoundary_full_path=$(find /gluster/@/secondaries/ -name "$UUID" -print)
echo $secoundary_full_path >> $log 2>&1
grep -R '/atsign/config' $secoundary_full_path/docker-compose.yml
if [ $? -eq 0 ]; then
echo "Mount point(/atsign/config) exist"
sed -i '/\/atsign\/config/d' $secoundary_full_path/docker-compose.yml
docker stack deploy --compose-file $secoundary_full_path/docker-compose.yml $UUID --with-registry-auth
echo "Secondarie migration completed" >> $log 2>&1
echo "===============================" >> $log 2>&1
else
echo "Secoundary Not exist"
fi
done
After upgrade -
version: "3.8"
services:
secondary:
image: reg.buzz.do-sf2.atsign.zone/atsigncompany/secondary:dev_env
command: "-a @parliamentary1taekwondo -p 2351 -s 854812a8948bc4c94149dfa296d46d86249a634df719d46fd78609f6cee09f15862d8ccee56f981a6f975427786e281855f51f750aa0ab51f2a6da292f1f168a"
deploy:
placement:
constraints: [node.role == worker]
resources:
limits:
cpus: '0.25'
memory: '50M'
reservations:
#cpus: '0.01'
memory: '6M'
ports:
- target: 2351
published: 2351
volumes:
- "/gluster/@/secondaries/14/8d/5b/57/148d5b573c88ed387208ce420c6706bb/8d7de2eb-2aa3-54f9-bb39-453745ebb9a4:/atsign"
- "/etc/letsencrypt/live/8d7de2eb-2aa3-54f9-bb39-453745ebb9a4.buzz.do-sf2.atsign.zone:/atsign/certs"
networks:
second: {}
networks:
second:
external: true
name: secondaries
Please review and share your feedback.
Performed testing on below secondaries. 583wrong parliamentary1taekwondo correspondingshaokao5
@kumarnarendra701 the script is missing the most important piece, which is to mount /atsign/storage rather than /atsign
@cpswan - Please find updated script.
#!/bin/bash
#for UUID in $(docker stack ls | grep -Ev "portainer|traefik|NAME" | awk '{print $1}')
for UUID in 9eda0f61-afe5-593f-bcf7-c884a7d9709b
do
log='secoundaries-logs/temp/'$UUID'-create_log.txt'
echo "=============================" >> $log 2>&1
echo "Secondarie migration started" >> $log 2>&1
secoundary_full_path=$(find /gluster/@/secondaries/ -name "$UUID" -print)
echo $secoundary_full_path >> $log 2>&1
grep -R '/atsign/config' $secoundary_full_path/docker-compose.yml
if [ $? -eq 0 ]; then
echo "Mount point(/atsign/config) exist"
cp -rpf $secoundary_full_path/docker-compose.yml $secoundary_full_path/docker-compose.yml-$(date '+%Y-%m-%d')
sed -i '/\/atsign\/config/d' $secoundary_full_path/docker-compose.yml
sed -i "s/$UUID\:\/atsign/$UUID\/storage\:\/atsign\/storage/g" $secoundary_full_path/docker-compose.yml
docker stack deploy --compose-file $secoundary_full_path/docker-compose.yml $UUID --with-registry-auth
echo "Secondarie migration completed" >> $log 2>&1
echo "===============================" >> $log 2>&1
else
echo "Secoundary Not exist"
fi
done
docker-compose.yml
version: "3.8"
services:
secondary:
image: reg.buzz.do-sf2.atsign.zone/atsigncompany/secondary:dev_env
command: "-a @acutesaudathedeal3 -p 2459 -s c4857ae1ea8eac9e40a08e8857d8389564167745020fd64333ce2b9ace9a37416a4aa0193d55b075af651f936b6da82000435770c355e65aef23881408cd3547"
deploy:
placement:
constraints: [node.role == worker]
resources:
limits:
cpus: '0.25'
memory: '50M'
reservations:
#cpus: '0.01'
memory: '6M'
ports:
- target: 2459
published: 2459
volumes:
- "/gluster/@/secondaries/80/dd/08/96/80dd089641a34c95fdf950e9c66c60fd/a6582237-a0d2-5987-89f3-290f119d7e7f/storage:/atsign/storage"
- "/etc/letsencrypt/live/a6582237-a0d2-5987-89f3-290f119d7e7f.buzz.do-sf2.atsign.zone:/atsign/certs"
networks:
second: {}
networks:
second:
external: true
name: secondaries
@kumarnarendra701 LGTM, though you might want to s/ecound/econd/ (which is right in all the places it needs to be, but wrong everywhere else).
@cpswan - Updated script
#!/bin/bash
#for UUID in $(docker stack ls | grep -Ev "portainer|traefik|NAME" | awk '{print $1}')
for UUID in 9eda0f61-afe5-593f-bcf7-c884a7d9709b
do
log='secondaries-logs/temp/'$UUID'-create_log.txt'
echo "=============================" >> $log 2>&1
echo "Secondary migration started" >> $log 2>&1
secondary_full_path=$(find /gluster/@/secondaries/ -name "$UUID" -print)
echo $secondary_full_path >> $log 2>&1
grep -R '/atsign/config' $secondary_full_path/docker-compose.yml
if [ $? -eq 0 ]; then
echo "Mount point(/atsign/config) exist"
cp -rpf $secondary_full_path/docker-compose.yml $secondary_full_path/docker-compose.yml-$(date '+%Y-%m-%d')
sed -i '/\/atsign\/config/d' $secondary_full_path/docker-compose.yml
sed -i "s/$UUID\:\/atsign/$UUID\/storage\:\/atsign\/storage/g" $secondary_full_path/docker-compose.yml
docker stack deploy --compose-file $secondary_full_path/docker-compose.yml $UUID --with-registry-auth
echo "Secondary migration completed" >> $log 2>&1
echo "===============================" >> $log 2>&1
else
echo "Secondary Not exist"
fi
done
Thanks @kumarnarendra701 LGTM
Have you also updated the secondary creation script?
@cpswan - I was waiting for the final review, now I'll make changes to the creation script, and also let me should I run the script for stacks migration.
The first priority is to get the creation script changed. So that we're not creating new secondaries with mounts to /atsign
Once that's done we can remount the existing atsigns, and change their docker-compose files so that they won't get reverted.
As usual, let's do staging first, and once that's done move on to production.
NB there are some secondaries that have already been moved by hand (but only their live config, not their compose files).
@cpswan - I have updated the base docker-compose file and create bash script.
version: "3.8"
services:
secondary:
image: reg.buzz.do-sf2.atsign.zone/atsigncompany/secondary:dev_env
command: "-a @ATSIGN -p PORT -s ATSECRET"
environment:
- rootServerUrl=${root_server_url}
deploy:
placement:
constraints: [node.role == worker]
resources:
limits:
cpus: '0.25'
memory: '50M'
reservations:
#cpus: '0.01'
memory: '6M'
ports:
- target: PORT
published: PORT
volumes:
- "/gluster/@/secondaries/DIR/NAME/storage:/atsign/storage"
- "/etc/letsencrypt/live/UUID.buzz.do-sf2.atsign.zone:/atsign/certs"
networks:
second: {}
networks:
second:
external: true
name: secondaries
Also, added the below line in the creation script, because we need a storage directory for the mount. mkdir -p ../secondaries/$DIR/$UUID/storage
Thanks @kumarnarendra701
Can you now please update all the existing secondaries in staging
@kumarnarendra701 reports that this has now been done for staging, and I've confirmed with some spot checks.
Changes haven't been applied to prod yet, so needs to be carried over to PR26
@cpswan - Updated creation script on prod and migrated 5 secondaries for monitoring if all good for some time then I'll trigger for all stacks.
@kumarnarendra701 please go ahead with the rest of the secondaries in prod. We know that this change works, and so if we hit issues it will be because of some peculiarity in a given secondary rather than because the overall approach is wrong.
@cpswan - Migration script started on prod Tmux Session - "tmux attach -t 5".
@kumarnarendra701 has confirmed that the prod remount script has now completed
Is your feature request related to a problem? Please describe. In trying to understand why the issues seen in #290 were passing end to end tests I looked for differences between the environments.
Here's what we do in dess (used for cicd1 & cicd2 instances in end to end tests):
If I break it down line by line:
/atsign/certs
is mapped to where the certs are held on the host. FWIW I don't see any good reason for this being writable from the container, so it should probably now be:rw
/archive/$FQDN
is mapped to the cert archive. WHY?/atsign/storage
is mapped to a place on the host to keep the state of the secondary.Turning to the staging environment (or production) we have something like:
Again breaking it down line by line:
/atsign
is mounted to the place on the host to keep the state of the secondary. Why not/atsign/storage
?/atsign/certs
is mapped to where the certs are held on the host./atsign/config
unknown why this is here?NB this time there's no attempt to map archived certs.
Describe the solution you'd like
I think we can just mount:
/atsign/storage
:rw to where we keep state/atsign/certs
:ro to where we keep certsThis should be uniform across dess, staging and production.
Describe alternatives you've considered We could put the certs into the state area, and reduce down to one mount point. But that might create extra complexity for cert management.