search

MichaIng / DietPi

Lightweight justice for your single-board computer!
https://dietpi.com/
GNU General Public License v2.0
4.8k stars 494 forks source link

DietPi-Software | Portainer: Failsafe reinstall/uninstall by forcing image tag #4791

Open cocoflan opened 2 years ago

cocoflan commented 2 years ago

Trick to update Portainer on Dietpi

  1. Take a backup of your Portainer settings in Portainer under the settings.

  2. Uninstall Portainer from dietpi-software.

  3. Reinstall Portainer from dietpi-software.

  4. Reopen Portainer and choose Restore Portainer from backup.

Your Portainer system is updated and all settings are back.

Joulinar commented 2 years ago

simply run dietpi-software reinstall 185 as described on our online docs https://dietpi.com/docs/software/programming/#portainer

cocoflan commented 2 years ago

so the config will also be saved by doing that?

Joulinar commented 2 years ago

yes, we drop container only and will keep the config files which are located outside the container on an own docker volume

root@DietPiProd:~# docker volume ls
DRIVER    VOLUME NAME
local     portainer_data
root@DietPiProd:~#

Just verified on my prod system and updated from 2.6.1 to 2.9.

cocoflan commented 2 years ago

Ok thx for the info.

Joulinar commented 2 years ago

ok I'm going to close the issue. Feel free to reopen if needed.

cocoflan commented 2 years ago

I tested your way on one of my systems and it makes an error, when reinstalling, only Portainer needs to be reinstall and update, so i think there is a problem, when i use my upgrade procedure no problem, tested it on 3 systems and only portainer is reinstalled and updated. The system tested with dietpi-software reinstall 185, i get error.

Error response from daemon: conflict: unable to delete 8377e6877145 (cannot  │
│ be forced) - image is being used by running container f43cf3f4cc41
Joulinar commented 2 years ago

Our script is going to identify the container and will try to drop it. Once done the image will be removed. But somehow the image is still in use in your case.

Could you share following

docker ps -a
docker images -a
cocoflan commented 2 years ago 
CONTAINER ID   IMAGE                            COMMAND                  CREATED       STATUS                     PORTS                                        NAMES
f43cf3f4cc41   portainer/portainer-ce:latest    "/portainer"             3 days ago    Up 11 minutes              8000/tcp, 9443/tcp, 0.0.0.0:9002->9000/tcp   portainer
portainer/portainer-ce       latest         8377e6877145   7 days ago      251MB
debian                       buster         730bc7b177ec   3 weeks ago     114MB
maltrail                     latest         b3b676f60d91   4 weeks ago     516MB
<none>                       <none>         a45caa94b927   4 weeks ago     516MB
<none>                       <none>         326324005abb   4 weeks ago     516MB
<none>                       <none>         67ce1fed89b3   4 weeks ago     516MB
<none>                       <none>         65e57316e270   4 weeks ago     516MB
<none>                       <none>         6ee7205ee6f6   4 weeks ago     516MB
<none>                       <none>         0fdbda74ed16   4 weeks ago     516MB
<none>                       <none>         6a65e6e23c19   4 weeks ago     516MB
<none>                       <none>         57cff1aaf702   4 weeks ago     516MB
ubuntu                       focal          fb52e22af1b0   4 weeks ago     72.8MB
portainer/portainer-ce       <none>         5121527e11b8   4 weeks ago     206MB
cocoflan commented 2 years ago

I am going to use my way, worked on 3 systems without a problem.

cocoflan commented 2 years ago

Must be something wrong with my setup sorry, i am removing docker and portainer and wil restore with my config file, i am running bullseye, maby something with that to do.

Joulinar commented 2 years ago

ahh the image name is different than we expect. It is portainer/portainer-ce:latest but we expect portainer/portainer-ce. I guess because you have 2 images running. You just need to remove the 2nd portainer image 

docker rmi 5121527e11b8
cocoflan commented 2 years ago

Yes indeed, that was the error, ok thx for your info clear info thx.

Joulinar commented 2 years ago

how does docker ps -a looks now?

cocoflan commented 2 years ago

i am restoring my system, i let you know soon.

cocoflan commented 2 years ago

Strange, restored my system and portainer is updated now.

MichaIng commented 2 years ago

Just verified on a fresh install it is shown without :lastest even that the latest version is installed:

[ INFO ] DietPi-Software | docker run -d -p 9002:9000 --name=portainer --restart=always -v /run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce, please wait...
Unable to find image 'portainer/portainer-ce:latest' locally
latest: Pulling from portainer/portainer-ce
7721cab3d696: Pulling fs layer
0645e7e2a110: Pulling fs layer
86253b00ae0d: Pulling fs layer
0645e7e2a110: Verifying Checksum
0645e7e2a110: Download complete
7721cab3d696: Download complete
7721cab3d696: Pull complete
0645e7e2a110: Pull complete
86253b00ae0d: Verifying Checksum
86253b00ae0d: Download complete
86253b00ae0d: Pull complete
Digest: sha256:689908f8396e840e7fcff09ce85532291bab9a907b9801ff9c9ded83a18167b9
Status: Downloaded newer image for portainer/portainer-ce:latest
5724687e6c661dd4335f133c4ec9da3a6cf759c7856bec01bb08d0a191d286a7
[  OK  ] DietPi-Software | docker run -d -p 9002:9000 --name=portainer --restart=always -v /run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce

 DietPi-Software
─────────────────────────────────────────────────────
 Step: Finalising install

[  OK  ] DietPi-Software | systemctl daemon-reload
[ SUB1 ] DietPi-Services > dietpi_controlled
[  OK  ] DietPi-Services | dietpi_controlled : docker
[  OK  ] DietPi-Services | dietpi_controlled : cron

 DietPi-Software
─────────────────────────────────────────────────────
 Step: Install completed

[  OK  ] DietPi-Survey | Sending survey data
[ SUB1 ] DietPi-Services > restart
[  OK  ] DietPi-Services | restart : docker
[  OK  ] DietPi-Services | restart : cron
2021-09-30 23:36:09 root@VM-Bullseye:~# docker ps -a
docker images -a
CONTAINER ID   IMAGE                    COMMAND        CREATED          STATUS          PORTS                                                           NAMES
5724687e6c66   portainer/portainer-ce   "/portainer"   51 seconds ago   Up 43 seconds   8000/tcp, 9443/tcp, 0.0.0.0:9002->9000/tcp, :::9002->9000/tcp   portainer
REPOSITORY               TAG       IMAGE ID       CREATED      SIZE
portainer/portainer-ce   latest    8377e6877145   7 days ago   251MB

We could relax the check to allow a version suffix, but it must be assured that we are not trying to remove a different image than what was installed or two images (which match the relaxed pattern). If a second image portainer/portainer-ce is installed, both get a version suffix? In such case, is it possible to determine which of both is that one which did hold the previously removed portainer/portainer-ce container? 🤔

cocoflan commented 2 years ago

docker ps -a

PORTS                                        NAMES
f43cf3f4cc41   portainer/portainer-ce:latest    "/portainer"             3 days ago    Up 3 minutes               8000/tcp, 9443/tcp, 0.0.0.0:9002->9000/tcp   portainer
Joulinar commented 2 years ago

ok and the list of images? docker images -a

cocoflan commented 2 years ago 
portainer/portainer-ce       latest         8377e6877145   7 days ago      251MB
debian                       buster         730bc7b177ec   3 weeks ago     114MB
maltrail                     latest         b3b676f60d91   4 weeks ago     516MB
<none>                       <none>         a45caa94b927   4 weeks ago     516MB
<none>                       <none>         326324005abb   4 weeks ago     516MB
<none>                       <none>         67ce1fed89b3   4 weeks ago     516MB
<none>                       <none>         65e57316e270   4 weeks ago     516MB
<none>                       <none>         6ee7205ee6f6   4 weeks ago     516MB
<none>                       <none>         0fdbda74ed16   4 weeks ago     516MB
<none>                       <none>         6a65e6e23c19   4 weeks ago     516MB
<none>                       <none>         57cff1aaf702   4 weeks ago     516MB
ubuntu                       focal          fb52e22af1b0   4 weeks ago     72.8MB
portainer/portainer-ce       <none>         5121527e11b8   4 weeks ago     206MB
Joulinar commented 2 years ago

pls do docker rmi 5121527e11b8

how does docker ps -a looks now

cocoflan commented 2 years ago

And Portainer is now updated after i did a restore, strange.

MichaIng commented 2 years ago

Ah there are two portainer images. Probably the container has the suffix to indicate that it is running on the image with the latest tag and not in the one with <none> tag.

Joulinar commented 2 years ago

correct @MichaIng . I see you getting used to docker 🤣

MichaIng commented 2 years ago

So the letter image is unused and can be removed, right?

docker rmi 5121527e11b8

as suggested by Joulinar before. Probably then the container also does not show the suffix anymore.

I see you getting used to docker 🤣

I never ran multiple Docker containers or multiple images, so yeah 😄.

cocoflan commented 2 years ago

Ok it's ok now 

portainer/portainer-ce       latest         8377e6877145   7 days ago      251MB
debian                       buster         730bc7b177ec   3 weeks ago     114MB
maltrail                     latest         b3b676f60d91   4 weeks ago     516MB
<none>                       <none>         a45caa94b927   4 weeks ago     516MB
<none>                       <none>         326324005abb   4 weeks ago     516MB
<none>                       <none>         67ce1fed89b3   4 weeks ago     516MB
<none>                       <none>         65e57316e270   4 weeks ago     516MB
<none>                       <none>         6ee7205ee6f6   4 weeks ago     516MB
<none>                       <none>         0fdbda74ed16   4 weeks ago     516MB
<none>                       <none>         6a65e6e23c19   4 weeks ago     516MB
<none>                       <none>         57cff1aaf702   4 weeks ago     516MB
Joulinar commented 2 years ago

and what about docker ps -a

cocoflan commented 2 years ago 
CONTAINER ID   IMAGE                            COMMAND                  CREATED       STATUS                     PORTS                                        NAMES
f43cf3f4cc41   portainer/portainer-ce:latest    "/portainer"             3 days ago    Up 13 minutes              8000/tcp, 9443/tcp, 0.0.0.0:9002->9000/tcp   portainer
cocoflan commented 2 years ago

i think it will be watchtower, did install that some time ago and it auto updates containers.

cocoflan commented 2 years ago

Have a question, i am experimenting with unbound in combination with pi-hole to run recursive dns, works good.

Something to use on dietpi?

https://docs.pi-hole.net/guides/dns/unbound/

Joulinar commented 2 years ago

I guess the image got lable during install as Docker detected a 2nd image with same name but different version? best would be drop the container and remove this image as well. Once done reinstall Portianer

docker rm -f f43cf3f4cc41
docker rmi 8377e6877145
docker run -d -p 9002:9000 --name=portainer --restart=always -v /run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce
MichaIng commented 2 years ago

Okay, the suffix seems to be burnt-in now, at least until a Docker or server restart. Hmm, can we be sure that the Portainer image installed via dietpi-software has and keeps the latest tag and that the container hence may contain it as well (and no other tag and no other container))? Or can this change?

watchtower updates images and seems to move the container over to the new image, so probably it is responsible for the second image. But not sure whether it is also responsible for the suffix, or whether this is done by Docker itself when multiple images are present (which would make sense).

Joulinar commented 2 years ago

DietPi offers to install PiHole + Unbound automatically combined.

EDIT BTW: quite some unused container that could be removed to gain a lot of disk space

<none>                       <none>         a45caa94b927   4 weeks ago     516MB
<none>                       <none>         326324005abb   4 weeks ago     516MB
<none>                       <none>         67ce1fed89b3   4 weeks ago     516MB
<none>                       <none>         65e57316e270   4 weeks ago     516MB
<none>                       <none>         6ee7205ee6f6   4 weeks ago     516MB
<none>                       <none>         0fdbda74ed16   4 weeks ago     516MB
<none>                       <none>         6a65e6e23c19   4 weeks ago     516MB
<none>                       <none>         57cff1aaf702   4 weeks ago     516MB
cocoflan commented 2 years ago

ok so only need to config the system to use the unbound recursive dns, like i did.

cocoflan commented 2 years ago

Yes i know it is a test system that are indeed parts to remove, i think from watchtower, i remove all of it. just did some test on that system.

cocoflan commented 2 years ago

Thnx for your time Joulinar and MichaIng.

Greetings

MichaIng commented 2 years ago

ok so only need to config the system to use the unbound recursive dns, like i did.

If Pi-hole is installed already, you can simply install Unbound on top of it and it will automatically configure Unbound and Pi-hole to work together with Pi-hole sending requests to Unbound, and Unbound being the recursive DNS server, sending requests to DNS root servers.

Joulinar commented 2 years ago

@MichaIng if there is no image available locally, latest image will be pulled

cocoflan commented 2 years ago

Ok i used the info on pi-hole with the config file. and it runs very well.

MichaIng commented 2 years ago

The alternative for Unbound:

dietpi-software install 182

No manual configuration required. But the config from the Pi-hole docs works as well, of course.

Joulinar commented 2 years ago

our config is nearly same. Just smaller differences https://github.com/MichaIng/DietPi/blob/master/.conf/dps_182/unbound.conf

but we are moving off topic

cocoflan commented 2 years ago

ok will check that, nice to talk, see you later and many thx.

MichaIng commented 2 years ago

@MichaIng if there is no image available locally, latest image will be pulled

And if there is already an image? I'm just wondering if there is a way to reliably know which container was installed by dietpi-software and which image it runs on. Okay the letter is shown in the container list, including the tag, but for the container? I just wonder whether it is safe enough to also allow the latest suffix to match for removal or whether we then risk to remove a different container than intended. If in case of two containers (or once two, so that the last one may still have the suffix) we cannot be sure whether we remove the right one, I think it's better to keep the reinstall failing so that admins need to manually sort things, before we guess wrong.

Joulinar commented 2 years ago

just to complete, both are working fine on my prod system

root@DietPiProd:~# dietpi-software list | grep 'unbound\|hole'
ID 93 | =2 | Pi-hole: block adverts for any device on your network | +Git +SQLite +PHP +webserver | https://dietpi.com/docs/software/dns_servers/#pi-hole
ID 182 | =2 | Unbound: validating, recursive, caching DNS resolver | | https://dietpi.com/docs/software/dns_servers/#unbound
root@DietPiProd:~#
Joulinar commented 2 years ago

@MichaIng

I think it's better to keep the reinstall failing so that admins need to manually sort things, before we guess wrong.

Fully agree with this

MichaIng commented 2 years ago

Okay. What we could do is test watchtower and see whether it updates images in a consistent way (i.e. that the new and active image always keeps the latest tag while the old one always looses it's tag). If this is a common container for keeping the others up-to-date, then it may be reasonable to handle it gracefully.

Joulinar commented 2 years ago

What we might could do is to force the usages of :latest by running

docker run -d -p 9002:9000 --name=portainer --restart=always -v /run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce:latest

this will result in

root@DietPiProd:~# docker ps -a
CONTAINER ID   IMAGE                           COMMAND                  CREATED          STATUS          PORTS                                        NAMES
052f73df86b5   portainer/portainer-ce:latest   "/portainer"             51 seconds ago   Up 47 seconds   8000/tcp, 9443/tcp, 0.0.0.0:9002->9000/tcp   portainer

Maybe this is what watchtower does, forcing the tag

Joulinar commented 2 years ago

ok the main issue was that we could not identify the container to drop due to the unexpected name. To be 100% sure we drop our container, we could use an own lable/name for the container like --name=portainer.dietpi

For testing, I'm running 2 Portainer container

root@DietPiProd:~# docker ps -a
CONTAINER ID   IMAGE                    COMMAND                  CREATED         STATUS                  PORTS                                        NAMES
d52fc8b5459e   portainer/portainer-ce   "/portainer"             3 seconds ago   Up Less than a second   8000/tcp, 9443/tcp, 0.0.0.0:9003->9000/tcp   portainer.dietpi
fc17124d1da3   portainer/portainer-ce   "/portainer"             7 minutes ago   Up 7 minutes            8000/tcp, 9443/tcp, 0.0.0.0:9002->9000/tcp   portainer

But we could simply identify our container due to the unique name and drop it

root@DietPiProd:~# docker ps -a | mawk '/portainer\.dietpi?( |$)/{print $1;exit}'
d52fc8b5459e
root@DietPiProd:~#

As well we could assume always to drop latest image, even if there are more version

root@DietPiProd:~# docker images -a
REPOSITORY               TAG       IMAGE ID       CREATED        SIZE
portainer/portainer-ce   latest    53047605f782   7 days ago     216MB
portainer/portainer-ce   2.6.3     bd3c978cdaec   5 weeks ago    157MB
root@DietPiProd:~# docker images portainer/portainer*:latest
REPOSITORY               TAG       IMAGE ID       CREATED      SIZE
portainer/portainer-ce   latest    53047605f782   7 days ago   216MB
root@DietPiProd:~#

btw: why are we using mawk to grep the ID's. It could be so simple using -q option on docker commands 🤣 

docker rm -f $(docker ps --filter "name=portainer.dietpi" -q)
docker rmi $(docker images portainer/portainer*:latest -q)
MichaIng commented 2 years ago

Interesting that in your case, even with two images and containers, those don't have the tag attached to their IMAGE field. Probably in this case not as they have different names already? Actually I don't see any way now to know which image each container is connected to 🤔.

Giving the container a unique name is a good idea, but awesome would be to also give the image a unique tag. Actually, if we decide to always remove the latest image, it may be simpler to go with your previous suggestion to install the container with forced :latest tag and then also remove the container which shows that tag. This is also the only way, from what I see now, to assure that container and image match, as both then show the unique tag. About the concerns we had: Whether uninstalling container and image without tag or doing this with latest tag has both the same chance that an unintended container is uninstalled, I guess. 100% certainty would be only given if we saved the IDs of container and image, but this would likely also break when watchtower was used for updating.

Nice that the docker command allows to filter outputs internally, makes sense to use it instead of mawk.

Okay if we go with latest tag, how do we handle previously installed instances which do not show it? Force a Portainer update on next DietPi update?

Joulinar commented 2 years ago

Theoretically we could force an update of the container and change container name to a unique one or we simply use docker rename

root@DietPi3:~# docker ps -a
CONTAINER ID   IMAGE                 COMMAND        CREATED         STATUS         PORTS                                        NAMES
7e01e793a248   dietpi/portainer-ce   "/portainer"   4 minutes ago   Up 4 minutes   8000/tcp, 9443/tcp, 0.0.0.0:9002->9000/tcp   portainer
root@DietPi3:~#
root@DietPi3:~# docker rename portainer portainer.dietpi
root@DietPi3:~#
root@DietPi3:~# docker ps -a
CONTAINER ID   IMAGE                 COMMAND        CREATED         STATUS          PORTS                                        NAMES
7e01e793a248   dietpi/portainer-ce   "/portainer"   5 minutes ago   Up 10 seconds   8000/tcp, 9443/tcp, 0.0.0.0:9002->9000/tcp   portainer.dietpi

For images, we could create a new name/tag for the image to identify it later on.

root@DietPi3:~# docker images -a
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE
root@DietPi3:~# docker pull portainer/portainer-ce
Using default tag: latest
latest: Pulling from portainer/portainer-ce
7721cab3d696: Pull complete
0645e7e2a110: Pull complete
5a0aaee757e7: Pull complete
Digest: sha256:689908f8396e840e7fcff09ce85532291bab9a907b9801ff9c9ded83a18167b9
Status: Downloaded newer image for portainer/portainer-ce:latest
docker.io/portainer/portainer-ce:latest
root@DietPi3:~# docker images -a
REPOSITORY               TAG       IMAGE ID       CREATED      SIZE
portainer/portainer-ce   latest    53047605f782   9 days ago   216MB
root@DietPi3:~# docker image tag portainer/portainer-ce:latest dietpi/portainer-ce:latest
root@DietPi3:~# docker images -a
REPOSITORY               TAG       IMAGE ID       CREATED      SIZE
dietpi/portainer-ce      latest    53047605f782   9 days ago   216MB
portainer/portainer-ce   latest    53047605f782   9 days ago   216MB
root@DietPi3:~# docker rmi portainer/portainer-ce:latest
Untagged: portainer/portainer-ce:latest
Untagged: portainer/portainer-ce@sha256:689908f8396e840e7fcff09ce85532291bab9a907b9801ff9c9ded83a18167b9
root@DietPi3:~# docker images -a
REPOSITORY            TAG       IMAGE ID       CREATED      SIZE
dietpi/portainer-ce   latest    53047605f782   9 days ago   216MB
root@DietPi3:~# docker run -d -p 9002:9000 --name=portainer.dietpi --restart=always -v /run/docker.sock:/var/run/docker.sock -v portainer_data:/data dietpi/portainer-ce
7e01e793a2484771e9a184794fe12247306987cd2e22872ea3d79292ae734dc8
root@DietPi3:~# docker ps -a
CONTAINER ID   IMAGE                 COMMAND        CREATED          STATUS          PORTS                                        NAMES
7e01e793a248   dietpi/portainer-ce   "/portainer"   55 seconds ago   Up 52 seconds   8000/tcp, 9443/tcp, 0.0.0.0:9002->9000/tcp   portainer.dietpi

Quest ist how failsafe we like to do it? Are we doing some over-engineering? How common will it be that people play with their container? As long as nobody is dropping the docker volume, no customer information should be lost. More simple would be to assume defaults as we install it?