pgAdmin4 images - Githubissues

SensorsIot / IOTstack

Docker stack for getting started on IOT on the Raspberry PI

GNU General Public License v3.0

1.43k stars 303 forks source link

pgAdmin4 images #647

Closed gpongelli closed 1 year ago

gpongelli commented 1 year ago

Hi, I've recently worked on making raspberry images for pgAdmin4 (tags) , can you add it to the stack?

Hope you appreciate it. Thanks.

Paraphraser commented 1 year ago

Hi Gabriele,

I'll take a look at doing this.

I'm not the admin for IOTstack, just an ordinary user, but I've done a few of these so I've got a good idea of what's involved.

What I'm assuming you've done is to start with github.com/thaJeztah/pgadmin4-docker and use a local Dockerfile to build an ARM-specific image. Yes?

If yes then the information at hub.docker.com/r/gpongelli/pgadmin4-docker-armv7 should probably be changed to reflect that. For example:

this:

docker run -d -p 5050:5050 -v /Users/me/pgadmin:/pgadmin thajeztah/pgadmin4

should probably be:

docker run -d -p 5050:5050 -v /Users/me/pgadmin:/pgadmin gpongelli/pgadmin4-docker-armv7

If you don't have the correct details in the DockerHub documentation then people getting to this image via DockerHub are going to pull the non-ARM version and wonder why it doesn't work.

I realise you've already done this at github.com/gpongelli/pgadmin4-docker-armv7. I don't understand the mechanism by which the DockerHub page gets updated, just pointing out that it's out of sync.

Also, if I just "pull" your image "as is", this is what happens:

$ docker pull gpongelli/pgadmin4-docker-armv7
Using default tag: latest
Error response from daemon: manifest for gpongelli/pgadmin4-docker-armv7:latest not found: manifest unknown: manifest unknown

What I need to do is to adopt the actual tag you've used (ie just as you've documented it):

$ docker pull gpongelli/pgadmin4-docker-armv7:6.19-py3.11
6.19-py3.11: Pulling from gpongelli/pgadmin4-docker-armv7
e44ba29d168a: Pull complete 
…

There's nothing wrong with pulling an explicit tag but it does mean that every IOTstack installation using this service will be pinned to that version - forever.

I'd rather not set an explicit tag in an IOTstack service definition because it creates an ongoing maintenance problem.You push a new image on DockerHub and nobody gets it until (a) we update the template, and (b) every single user of the service becomes aware of the update and applies it by hand. The IOTstack menu is quite good at initial setup, not so good at propagating template changes.

At the risk of telling you things you already know, it's more usual to post on DockerHub both an explicit tag (like :6.19-py3.11) and a generic tag like latest that both point to the same image. Then, when you build a new image, you leave :6.19-py3.11 in place, upload the new image (say :6.20-py3.11) and then latest points to 6.20, while 6.19 is left in place in case the new image doesn't work for somebody and they need to roll back.

I hope that all makes sense.

If you can fix the documentation and create an image with a latest tag then I'll set about doing what needs to be done on the IOTstack side.

Phill

gpongelli commented 1 year ago

Hi Phil,

What I'm assuming you've done is to start with github.com/thaJeztah/pgadmin4-docker and use a local Dockerfile to build an ARM-specific image. Yes?

Not properly from that image, but from the one I’ve added on the “original credit” paragraph.

If yes then the information at hub.docker.com/r/gpongelli/pgadmin4-docker-armv7 should probably be changed to reflect that. For example:
this:
docker run -d -p 5050:5050 -v /Users/me/pgadmin:/pgadmin thajeztah/pgadmin4
should probably be:
docker run -d -p 5050:5050 -v /Users/me/pgadmin:/pgadmin gpongelli/pgadmin4-docker-armv7
If you don't have the correct details in the DockerHub documentation then people getting to this image via DockerHub are going to pull the non-ARM version and wonder why it doesn't work.

Right, in fact I’ve done a commit on GitHub repo to fix the read me just yesterday.

I realise you've already done this at github.com/gpongelli/pgadmin4-docker-armv7. I don't understand the mechanism by which the DockerHub page gets updated, just pointing out that it's out of sync.

the page on dockerhub is updated after a build is made. Next one is planned on February. I can also manually change that description.

Also, if I just "pull" your image "as is", this is what happens:
$ docker pull gpongelli/pgadmin4-docker-armv7
Using default tag: latest
Error response from daemon: manifest for gpongelli/pgadmin4-docker-armv7:latest not found: manifest unknown: manifest unknown
What I need to do is to adopt the actual tag you've used (ie just as you've documented it):
$ docker pull gpongelli/pgadmin4-docker-armv7:6.19-py3.11
6.19-py3.11: Pulling from gpongelli/pgadmin4-docker-armv7
e44ba29d168a: Pull complete 
…
There's nothing wrong with pulling an explicit tag but it does mean that every IOTstack installation using this service will be pinned to that version - forever.

this is a personal choice and also it’s bound to how the original project was: build images for all the combinations of “pgAdmin, python, OS” tuple , so the “latest” tag has no sense. When I’ve started my project, I’ve also added a way to limit the combinations due to timeout execution of GitHub actions pipeline, reducing everything to the most recent combination of language, pgadmin version, and the only alpine dietro that works.

Consider that, if you use the latest alpine distro docker tag, pgadmin does not work (I’ve done many test and lost a week for them), so I’ve pinned alpine to only the working version.

Then, I use IOTstack since gcgarner project and during my years of usage with “latest” tag I’ve encountered many issue when images were not correctly built (because they’re from nightly build pipelines), so I switched to pinned docker images that saved me times.

I'd rather not set an explicit tag in an IOTstack service definition because it creates an ongoing maintenance problem.You push a new image on DockerHub and nobody gets it until (a) we update the template, and (b) every single user of the service becomes aware of the update and applies it by hand. The IOTstack menu is quite good at initial setup, not so good at propagating template changes.

the point that led people to change the image in use should be: I need a new feature available only on new version or the version installed is not working properly. And it needs to be done manually. Using latest does not assure that the pointing version fixes the issue or it’s stable.

then, from my experience with iotstack, the generated docker file needs manual changes (I did add volumes and network by hand, now I don’t know if they’re managed during setup execution).

Paraphraser commented 1 year ago

I'm not wedded to "latest", just that it's the default if the tag is omitted. I'd be happy with "main" or "current" or anything which doesn't require either updates to the template or expects users to figure out the version from DockerHub and then change the tag themselves.

Do you follow the IOTstack Discord channel? If you do, I'm sure you've noticed plenty of people who are using IOTstack as a means to an end with no real expertise in Unix, Docker or anything else. Some, in fact, who are challenged by the need to use a text editor on a compose file. It's the needs of people in that group that I try to keep front of mind when making decisions. So, while I grok your reasoning, the idea still makes me uncomfortable. I'd urge you to reconsider.

To be clear, I have no trouble saying to someone, "if version B of container X doesn't work, edit your compose file to pin to version A" and, if asked, suggest strategies for doing that. Having made such a change, it's the user's responsibility to remember having done it and how to undo it. That doesn't concern me. What concerns me is (a) the maintenance problem - which includes who is going to track the need to update the template, and (b) that the IOTstack menu is not geared towards propagating changes like this.

At this point I've been assuming that this container and Postgres would be independent templates (ie a user would need to pick both in the menu) but what you've just said has me wondering whether it might not be better to follow the NextCloud model where NextCloud and its MariaDB service definitions are in the same template? If we did wind up pinning pgadmin4 to a specific version, it might be appropriate to pin Postgres to a matching version. Thoughts?

gpongelli commented 1 year ago

Hi Phil, I've just searched on how to set a default tag name, different than "latest", on docker and AFAICT seems not possible. So I've just added a "latest" tag and updated the description page.

Do you follow the IOTstack Discord channel?

No I don't, sorry.

At this point I've been assuming that this container and Postgres would be independent templates (ie a user would need to pick both in the menu) but what you've just said has me wondering whether it might not be better to follow the NextCloud model where NextCloud and its MariaDB service definitions are in the same template? If we did wind up pinning pgadmin4 to a specific version, it might be appropriate to pin Postgres to a matching version. Thoughts?

Yes, they are independent, and it's better to have them separated. There could be people who needs only database (and look the data through grafana, for example), people that needs postgresql+any extension (like postgis or timescaledb, like me) that does not need pgadmin,, those who use admirer, and some of previous combination that want to use pgadmin.

Ps I'm new to docker and dockerhub, so if you know how to set a default tag, please let me know.

Paraphraser commented 1 year ago

OK. Separate service definitions it is.

But, hey man, you've actually pushed an image to DockerHub. Waaay ahead of me!

However, as I understand it, you do as you do now to build the local image with your chosen tag and push that to DockerHub, then you use docker tag to associate the "latest" tag with the same image, then push that. Something like:

B8DB430F-8757-4FF0-A1B2-5C4A5F692B61

My understanding is docker tag just manipulates metadata - doesn't make a copy.

You can see I used the imageID as the first argument but you could also use gpongelli/pgadmin4-docker-armv7:6.19-py3.11. I mostly use the "tag" command after a "pull" to re-tag any <none> images as prior. That way, if a newly-pulled image doesn't work, I can revert by changing the compose file to :prior without having to go to DockerHub to figure out the actual tag of the last known good image. I got into that habit when I was wrong-footed by someone who didn't leave past images on DockerHub. I had a broken container and no way back. Fortunately, I run both a "live" and a "test" Pi so I was able to go to the live Pi, export the working image to an archive, move it to the test Pi, load it, tag it, and keep truckin'.

Paraphraser commented 1 year ago

Hello again, Gabriele.

Sorry for not getting back to you sooner. I was kinda waiting for an email notification from GitHub about an update on this issue to alert me after you posted latest.

I just cycled back to check and noticed the tag is there.

Now I find I'm stuck. These are in no particular order.

Here's a service definition for reference:

  pg_admin:
    container_name: pg_admin
    image: gpongelli/pgadmin4-docker-armv7
    platform: linux/arm/v7
    restart: unless-stopped
    environment:
    - TZ=${TZ:-Etc/UTC}
    ports:
    - 5050:5050
    volumes:
    - ./volumes/pg_admin:/pgadmin

The first issue is consistent use of pg_admin throughout.

IOTstack conventions assume the service name, container name, and top level folder on the left hand side of the persistent store will be the same.

I've adopted pg_admin for the moment but I don't really care whether we wind up calling it that or pgadmin, or pgAdmin, or something else.

Next issue (I won't call it a problem). There's no ARMv8 architecture. I realise you explicitly posted ARMv7 but I only have 64-bit Bullseye systems to test on so this occurs when I try to bring up the container:

The requested image's platform (linux/arm/v7) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

I'm unlikely to be the only IOTstack user running 64-bit Bullseye so…

I've silenced the error via the added:

platform: linux/arm/v7

but that means it's stuck on ARMv7. I'll leave it to you to decide whether that's acceptable or if you want to build an ARMv8 image at the same time.

Yes, project already growing like topsy. No good deed goes unpunished. And all that. I sympathise. I'm just the messenger.

The container respects time-zone so I've added:

    - TZ=${TZ:-Etc/UTC}

The next issue concerns the volumes clause. This is a bit of a can of worms so please bear with me.

As written on your DockerHub/GitHub page, you have:

- ./volumes/pg_admin/servers.json:/pgadmin/servers.json

Given a clean install (where ./volumes/pg_admin does not exist, if you "up" the container like that, the result is:

$ tree -apug ~/IOTstack/volumes/pg_admin
/home/pi/IOTstack/volumes/pg_admin
└── [drwxr-xr-x root     root    ]  servers.json

1 directory, 0 files

Notice that servers.json is a directory.

Now, strangely enough, pg_admin works properly in this situation. I can create a connection to the postgres container and all is smooth and sweatless.

The log reports:

$ docker logs pg_admin 
NOTE: Configuring authentication for DESKTOP mode.
pgAdmin 4 - Application Initialisation
======================================

Starting pgAdmin 4. Please navigate to http://0.0.0.0:5050 in your browser.
 * Serving Flask app 'pgadmin' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off

However, nothing persists. Not the master password and not the connection setup. Every run is like a clean install.

If I start over from scratch by erasing ./volumes/pg_admin and changing the service definition to remove the /servers.json from each side (ie as per my reference definition above), the container barfs with:

ERROR  : Failed to create the directory /pgadmin/config:
           [Errno 13] Permission denied: '/pgadmin/config'
HINT :   Create the directory /pgadmin/config, ensure it is writeable by
         'pgadmin', and try again, or, create a config_local.py file
         and override the SQLITE_PATH setting per
         https://www.pgadmin.org/docs/pgadmin4/6.19/config_py.html
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/pgadmin4/pgadmin/setup/data_directory.py", line 33, in create_app_data_directory
    is_directory_created = _create_directory_if_not_exists(
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pgadmin4/pgadmin/setup/data_directory.py", line 20, in _create_directory_if_not_exists
    os.mkdir(_path)
PermissionError: [Errno 13] Permission denied: '/pgadmin/config'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/pgadmin4/pgAdmin4.py", line 93, in <module>
    app = create_app()
          ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pgadmin4/pgadmin/__init__.py", line 265, in create_app
    create_app_data_directory(config)
  File "/usr/local/lib/python3.11/site-packages/pgadmin4/pgadmin/setup/data_directory.py", line 46, in create_app_data_directory
    sys.exit(1)
    ^^^
NameError: name 'sys' is not defined

If I terminate the container, then:

$ sudo chown $USER:$USER ~/IOTstack/volumes/pg_admin

and then bring up the container again, we're back to the behaviour when servers.json is in place but, this time, a sensible structure gets created in the persistent store, and the master password and connection details get saved:

$ tree -apug ~/IOTstack/volumes/pg_admin
/home/pi/IOTstack/volumes/pg_admin
├── [drwxr-xr-x pi       staff   ]  azurecredentialcache
├── [drwx------ pi       staff   ]  config
│   └── [-rw------- pi       staff   ]  pgadmin4.db
└── [drwxr-xr-x pi       staff   ]  storage

3 directories, 1 file

As an aside, notice that servers.json isn't mentioned so I've really got no idea where that came from. Everything seems to wind up in pgadmin4.db (an SQLite database).

This issue of containers having conniptions on clean install is a fairly common pattern and it's probably best if you tackle it (if you don't mind, that is).

The usual way to solve it is with some "self-repair" code in the entry-point script that forces correct ownership on each launch. Your repo has an entrypoint.sh but it doesn't appear to be making it into the image.

I've done a proof-of-concept by changing the service definition to be:

#   image: gpongelli/pgadmin4-docker-armv7
    build: ./.templates/pg_admin/.

And, then, in the .templates/pg_admin folder, a Dockerfile:

FROM gpongelli/pgadmin4-docker-armv7

# image starts as pgadmin - we need root for this
USER root

# add an entry-point script
ENV ENTRY_POINT="docker-entrypoint.sh"
COPY ${ENTRY_POINT} /${ENTRY_POINT}
RUN chmod 755 /${ENTRY_POINT}

# unset variables that are not needed
ENV ENTRY_POINT=

# away we go
ENTRYPOINT ["/docker-entrypoint.sh"]
CMD python /usr/local/lib/python3.11/site-packages/pgadmin4/pgAdmin4.py

# return to original user
#USER pgadmin

# EOF

with the contents of the docker-entrypoint.sh being:

#!/bin/ash
set -e

export HOME=/pgadmin

# can be overridden in the service definition
PUID=${PUID:-1000}
PGID=${PGID:-50}

# enforce ownership if running as root
user="$(id -u)"
if [ "$user" = '0' -a -d "$HOME" ]; then

   echo "[IOTstack] begin self-repair"

   chown -Rc "$PUID:$PGID" "$HOME"

   echo "[IOTstack] end self-repair"

else

   echo "[IOTstack] insufficient privileges to perform self-repair"

fi

exec "$@"

Because entrypoint.sh (mentioned before) is not making it into the current image, I surmised that it was not important so I didn't preserve its content. If that's a mistake, it's easy enough to merge the two and settle on the final file name.

Also, it's a fairly common convention to support PUID and PGID environment variables (so they can be set in the service definition), with appropriate defaults applied in the entry-point script. I wondered whether HOME needed similar treatment.

With me so far? So, with all that in place, the container builds and runs OK. It self-repairs properly and life is smooth.

If I uncomment the Dockerfile line so the user is put back to normal:

USER pgadmin

then the entry-point script has insufficient privileges to run…

… unless I change the service definition to include:

user: "0"

to force the container to run as root. Either way, the clear objective that you had in having the container run as UID=1000 is overridden.

Bottom line:

If the volumes map points to a file (servers.json) then that gets created as a folder. The container runs but nothing persists.
If ./volumes/pg_admin/servers.json is created in advance then the container runs but nothing persists.
If ./volumes/pg_admin/servers.json is created in advance and both the folder and the file are assigned to 1000:50 then the container runs but nothing persists.
If the volumes map points ./volumes/pg_admin, is initialised by docker-compose (with root ownership) the container barfs with the "Failed to create the directory" error.
If the volumes map points ./volumes/pg_admin and the structure is manually set to 1000:50 ownership then the container runs and everything persists.

Options 3 and 5 require the user to "just know" how to solve the problem when something goes wrong. That's unacceptable for IOTstack, which is why we have this idea of self-repair code.

Option 4 is unacceptable for obvious reasons.

Options 1 & 2 are possibilities but it seems to me the basic idea is the master password and connection details should persist.

The real issue is whether to just let the container run as root so an entry-point script has the privileges for self-repair, or come up with a mechanism by which the container downgrades its privileges once self-repair is complete. Mosquitto is an example of a container that starts as root, runs self-repair, then becomes user 1883.

If I wasn't having this discussion with you as the image maintainer, I would just go right ahead and use that Dockerfile plus entry-point script "as is" and not really worry too much about the fact that the container was running as root. There are plenty of other containers that run as root so one more hardly matters. But, seeing as you're here, you might see sense in adapting the image to deal with these problems.

The main pro/con of local Dockerfile in IOTstack is that "vanilla" images will get pulled by any docker-compose pull whereas images based on local Dockerfiles need the user to "just know" when to force a rebuild. All things considered, a vanilla image would always be my first choice.

Anyway, that's as far as I've gone. I'll wait to hear from you as to how best to proceed.

gpongelli commented 1 year ago

Hi Phill, that's a long post, hope to cover all the points.

Here's a service definition for reference:
  pg_admin:
    container_name: pg_admin
    image: gpongelli/pgadmin4-docker-armv7
    platform: linux/arm/v7
    restart: unless-stopped
    environment:
    - TZ=${TZ:-Etc/UTC}
    ports:
    - 5050:5050
    volumes:
    - ./volumes/pg_admin:/pgadmin
The first issue is consistent use of pg_admin throughout.

IOTstack conventions assume the service name, container name, and top level folder on the left hand side of the persistent store will be the same.

I've adopted pg_admin for the moment but I don't really care whether we wind up calling it that or pgadmin, or pgAdmin, or something else.

If you would follow that convention, you should call them "pgAdmin4", but it's something any user can change and does not impact the docker image 😉 am I right ?

Next issue (I won't call it a problem). There's no ARMv8 architecture. I realise you explicitly posted ARMv7 but I only have 64-bit Bullseye systems to test on so this occurs when I try to bring up the container:
The requested image's platform (linux/arm/v7) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
I'm unlikely to be the only IOTstack user running 64-bit Bullseye so…

I've silenced the error via the added:
platform: linux/arm/v7
but that means it's stuck on ARMv7. I'll leave it to you to decide whether that's acceptable or if you want to build an ARMv8 image at the same time.

I've built only armv7l architecture because it's the one I have on my raspberry. I just saw yesterday that newer raspberry have armv8 architecture (64bit). To avoid I forget this, please add an issue on my GitHub repo, so I can work on it sooner or later.

then, going back to previous discussion point, having and armv7 and another armv8 docker images with same pgAdmin version, which one should be tagged as "latest" ?

The next issue concerns the volumes clause. This is a bit of a can of worms so please bear with me.

As written on your DockerHub/GitHub page, you have:
- ./volumes/pg_admin/servers.json:/pgadmin/servers.json
Given a clean install (where ./volumes/pg_admin does not exist, if you "up" the container like that, the result is:
$ tree -apug ~/IOTstack/volumes/pg_admin
/home/pi/IOTstack/volumes/pg_admin
└── [drwxr-xr-x root     root    ]  servers.json

1 directory, 0 files
Notice that servers.json is a directory.

IOTStack from gcgarner makes the folder under volumes (if I remember correctly). anyway, servers.json is a server list file I have and I use it to import the db connections without making them each time; on previous pgAdmin docker images I've found the import capability does not work, so it's part of my "basic test" before doing a release. Anyway, that "volumes" row can be removed from docker-compose file, a clean installation does not have existing servers connection saved.

Now, strangely enough, pg_admin works properly in this situation. I can create a connection to the postgres container and all is smooth and sweatless.

The log reports:
$ docker logs pg_admin 
NOTE: Configuring authentication for DESKTOP mode.
pgAdmin 4 - Application Initialisation
======================================

Starting pgAdmin 4. Please navigate to http://0.0.0.0:5050 in your browser.
 * Serving Flask app 'pgadmin' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
However, nothing persists. Not the master password and not the connection setup. Every run is like a clean install.

Probably I've to check the VOLUMES declaration, adding what's missing. Another issue that can be opened on my repo 😉

ERROR  : Failed to create the directory /pgadmin/config:
           [Errno 13] Permission denied: '/pgadmin/config'
HINT :   Create the directory /pgadmin/config, ensure it is writeable by
         'pgadmin', and try again, or, create a config_local.py file
         and override the SQLITE_PATH setting per
         https://www.pgadmin.org/docs/pgadmin4/6.19/config_py.html
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/pgadmin4/pgadmin/setup/data_directory.py", line 33, in create_app_data_directory
    is_directory_created = _create_directory_if_not_exists(
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pgadmin4/pgadmin/setup/data_directory.py", line 20, in _create_directory_if_not_exists
    os.mkdir(_path)
PermissionError: [Errno 13] Permission denied: '/pgadmin/config'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/pgadmin4/pgAdmin4.py", line 93, in <module>
    app = create_app()
          ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pgadmin4/pgadmin/__init__.py", line 265, in create_app
    create_app_data_directory(config)
  File "/usr/local/lib/python3.11/site-packages/pgadmin4/pgadmin/setup/data_directory.py", line 46, in create_app_data_directory
    sys.exit(1)
    ^^^
NameError: name 'sys' is not defined

If I terminate the container, then:

$ sudo chown $USER:$USER ~/IOTstack/volumes/pg_admin

$ tree -apug ~/IOTstack/volumes/pg_admin
/home/pi/IOTstack/volumes/pg_admin
├── [drwxr-xr-x pi       staff   ]  azurecredentialcache
├── [drwx------ pi       staff   ]  config
│   └── [-rw------- pi       staff   ]  pgadmin4.db
└── [drwxr-xr-x pi       staff   ]  storage

3 directories, 1 file

This probably happens because the image expects a folder that does no more exist.

As an aside, notice that servers.json isn't mentioned so I've really got no idea where that came from. Everything seems to wind up in pgadmin4.db (an SQLite database).

This issue of containers having conniptions on clean install is a fairly common pattern and it's probably best if you tackle it (if you don't mind, that is).

The usual way to solve it is with some "self-repair" code in the entry-point script that forces correct ownership on each launch. Your repo has an entrypoint.sh but it doesn't appear to be making it into the image.

It came from the "original" project, that does not build for arm architecture.

I've done a proof-of-concept by changing the service definition to be:

#   image: gpongelli/pgadmin4-docker-armv7
    build: ./.templates/pg_admin/.

And, then, in the .templates/pg_admin folder, a Dockerfile:

FROM gpongelli/pgadmin4-docker-armv7

# image starts as pgadmin - we need root for this
USER root

# add an entry-point script
ENV ENTRY_POINT="docker-entrypoint.sh"
COPY ${ENTRY_POINT} /${ENTRY_POINT}
RUN chmod 755 /${ENTRY_POINT}

# unset variables that are not needed
ENV ENTRY_POINT=

# away we go
ENTRYPOINT ["/docker-entrypoint.sh"]
CMD python /usr/local/lib/python3.11/site-packages/pgadmin4/pgAdmin4.py

# return to original user
#USER pgadmin

# EOF

on linux, it's better to never use root user 😉

with the contents of the docker-entrypoint.sh being:
#!/bin/ash
set -e

export HOME=/pgadmin

# can be overridden in the service definition
PUID=${PUID:-1000}
PGID=${PGID:-50}

# enforce ownership if running as root
user="$(id -u)"
if [ "$user" = '0' -a -d "$HOME" ]; then

   echo "[IOTstack] begin self-repair"

   chown -Rc "$PUID:$PGID" "$HOME"

   echo "[IOTstack] end self-repair"

else

   echo "[IOTstack] insufficient privileges to perform self-repair"

fi

exec "$@"
Because entrypoint.sh (mentioned before) is not making it into the current image, I surmised that it was not important so I didn't preserve its content. If that's a mistake, it's easy enough to merge the two and settle on the final file name.

Also, it's a fairly common convention to support PUID and PGID environment variables (so they can be set in the service definition), with appropriate defaults applied in the entry-point script. I wondered whether HOME needed similar treatment.

With me so far? So, with all that in place, the container builds and runs OK. It self-repairs properly and life is smooth.

If I uncomment the Dockerfile line so the user is put back to normal:
USER pgadmin
then the entry-point script has insufficient privileges to run…

… unless I change the service definition to include:
user: "0"
to force the container to run as root. Either way, the clear objective that you had in having the container run as UID=1000 is overridden.

which is the user you're using to run docker-compose ? does it have the permission on the whole IOTStack folder ?

Bottom line:

If the volumes map points to a file (servers.json) then that gets created as a folder. The container runs but nothing persists.

If ./volumes/pg_admin/servers.json is created in advance then the container runs but nothing persists.

If ./volumes/pg_admin/servers.json is created in advance and both the folder and the file are assigned to 1000:50 then the container runs but nothing persists.

If the volumes map points ./volumes/pg_admin, is initialised by docker-compose (with root ownership) the container barfs with the "Failed to create the directory" error.

If the volumes map points ./volumes/pg_admin and the structure is manually set to 1000:50 ownership then the container runs and everything persists.

Options 3 and 5 require the user to "just know" how to solve the problem when something goes wrong. That's unacceptable for IOTstack, which is why we have this idea of self-repair code.

Option 4 is unacceptable for obvious reasons.

Options 1 & 2 are possibilities but it seems to me the basic idea is the master password and connection details should persist.

The real issue is whether to just let the container run as root so an entry-point script has the privileges for self-repair, or come up with a mechanism by which the container downgrades its privileges once self-repair is complete. Mosquitto is an example of a container that starts as root, runs self-repair, then becomes user 1883.

If I wasn't having this discussion with you as the image maintainer, I would just go right ahead and use that Dockerfile plus entry-point script "as is" and not really worry too much about the fact that the container was running as root. There are plenty of other containers that run as root so one more hardly matters. But, seeing as you're here, you might see sense in adapting the image to deal with these problems.

The main pro/con of local Dockerfile in IOTstack is that "vanilla" images will get pulled by any docker-compose pull whereas images based on local Dockerfiles need the user to "just know" when to force a rebuild. All things considered, a vanilla image would always be my first choice.

Anyway, that's as far as I've gone. I'll wait to hear from you as to how best to proceed.

I could investigate on mosquitto solution, doing some changes to avoid the "self-repair" script. The only big drawback on all of this is that an image takes at least 1hour to be made 😢

Paraphraser commented 1 year ago

If you would follow that convention, you should call them "pgAdmin4", but it's something any user can change and does not impact the docker image 😉 am I right ?

I'm happy with "pgAdmin4". I think it's better than having an underscore in the name. I also don't mind the "A" but there are some purists who prefer their Unix to be all lower case. You are correct when you say a user could change anything and it would not impact the image but what tends to happen is users accept the template, which is why I listen to hints, tips, moans and complaints on this topic and address them ahead of time.

Anyway, we'll go with "pgAdmin4".

whoops - hold the phone !

Using upper-case A gets:

Error response from daemon: no such image: iotstack-pgAdmin4: invalid reference format: repository name must be lowercase

Rewind. Adhering to docker-compose's finicky rules gets us to:

  pgadmin4:
    container_name: pgadmin4
    image: gpongelli/pgadmin4-docker-armv7
    platform: linux/arm/v7
    restart: unless-stopped
    environment:
    - TZ=${TZ:-Etc/UTC}
    ports:
    - 5050:5050
    volumes:
    - ./volumes/pgadmin4:/pgadmin

Should the internal directory be set up to use pgadmin4 as well? Again, just a cosmetic thing. I'm only really mentioning it because we're working in the area.

I've built only armv7l architecture because it's the one I have on my raspberry. I just saw yesterday that newer raspberry have armv8 architecture (64bit). To avoid I forget this, please add an issue on my GitHub repo, so I can work on it sooner or later.

It's now my morning (UTC+11) and some hours since all of the above. It looks to me like you've already started on this (DockerHub, GitHub) - do you still want an issue?

then, going back to previous discussion point, having and armv7 and another armv8 docker images with same pgAdmin version, which one should be tagged as "latest" ?

Err, real ["Dumb Look"] emoji needed here. Perhaps take a look at grafana. I have no idea how you get three architectures associated with a single tag but you see this pattern everywhere so it's obviously possible.

Then also take a look at nodered. You can add to the list of things I don't know the question of why that has no ARMv8 yet doesn't chuck up the "The requested image's platform does not match". Again, this pattern of "no explicit ARMv8 yet works without complaint on 64-bit Bullseye" is pretty common. Indeed, it really surprised me when it happened on yours. First time I'd ever seen it and it took me a while to figure out that the platform clause would silence the warning (never used it before).

IOTStack from gcgarner makes the folder under volumes (if I remember correctly). anyway, servers.json is a server list file I have and I use it to import the db connections without making them each time; on previous pgAdmin docker images I've found the import capability does not work, so it's part of my "basic test" before doing a release. Anyway, that "volumes" row can be removed from docker-compose file, a clean installation does not have existing servers connection saved.

I'll split this into two. Given any arbitrary ./volumes/X where X can be a single or multiple path components, docker-compose (not IOTstack) does the equivalent of:

$ cd ~/IOTstack
$ sudo mkdir -p ./volumes/X

The first container to be spun up creates volumes. Subsequent containers create their X. All owned by root.

The original gcgarner IOTstack had the notion of a directoryfix.sh script. If that was present in the template and a container was chosen for the first time (or you did a "pull full service from template"), then the menu would run the directoryfix.sh script. The "new" menu (SensorsIot IOTstack) has a similar notion in build.sh but it doesn't have "pull full service from template" so the only time build.sh runs is the first time you select a container.

The basic problem is there is no mechanism in docker-compose for saying "oh, here's a setup script that will prepare the ground for a container before it comes up - please run it for me each time just before starting the container". There have been several issue requests against docker-compose over the years asking for such functionality but each request has either been ignored or turned down. The clear expectation is that containers will take care of themselves.

I went into this in some detail in #331 and it might be useful to read it for some background.

It's still true that Node-RED, Influx, Grafana, Pi-hole (and many others) are well behaved - albeit for certain values of "well behaved". Nuke their persistent stores and they re-populate and fix everything. It's equally true that quite a few containers can have conniptions if their persistent stores are not as they expect them, particularly on first launch. Nobody ever complains about well-behaved containers. Non-well-behaved containers are a frequent source of complaint. It's just easier to forestall the complaints with local Dockerfiles than constantly have to explain the how-to of manual fixes.

On "servers.json", it seems to me that one candidate might be a linkage to the (assumed) local instance of postgres. Yes? If so, then we might also be able to figure out a mechanism for building that on-the-fly on first launch, passing the default database and credentials via environment variables. Not saying that's a must-do, more a thought bubble.

Probably I've to check the VOLUMES declaration, adding what's missing. Another issue that can be opened on my repo 😉

I have never seen a situation where that affects anything. Anyway, it looks OK to me in your current Dockerfile.

This probably happens because the image expects a folder that does no more exist.

We might be at cross purposes. I was just iterating through possible pre-conditions (no ./volumes/pg_admin, exists with root ownership, exists with ID=1000 ownership, what happened if servers.json was absent, present as a folder, present as a file). It's the kind of testing I do to figure out what some "self repair" code actually needs to do in an entry-point script.

To elaborate, the tests demonstrate that fixing ownership is sufficient because pg4admin will re-populate its persistent store if it is nuked.

Mosquitto, on the other hand, has different behaviour. It does not re-populate its persistent store so the self-repair code needs to both fix ownership and replace any files that have gone missing. If you don't do both, the container goes into a restart loop.

I hope that makes sense.

on linux, it's better to never use root user 😉

Agreed. In principle.

Containers are rarely perfect. This is my priority order:

Container launches as root, performs any necessary self-repair, then downgrades its own privileges to bare minimum.
Container launches as root, performs any necessary self-repair, then continues to run as root because it lacks the smarts to downgrade.
Container launches as non-root, can't deal with the default structures set up by docker-compose, can't perform self-repair, and either misbehaves (eg can't write expected data to persistent store) or goes into a restart loop.

Out of the box:

Mosquitto is a 3. Adding the local Dockerfile has turned it into a 1. But that's because the broker plays nice and downgrades its privileges.
Node-RED is a 3 but becomes a 2 via a user: "0" in the service definition.
Pi-hole is a 1.
Portainer, Grafana and InfluxDB are 2.

Bottom line, I'm never going to make the mistake of letting the "perfect" (the principle of not using root) be the enemy of the "good". At the end of the day, I want containers that work first time, every time. If that means a container runs as root, so be it.

which is the user you're using to run docker-compose ? does it have the permission on the whole IOTStack folder ?

No. On a clean install of IOTstack (or if you nuke the entire volumes folder) it's as I explained above. docker-compose does the equivalent of a sudo mkdir -p ./volumes/X so, by default, volumes will always be owned by root.

To be honest, I have no idea how docker-compose does its thing. Some of it will be because the current user is a member of the docker group. The app itself doesn't seem to have setUID on its directory entry. I believe it's written in go. It could be just as straightforward as making shell calls with sudo and relying on the current user being a sudoer. No idea. I just look at cause-and-effect:

non-root user
no volumes folder
docker-compose up -d
volumes owned by root.

Must somehow acquire the privileges to do that.

After volumes is created with root ownership, it does not actually matter if you change the ownership of volumes to be the current user. Nothing breaks. Some people have proposed adding a command to the menu to do that each time the menu runs. I don't see the point because, unless a container changes the ownership on the folder(s) under its control, those will be owned by root too.

We have enough trouble with inappropriate use of sudo as it is. I'd rather stick with the principle that:

Only volumes and backups/influxdb needs sudo and you really should be staying away from backups/influxdb anyway.
Nothing else inside ~/IOTstack should ever need sudo and it indicates a problem if that isn't true.

As it happens, backups/influxdb is an historical artifact. It would have been much better to put that into volumes but we're stuck with it.

I could investigate on mosquitto solution, doing some changes to avoid the "self-repair" script.

I might be wrong but that makes me think I might've misled you.

If you just pull the Mosquitto image straight from DockerHub and spin it up, it will almost certainly fail. In terms of IOTstack, Mosquitto is a non-well-behaved container.

We (IOTstack) apply a local Dockerfile to turn it into a well-behaved container. The Dockerfile adds a custom entry-point script which (a) fixes ownership and (b) copies any missing required files into place before the broker starts.

Even though there may be alternatives to an entry-point script (for doing "self-repair"), you still face the basic conundrum. If you build a container like pgadmin4 the way you are (so it runs as UID=1000) then, given the way docker-compose creates volumes structures with root ownership, the container will lack the privileges it needs - to do anything.

If you build a container so that it starts by running as root, then an entry-point script (or whatever other scheme you come up with) can perform necessary self-repair, and then you can downgrade privileges to UID=1000 when you launch the substantive process.

To make a local Dockerfile work when a container is built so it runs as non-root, you either need a user: "0" in the service definition to override the non-root or you add a USER root to the Dockerfile. Then the entry-point can run, and the container will run as root. Unless, perhaps, someone figures out how to replace the exec $@ at the end of the entry-point with a call that invokes the $@ (the contents of the "command") as a different user. I don't think you can do it with an exec itself but you might be able to combine exec and ash. I've never experimented with this.

Incidentally, while I think of it, your container doesn't have "bash" so any scripts with she-bangs leading to "bash" should probably be replaced with "ash". I just noticed that when I was looking at entrypoint.sh.

gpongelli commented 1 year ago

Hi Phil, I've already done some work to speed up docker image build with multiple arch.

Rewind. Adhering to docker-compose's finicky rules gets us to:
  pgadmin4:
    container_name: pgadmin4
    image: gpongelli/pgadmin4-docker-armv7
    platform: linux/arm/v7
    restart: unless-stopped
    environment:
    - TZ=${TZ:-Etc/UTC}
    ports:
    - 5050:5050
    volumes:
    - ./volumes/pgadmin4:/pgadmin
Should the internal directory be set up to use pgadmin4 as well? Again, just a cosmetic thing. I'm only really mentioning it because we're working in the area.

I've not changed the Dockerfile, I need some investigation on this topic.

It's now my morning (UTC+11) and some hours since all of the above. It looks to me like you've already started on this (DockerHub, GitHub) - do you still want an issue?

No, I've added the issues by myself 😉 For further reference, the changes I'm going to do will publish images to the new, now still empty, DockerHub repo, because "armv7" has no more sense if it'll be compiled for armv8 too 😉

Now, that some "build stage" things are resolved, I can look at the examples you've posted, to understand what's wrong.

On "servers.json", it seems to me that one candidate might be a linkage to the (assumed) local instance of postgres. Yes? If so, then we might also be able to figure out a mechanism for building that on-the-fly on first launch, passing the default database and credentials via environment variables. Not saying that's a must-do, more a thought bubble.

As first stage, I think it's better to avoid that on-the-fly mechanism. It's better to remove the row from docker-compose and let people to create pgadmin's connection as widely described elsewhere

gpongelli commented 1 year ago

Hi Phil, I've done some changes on dockerfile today and launched the build.

If pipeline ends correctly :

REMOVED
following nodered and others, now the first execution should work (the folder has root group, so root can change its content, but it's owned by pgadmin4 user)
the volume is now pgadmin4, the user is pgadmin4
docker images will be put under this repo

Docker compose file should be

  pgadmin4:
    container_name: pgadmin4
    image: gpongelli/pgadmin4-arm:....-armv8
    restart: unless-stopped
    ports:
    - 5050:5050
    volumes:
    - ./volumes/pgadmin4:/pgadmin4

if all the tests are ok, I'll also add a "latest" tag.

Let me know, thanks.

Paraphraser commented 1 year ago

Can I take it that this build also includes an entry-point.sh or equivalent which does the self-repair, or is that on the to-do list?

I'd also like to tap your knowledge of postgres, please.

The context is adding backup functionality to IOTstackBackup. At the moment, the scripts simply bypass postgres and it's high time I fixed that.

This service definition isn't the one in the current template but you can assume I'll submit a pull request:

postgres
  postgres:
    container_name: postgres
    image: postgres
    restart: unless-stopped
    environment:
    - TZ=${TZ:-Etc/UTC}
    - POSTGRES_USER=${IOTSTACK_POSTGRES_USER:-postuser}
    - POSTGRES_PASSWORD=${IOTSTACK_POSTGRES_INITIAL_PASSWORD:-IOtSt4ckpostgresDbPw}
    - POSTGRES_DB=${IOTSTACK_POSTGRES_DATABASE:-postdb}
    ports:
    - "5432:5432"
    volumes:
    - ./volumes/postgres/data:/var/lib/postgresql/data
    - ./volumes/postgres/db_backup:/backup

My intention is to use the following command structure to backup postgres while the container is running. Assume invocation via docker exec:

# pg_dumpall -U $POSTGRES_USER | gzip > /backup/yyyy-mm-dd_hhmm.«hostname».postgres-backup.gz

To be clear, the $POSTGRES_USER is evaluated inside the container. Plus the # prompt assumes running as root.

The resulting file in ./volumes/postgres/db_backup would be moved into ~/IOTstack/backups so it could be synchronised via whatever scheme the user chooses.

In the reverse direction, the assumption is the persistent storage area is erased, the container is launched (thereby initialising persistent storage), the backup file being restored is moved into the newly-created ./volumes/postgres/db_backup and then, via docker exec, the running container is instructed to restore using the following command structure:

# gunzip -c /backup/postgres_backup.gz | psql -U $POSTGRES_USER

A small amount of testing suggests that this works but it has been close on 15 years since I last used PostgreSQL in anger so I'd rather get a thumbs-up or pointers to an improved approach before adopt this.

On the general topic of backup and restore, I see no need for any special handling of the pgadmin4 persistent store. The pgadmin4.db is an SQLite database and that's copy-safe.

gpongelli commented 1 year ago

Entry point script should not be needed (do your test and let me know), because the group for volume folder is now set to root -> root user is able to modify it -> no need of entry point script (at least in theory).

Anyway it’s the same solution from nodered .

Il giorno 1 feb 2023, alle ore 13:24, Phill @.***> ha scritto:

Can I take it that this build also includes an entry-point.sh or equivalent which does the self-repair, or is that on the to-do list?

I'd also like to tap your knowledge of postgres, please.

The context is adding backup functionality to IOTstackBackup. At the moment, the scripts simply bypass postgres and it's high time I fixed that.

This service definition isn't the one in the current template but you can assume I'll submit a pull request:

postgres postgres: container_name: postgres image: postgres restart: unless-stopped environment:

TZ=${TZ:-Etc/UTC}

POSTGRES_USER=${IOTSTACK_POSTGRES_USER:-postuser}

POSTGRES_PASSWORD=${IOTSTACK_POSTGRES_INITIAL_PASSWORD:-IOtSt4ckpostgresDbPw}

POSTGRES_DB=${IOTSTACK_POSTGRES_DATABASE:-postdb} ports:

"5432:5432" volumes:

./volumes/postgres/data:/var/lib/postgresql/data

./volumes/postgres/db_backup:/backup My intention is to use the following command structure to backup postgres while the container is running. Assume invocation via docker exec:

pg_dumpall -U $POSTGRES_USER | gzip > /backup/yyyy-mm-dd_hhmm.«hostname».postgres-backup.gz

To be clear, the $POSTGRES_USER is evaluated inside the container. Plus the # prompt assumes running as root.

The resulting file in ./volumes/postgres/db_backup would be moved into ~/IOTstack/backups so it could be synchronised via whatever scheme the user chooses.

In the reverse direction, the assumption is the persistent storage area is erased, the container is launched (thereby initialising persistent storage), the backup file being restored is moved into the newly-created ./volumes/postgres/db_backup and then, via docker exec, the running container is instructed to restore using the following command structure:

gunzip -c /backup/postgres_backup.gz | psql -U $POSTGRES_USER

A small amount of testing suggests that this works but it has been close on 15 years since I last used PostgreSQL in anger so I'd rather get a thumbs-up or pointers to an improved approach before adopt this.

On the general topic of backup and restore, I see no need for any special handling of the pgadmin4 persistent store. The pgadmin4.db is an SQLite database and that's copy-safe.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.

gpongelli commented 1 year ago

after some debug, I've found that python's package docker api does NOT support multiple platform.

I've revert back to build multiple images -> your docker-compose.yml must refer to the correct tag.

Paraphraser commented 1 year ago

OK. As far as root goes, your container, your rules.

Although the menu lacks smarts for platform-specific image selections, we do have at least one precedent:

home_assistant:
  container_name: home_assistant
  image: ghcr.io/home-assistant/home-assistant:stable
  #image: ghcr.io/home-assistant/raspberrypi3-homeassistant:stable
  #image: ghcr.io/home-assistant/raspberrypi4-homeassistant:stable

So, one way forward is to mimic that:

  pgadmin4:
    container_name: pgadmin4
    image: gpongelli/pgadmin4-arm:armv8-latest
   #image: gpongelli/pgadmin4-arm:armv7-latest

That's assuming, based on no hard evidence whatsoever, that full 64-bit Bullseye is the most common implementation these days so it should be the default.

The disadvantage is it will barf on ARMv7. But, the "advantage" of that disadvantage, so to speak, is that it forces each ARMv7 user to choose the correct image and, come the day that person upgrades to ARMv8, the container will barf again and force them to put it back.

A slightly more user-friendly alternative is:

  pgadmin4:
    container_name: pgadmin4
    # to configure for full 64-bit ARMv8:
    # 1. change "armv7" to "armv8"
    # 2. remove of comment-out the "platform"
    image: gpongelli/pgadmin4-arm:armv7-latest
    platform: linux/arm/v7

This has the advantage of working on either architecture at the price of a small loss of efficiency on 64-bit systems, which can be had back for the one-time price of following the embedded instructions (or perhaps I'll put them in the doco).

I'm leaning towards the second. What do you think?

And I assume you've also noticed this depends on the idea of you adding "«arch»-latest" tags. Hint hint.

Now, out of the box, pulling pgadmin4-arm:6.19-py3.11-armv7 in the absence of a platform clause produces the "does not match" warning, as before.

Adding the platform clause silences the warning, as before.

Container down, persistent store erased, container up, produces:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/pgadmin4/pgAdmin4.py", line 93, in <module>
    app = create_app()
          ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pgadmin4/pgadmin/__init__.py", line 265, in create_app
    create_app_data_directory(config)
  File "/usr/local/lib/python3.11/site-packages/pgadmin4/pgadmin/setup/data_directory.py", line 33, in create_app_data_directory
    is_directory_created = _create_directory_if_not_exists(
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pgadmin4/pgadmin/setup/data_directory.py", line 20, in _create_directory_if_not_exists
    os.mkdir(_path)
FileNotFoundError: [Errno 2] No such file or directory: '/pgadmin/config'

Container down, persistent store erased, user: "0" clause added to service definition, container up, produces the same error as above.

I am not 100% sure but I think this is a change in behaviour, but I might be getting confused about the point where (in the previous iteration) I decided to use a local Dockerfile.

Changing ownership (either 1000:50 or 1000:1000) or permissions (777) on the persistent store does not help.

The actual error is different from the earlier traceback which was complaining about permissions. This one is "no such file or directory". I'm wondering whether the path in the last line should perhaps be pgadmin4 rather than pgadmin?

Bottom line: it won't run on my system.

This is getting to be a bit of a saga for the ages. I assure you I'm in it for the long haul and am not at all fussed by any of this. We learn by doing. I hope you aren't finding it too bothersome. We'll get there in the end.

gpongelli commented 1 year ago

Ok to have -latest for both tags, I see no alternatives.

New build is going, last time I’ve missed to change folder name on a file, that’s why if the create_directory error. try the new one and let me know.

about armv8 let me know if that image works on your rpi, I cannot run it.

Paraphraser commented 1 year ago

To answer your question about whether the armv8 image works on my Pi: a resounding YES!

Otherwise, it's a mixed bag of results. I will go step by step so you can tell me whether my findings meet your intention.

For clarity:

  pgadmin4:
    container_name: pgadmin4
    image: gpongelli/pgadmin4-arm:6.19-py3.11-armv7
    platform: linux/arm/v7
#   image: gpongelli/pgadmin4-arm:6.19-py3.11-armv8
    restart: unless-stopped
    environment:
    - TZ=${TZ:-Etc/UTC}
#   user: "0"
    ports:
    - "5050:5050"
    volumes:
    - ./volumes/pgadmin4:/pgadmin4

Starting position is:

persistent store erased
no gpongelli images on the local system (ie anything I do will pull from DockerHub)

Test 1

Upping the container (ie armv7) results in:

ERROR  : Failed to create the directory /pgadmin4/config:
           [Errno 13] Permission denied: '/pgadmin4/config'
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/pgadmin4/pgadmin/setup/data_directory.py", line 33, in create_app_data_directory
HINT :   Create the directory /pgadmin4/config, ensure it is writeable by
         'pgadmin4', and try again, or, create a config_local.py file
         and override the SQLITE_PATH setting per
         https://www.pgadmin.org/docs/pgadmin4/6.19/config_py.html
    is_directory_created = _create_directory_if_not_exists(
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pgadmin4/pgadmin/setup/data_directory.py", line 20, in _create_directory_if_not_exists
    os.mkdir(_path)
PermissionError: [Errno 13] Permission denied: '/pgadmin4/config'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/pgadmin4/pgAdmin4.py", line 93, in <module>
    app = create_app()
          ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pgadmin4/pgadmin/__init__.py", line 265, in create_app
    create_app_data_directory(config)
  File "/usr/local/lib/python3.11/site-packages/pgadmin4/pgadmin/setup/data_directory.py", line 46, in create_app_data_directory
    sys.exit(1)
    ^^^
NameError: name 'sys' is not defined
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/pgadmin4/pgadmin/setup/data_directory.py", line 33, in create_app_data_directory

$ ls -ld pgadmin4; sudo tree -apug pgadmin4/
drwxr-xr-x 2 root root 4096 Feb  3 09:50 pgadmin4
pgadmin4/

0 directories, 0 files

Test 2

Down. Erase persistent store. Activate user: "0". Up.

NOTE: Configuring authentication for DESKTOP mode.
pgAdmin 4 - Application Initialisation
======================================

Starting pgAdmin 4. Please navigate to http://0.0.0.0:5050 in your browser.
 * Serving Flask app 'pgadmin' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off

$ ls -ld pgadmin4; sudo tree -apug pgadmin4/
drwxr-xr-x 5 root root 4096 Feb  3 09:58 pgadmin4
pgadmin4/
├── [drwxr-xr-x root     root    ]  azurecredentialcache
├── [drwx------ root     root    ]  config
│   └── [-rw------- root     root    ]  pgadmin4.db
└── [drwxr-xr-x root     root    ]  storage

3 directories, 1 file

Test 3

Down. Erase persistent store. Deactivate user: "0". Deactivate armv7 image and platform statements. Activate armv8. Up (ie now as armv8).

Same error as Test 1. Expected behaviour.

Ditto structure of persistent store (only top-level folder created with root ownership). Expected behaviour.

Test 4

Down. Erase persistent store. Activate user: "0". Up.

Same success state as test 2. Expected behaviour.

Ditto structure of persistent store. Expected behaviour.

Test 5

Down. Erase persistent store. Deactivate user: "0". Prepare the persistent store ahead of time:

$ cd ~/IOTstack/volumes
$ sudo mkdir pgadmin4
$ sudo chown 1000:1000 pgadmin4

Up the container. This succeeds but the persistent store structure is a bit weird:

$ ls -ld pgadmin4; sudo tree -apug pgadmin4/
drwxr-xr-x 5 pi pi 4096 Feb  3 10:33 pgadmin4
pgadmin4/
├── [drwxr-xr-x pi       65533   ]  azurecredentialcache
├── [drwx------ pi       65533   ]  config
│   └── [-rw------- pi       65533   ]  pgadmin4.db
└── [drwxr-xr-x pi       65533   ]  storage

Inside the container:

$ docker exec pgadmin4 ls -al /pgadmin4
total 24
drwxr-xr-x    5 pgadmin4 1000          4096 Feb  3 10:36 .
drwxr-xr-x    1 root     root          4096 Feb  3 10:32 ..
drwxr-xr-x    2 pgadmin4 nogroup       4096 Feb  3 10:33 azurecredentialcache
drwx------    2 pgadmin4 nogroup       4096 Feb  3 10:33 config
drwxr-xr-x    2 pgadmin4 nogroup       4096 Feb  3 10:33 storage

My guess is that that's the result of something weird going on with the Dockerfile lines:

RUN addgroup -g 50 -S pgadmin \
 && adduser -D -S -h /pgadmin -s /sbin/nologin -u 1000 -G pgadmin pgadmin \

and which would appear to be confirmed by:

$ docker exec pgadmin4 ash -c 'grep pgadmin /etc/group ; grep pgadmin /etc/passwd'
nogroup:x:65533:pgadmin4
pgadmin4:x:1000:65533:Linux User,,,:/pgadmin4:/sbin/nologin

If I'm reading that right, "nogroup, GID=65533, has pgadmin4 as a member" and "the owner ID and group ID for pgadmin4 are 1000 and 65533, respectively. So the relationships are correct but why it got hooked to nogroup is…

Test 6

Same as Test 5, save for:

$ sudo chown 1000:50 pgadmin4

Identical results to Test 5 so it's not as simple as "if you just used the same GID as in the Dockerfile addgroup command."

Discussion

So, given that everything works when user: "0" is active, we can go with that. Node-RED, Grafana and Prometheus service definitions all need this so it's not without precedent.

I'm not entirely sure which repo/Dockerfile you're building from. I can't see any pushes in the last 24 hours that would match-up with what you've just put on DockerHub. Anyway, hardly matters.

Using pgadmin4-docker-arm as my example, it contains:

USER pgadmin:pgadmin

and I assume that, right now, there's a version where that says pgadmin4

My understanding is that the user: "0" in the service definition just overrides the USER statement in the Dockerfile. The same when passing -u 0 on a docker run.

As well as for IOTstack (where we "control" the service definition to some extent), I'm also thinking about your images in a general-use sense. To me, the tests above show the only way they'll work on a docker run or in any other compose environment is to override any non-root USER statement. That seems kinda pointless.

My suggestion is:

Lose the USER statement.
Probably lose the addgroup and adduser commands earlier in the Dockerfile too. It'll certainly save having to figure out just how/why those are broken.

gpongelli commented 1 year ago

Hi Phil, Some speedy notes:

armv7 and armv8 are built from same dockerfile, so you can test only one of them. I asked you to test v8 platform to know if the used architecture during build is right for newer raspberry. Thanks.
recently images come from the changes on this PR, on volume_permission branch, so look at the template-dockerfile there (and not to the Dockerfile)

Paraphraser commented 1 year ago

OK, anchoring a screen-shot here for convenience:

Line 49 explains why group 50 disappeared (not that that's a recommendation to put it back).

And, as I said before, I reckon you should just remove line 81.

gpongelli commented 1 year ago

Hi Phil, I've done some big changes:

adding an entrypoint script that should fix missing folder or wrong permission or missing group
removing the USER pgadmin4 clause into dockerfile, so root starts the container
adding gosu tool to switch from root to pgadmin user, after the above check, to start pgadmin tool

New docker images are being built, please do your usual tests, thanks.

Paraphraser commented 1 year ago

I'd been kinda hoping (as I'm sure you were too) that I'd be able to say "it works". Sorry, still getting errors.

Starting position:

pgadmin4 container not running
all gpongelli images removed
persistent store erased

service definition:

  pgadmin4:
    container_name: pgadmin4
#   image: gpongelli/pgadmin4-arm:6.19-py3.11-armv7
#   platform: linux/arm/v7
    image: gpongelli/pgadmin4-arm:6.19-py3.11-armv8
    restart: unless-stopped
    environment:
    - TZ=${TZ:-Etc/UTC}
#   user: "0"
    ports:
    - "5050:5050"
    volumes:
    - ./volumes/pgadmin4:/pgadmin4

Test 1

Up the container. ARMv8 image comes down, then:

Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/pgadmin4/entrypoint.sh": stat /pgadmin4/entrypoint.sh: no such file or directory: unknown

Test 2

Although I don't really expect a change in behaviour, alter the service definition to implement ARMv7. Erase persistent store just for consistency. Up. ARMv7 image comes down then goes splat with the same error.

Discussion

Studying template-raspberry.Dockerfile, there's no good reason to try to shove the entrypoint script into the /pgadmin4 directory. So that's the first point. The second point is that I do it like this:

# add an entry-point script
ENV ENTRY_POINT="docker-entrypoint.sh"
COPY ${ENTRY_POINT} /${ENTRY_POINT}
RUN chmod 755 /${ENTRY_POINT}
ENV ENTRY_POINT=

It's just a personal preference to do it with a variable so I can come back and change the name, once, without having to think too hard. So, yes, it could be a different file name and hard-coded in two lines:

COPY entrypoint.sh /entrypoint.sh
RUN chmod 755 /entrypoint.sh

and, then, if I wanted to stick it somewhere other than /:

COPY entrypoint.sh /pgadmin4/entrypoint.sh
RUN chmod 755 /pgadmin4/entrypoint.sh

or a+x rather than the octal mode I actually want … whatever.

But, notice how the second argument to the COPY explicitly names the file again. That's because, as I understand it, the Dockerfile COPY is not the same as a Unix cp. There's no implied "keep the original filename in the target directory". I'm not sure what actually happens when you do it the way you are at the moment but it's clear from the error message that there is no /pgadmin4/entrypoint.sh.

My own decision-making about /entrypoint.sh vs /pgadmin4/entrypoint.sh would, in similar circumstances, go like this:

What's the purpose of /pgadmin4? Answer, it's the home directory of the pgadmin4 user which is the container analogue of whatever UID=1000 maps to outside the container.
Is any purpose served by making entrypoint.sh available in the home directory of pgadmin4? Is there any situation in which pgadmin4 will actually need to run that script? The answer is "no need to run" and "no purpose is served".
What will happen if the user does actually try to run that script once the container is up and running? On the face of it, a second instance of pgadmin4 along with whatever arguments the user happens to pass (or none). Either way, there would likely be a mess.

In reality, that script is something that only root should run when starting the container. That's why I'd stick it in /.

The other thing I've noticed is that the COPY does preserve whatever permission bits are set in the external version of the file/folders at the moment the COPY runs. So the RUN chmod is really just a fail-safe.

I don't think it matters which way you set permission bits. I just thought I'd mention it.

Paraphraser commented 1 year ago

I've been studying these lines:

Let me start by saying I'm not an expert in this stuff (Dockerfiles, entry-points) in the sense of having any deep knowledge or understanding. I mostly just rattle around banging into problems and solve them as best I can.

But…

Although I think I can see what you're trying to achieve, I don't understand what problem you're actually trying to solve.

I also don't see how this can ever work.

In fact, I have a nagging feeling that you might be using "outside the container" thinking to solve an "inside the container" problem that might not even exist inside the container in the first place.

Let me explain.

This pgadmin4 container is based on Alpine Linux, right?

It doesn't actually make any difference to what I'm about to say whether the container is based on Alpine, Debian, Ubuntu, whatever. I just want to stay focused on the goal.

So, let me consider two other containers that I know are based on Alpine:

Mosquitto
Node-RED

Proof:

$ docker exec nodered grep "PRETTY_NAME" /etc/os-release
PRETTY_NAME="Alpine Linux v3.16"
$ docker exec mosquitto grep "PRETTY_NAME" /etc/os-release
PRETTY_NAME="Alpine Linux v3.16"

Now, let's run the getent group docker command inside each one:

$ docker exec nodered getent group docker
$ docker exec mosquitto getent group docker
$

Silence means there is no such group inside the container, so that means CUR_DOCKER_GID is always going to be a null string. Agreed?

Next, let's try the ls -ng /var/run/docker.sock command:

$ docker exec nodered ls -ng /var/run/docker.sock
srw-rw----    1 995              0 Jan 31 15:37 /var/run/docker.sock
$ docker exec mosquitto ls -ng /var/run/docker.sock
ls: /var/run/docker.sock: No such file or directory

Why the difference? It's because the Node-RED service definition for IOTstack maps that path:

$ grep -B 3 docker.sock ~/IOTstack/.templates/nodered/service.yml 
  volumes:
    - ./volumes/nodered/data:/data
    - ./volumes/nodered/ssh:/root/.ssh
    - /var/run/docker.sock:/var/run/docker.sock

Mosquitto doesn't map docker.sock so that explains the difference.

Thus, unless we map docker.sock, the pgadmin4 container is going to behave like Mosquitto, not Node-RED. That, in turn, means:

SOCK_DOCKER_GID will always be a null string, so
! -z "$SOCK_DOCKER_GID" will always fail, so

as an aside, you can replace ! -z with -n (is string non-empty).
the body of the conditional expression (line 26) will never execute.

Also, the side-effect of the ls -ng failing is a message on stderr which is going to show up in docker logs and look like a problem.

If I try to put the purpose of this code into words, it looks like you're trying to make sure that the pgadmin4 user is a member of the docker group, with lines 19…27 trying to create the conditions where that can happen. But line 29 will never execute anyway.

I think the key question to ask is, "what is the purpose of the docker group?" The answer is that it gives any user belonging to that group the privilege to execute docker commands.

Does a user (like pi), running on the host system, wanting to execute docker run and docker-compose commands, need those privileges? Yes!

Does a proxy user (like pgadmin4) running inside a container need to execute docker run and docker-compose commands? No.

Do containers typically even have docker and docker-compose installed? Let's ask all the containers I have running on my test Pi:

$ for C in postgres wireguard nodered zigbee2mqtt mosquitto influxdb pihole grafana ; do docker exec $C which docker docker-compose ; done
$

Silence. So, no.

And, even if a container did have those commands installed, would they have any effect? I don't really know the answer. My guess is it would depend on whether docker.sock was mapped, or if the goal was to try to run a whole docker environment inside a container. No idea if that's even possible.

With the possible exception of a container like Portainer, I reckon it sounds like a bad idea for any container to be trying to futz with the docker environment outside the container. That means the container doesn't need the docker and docker-compose commands. That means the "user" inside the container doesn't need to be a member of the docker group. That means the docker group serves no purpose and doesn't need to exist inside the container. Logical?

Seeing as I'm on a roll (and I hope you're not too upset with me for picking holes like this), I'll move onto lines 32 through 35.

HOME is /pgadmin4, right? That's already guaranteed to exist in the image by virtue of the Dockerfile. That means line 33 will always fail so line 34 will never execute.

Also, you don't really need to worry about creating the config and storage sub-directories. Previous tests in this issue have already shown that ./volumes/pgadmin4 got populated correctly long before the entrypoint.sh was involved.

Bottom line: I think you should remove lines 17 through 35 from the Dockerfile.

I reckon Dockerfile lines 15 and onwards should look like this:

user="$(id -u)"
if [ "$user" = '0' -a -d "$HOME" ]; then

    echo "begin self-repair"
    chown -Rc "$PUID:$PGID" "$HOME"
    echo "end self-repair"

    # Add call to gosu to drop from root user to pgadmin4 user
    exec gosu pgadmin4 "$@"

else

    echo "insufficient privileges to perform self-repair or $HOME does not exist"

fi

exec $@

Sure, lose the "IOTstack" in the echo statements but the remainder of the echo statements perform a valuable diagnostic function. I know I said before that /pgadmin4 is guaranteed to exist because it's set up in the Dockerfile but, in this case, it's defensive coding. Well, that's my story and I'm sticking to it.

Anyway, over to you.

gpongelli commented 1 year ago

I'd been kinda hoping (as I'm sure you were too) that I'd be able to say "it works". Sorry, still getting errors.

Starting position:
pgadmin4 container not running

all gpongelli images removed

persistent store erased
service definition:
 pgadmin4:
   container_name: pgadmin4
#   image: gpongelli/pgadmin4-arm:6.19-py3.11-armv7
#   platform: linux/arm/v7
   image: gpongelli/pgadmin4-arm:6.19-py3.11-armv8
   restart: unless-stopped
   environment:
   - TZ=${TZ:-Etc/UTC}
#   user: "0"
   ports:
   - "5050:5050"
   volumes:
   - ./volumes/pgadmin4:/pgadmin4
Test 1

Up the container. ARMv8 image comes down, then:
Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/pgadmin4/entrypoint.sh": stat /pgadmin4/entrypoint.sh: no such file or directory: unknown
Test 2

Although I don't really expect a change in behaviour, alter the service definition to implement ARMv7. Erase persistent store just for consistency. Up. ARMv7 image comes down then goes splat with the same error.

Discussion

Studying template-raspberry.Dockerfile, there's no good reason to try to shove the entrypoint script into the /pgadmin4 directory. So that's the first point. The second point is that I do it like this:
# add an entry-point script
ENV ENTRY_POINT="docker-entrypoint.sh"
COPY ${ENTRY_POINT} /${ENTRY_POINT}
RUN chmod 755 /${ENTRY_POINT}
ENV ENTRY_POINT=
It's just a personal preference to do it with a variable so I can come back and change the name, once, without having to think too hard. So, yes, it could be a different file name and hard-coded in two lines:
COPY entrypoint.sh /entrypoint.sh
RUN chmod 755 /entrypoint.sh
and, then, if I wanted to stick it somewhere other than /:
COPY entrypoint.sh /pgadmin4/entrypoint.sh
RUN chmod 755 /pgadmin4/entrypoint.sh
or a+x rather than the octal mode I actually want … whatever.

But, notice how the second argument to the COPY explicitly names the file again. That's because, as I understand it, the Dockerfile COPY is not the same as a Unix cp. There's no implied "keep the original filename in the target directory". I'm not sure what actually happens when you do it the way you are at the moment but it's clear from the error message that there is no /pgadmin4/entrypoint.sh.

My own decision-making about /entrypoint.sh vs /pgadmin4/entrypoint.sh would, in similar circumstances, go like this:

What's the purpose of /pgadmin4? Answer, it's the home directory of the pgadmin4 user which is the container analogue of whatever UID=1000 maps to outside the container.

Is any purpose served by making entrypoint.sh available in the home directory of pgadmin4? Is there any situation in which pgadmin4 will actually need to run that script? The answer is "no need to run" and "no purpose is served".

What will happen if the user does actually try to run that script once the container is up and running? On the face of it, a second instance of pgadmin4 along with whatever arguments the user happens to pass (or none). Either way, there would likely be a mess.

In reality, that script is something that only root should run when starting the container. That's why I'd stick it in /.

The other thing I've noticed is that the COPY does preserve whatever permission bits are set in the external version of the file/folders at the moment the COPY runs. So the RUN chmod is really just a fail-safe.

I don't think it matters which way you set permission bits. I just thought I'd mention it.

I had read the documentation, and there’s no need to name the destination file. Anyway, I’ll try your ENV solution that seems reasonable.

gpongelli commented 1 year ago

I've been studying these lines:

Let me start by saying I'm not an expert in this stuff (Dockerfiles, entry-points) in the sense of having any deep knowledge or understanding. I mostly just rattle around banging into problems and solve them as best I can.

But…

Although I think I can see what you're trying to achieve, I don't understand what problem you're actually trying to solve.

I also don't see how this can ever work.

In fact, I have a nagging feeling that you might be using "outside the container" thinking to solve an "inside the container" problem that might not even exist inside the container in the first place.

Let me explain.

This pgadmin4 container is based on Alpine Linux, right?

It doesn't actually make any difference to what I'm about to say whether the container is based on Alpine, Debian, Ubuntu, whatever. I just want to stay focused on the goal.

So, let me consider two other containers that I know are based on Alpine:

Mosquitto

Node-RED

Proof:
$ docker exec nodered grep "PRETTY_NAME" /etc/os-release
PRETTY_NAME="Alpine Linux v3.16"
$ docker exec mosquitto grep "PRETTY_NAME" /etc/os-release
PRETTY_NAME="Alpine Linux v3.16"
Now, let's run the getent group docker command inside each one:
$ docker exec nodered getent group docker
$ docker exec mosquitto getent group docker
$ 
Silence means there is no such group inside the container, so that means CUR_DOCKER_GID is always going to be a null string. Agreed?

Next, let's try the ls -ng /var/run/docker.sock command:
$ docker exec nodered ls -ng /var/run/docker.sock
srw-rw----    1 995              0 Jan 31 15:37 /var/run/docker.sock
$ docker exec mosquitto ls -ng /var/run/docker.sock
ls: /var/run/docker.sock: No such file or directory
Why the difference? It's because the Node-RED service definition for IOTstack maps that path:
$ grep -B 3 docker.sock ~/IOTstack/.templates/nodered/service.yml 
  volumes:
    - ./volumes/nodered/data:/data
    - ./volumes/nodered/ssh:/root/.ssh
    - /var/run/docker.sock:/var/run/docker.sock
Mosquitto doesn't map docker.sock so that explains the difference.

Thus, unless we map docker.sock, the pgadmin4 container is going to behave like Mosquitto, not Node-RED. That, in turn, means:

SOCK_DOCKER_GID will always be a null string, so

! -z "$SOCK_DOCKER_GID" will always fail, so

as an aside, you can replace ! -z with -n (is string non-empty).

the body of the conditional expression (line 26) will never execute.

Also, the side-effect of the ls -ng failing is a message on stderr which is going to show up in docker logs and look like a problem.

If I try to put the purpose of this code into words, it looks like you're trying to make sure that the pgadmin4 user is a member of the docker group, with lines 19…27 trying to create the conditions where that can happen. But line 29 will never execute anyway.

I think the key question to ask is, "what is the purpose of the docker group?" The answer is that it gives any user belonging to that group the privilege to execute docker commands.

Does a user (like pi), running on the host system, wanting to execute docker run and docker-compose commands, need those privileges? Yes!

Does a proxy user (like pgadmin4) running inside a container need to execute docker run and docker-compose commands? No.

Do containers typically even have docker and docker-compose installed? Let's ask all the containers I have running on my test Pi:
$ for C in postgres wireguard nodered zigbee2mqtt mosquitto influxdb pihole grafana ; do docker exec $C which docker docker-compose ; done
$
Silence. So, no.

And, even if a container did have those commands installed, would they have any effect? I don't really know the answer. My guess is it would depend on whether docker.sock was mapped, or if the goal was to try to run a whole docker environment inside a container. No idea if that's even possible.

With the possible exception of a container like Portainer, I reckon it sounds like a bad idea for any container to be trying to futz with the docker environment outside the container. That means the container doesn't need the docker and docker-compose commands. That means the "user" inside the container doesn't need to be a member of the docker group. That means the docker group serves no purpose and doesn't need to exist inside the container. Logical?

Seeing as I'm on a roll (and I hope you're not too upset with me for picking holes like this), I'll move onto lines 32 through 35.

HOME is /pgadmin4, right? That's already guaranteed to exist in the image by virtue of the Dockerfile. That means line 33 will always fail so line 34 will never execute.

Also, you don't really need to worry about creating the config and storage sub-directories. Previous tests in this issue have already shown that ./volumes/pgadmin4 got populated correctly long before the entrypoint.sh was involved.

Bottom line: I think you should remove lines 17 through 35 from the Dockerfile.

I reckon Dockerfile lines 15 and onwards should look like this:
user="$(id -u)"
if [ "$user" = '0' -a -d "$HOME" ]; then

    echo "begin self-repair"
    chown -Rc "$PUID:$PGID" "$HOME"
    echo "end self-repair"

    # Add call to gosu to drop from root user to pgadmin4 user
    exec gosu pgadmin4 "$@"

else

  echo "insufficient privileges to perform self-repair or $HOME does not exist"

fi

exec $@
Sure, lose the "IOTstack" in the echo statements but the remainder of the echo statements perform a valuable diagnostic function. I know I said before that /pgadmin4 is guaranteed to exist because it's set up in the Dockerfile but, in this case, it's defensive coding. Well, that's my story and I'm sticking to it.

Anyway, over to you.

I hoped that docker group exists inside the container, I could remove some of that checks.

i’ll keep the folder one, as per your previous test deleting the folder run in issue.

I’ll tell you when new build is in progress.

gpongelli commented 1 year ago

Changes made, new build started

Paraphraser commented 1 year ago

I had read the documentation, and there’s no need to name the destination file. Anyway, I’ll try your ENV solution that seems reasonable.

I will admit to two things:

I did not re-check the documentation; and
I did not test the theory in a Dockerfile.

I just looked at the error message (file not found), then I looked at your emerging Dockerfile and couldn't see any reason why that would not have worked, then I looked at one of my own Dockerfiles with where I had named the target explicitly and thought "ahah!"

In normal use, if I'd doing something like:

cp fred.txt directory

I may write the second argument as directory/ but, the only time I would ever explicitly put a filename on the right hand side is if I also wanted to change the name of the file during the copy.

So that's why it suggested itself as a possibility. I would almost certainly have written my first Dockerfile on the assumption that the filename was implied then, when it didn't work, would have tried making it explicit, found that that did work, and moved on.

All this "experience" where "be explicit" became baked-into my Dockerfiles is old so I could accept that the behaviour of COPY had changed.

Yet … it didn't work in your Dockerfile. 🤷‍♂️

Paraphraser commented 1 year ago

I hoped that docker group exists inside the container, I could remove some of that checks.

See, that's the bit I find confusing. I see no reason why the docker group would exist by default. The container doesn't inherit the host's /etc/passwd or /etc/group or other things like that.

I see no purpose in forcing the container to create the docker group. Outside container-space (ie on the Pi), that group is a by-product of running the "convenience script" to install docker support. You then add the user (eg "pi") to the docker group so the user gets the requisite permissions.

But inside container-space?

I'm not saying this is never applicable. I don't claim deep knowledge. And I'm not saying any of this just to be argumentative. I honestly can't think of a use-case.

Paraphraser commented 1 year ago

Still bad news, I'm afraid. We have a restart loop (the errors below occur over and over).

$ docker logs pgadmin4 
+ export 'HOME=/pgadmin4'
+ PUID=1000
+ PGID=50
+ id -u
+ user=0
+ '[' 0 '=' 0 ]
+ '[' '!' -d /pgadmin4/config ]
+ mkdir -p /pgadmin4/config
+ '[' '!' -d /pgadmin4/storage ]
+ mkdir -p /pgadmin4/storage
+ chown -Rc 1000:50 /pgadmin4
changed ownership of '/pgadmin4/config' to 1000:50
changed ownership of '/pgadmin4/storage' to 1000:50
changed ownership of '/pgadmin4' to 1000:50
+ exec gosu pgadmin4 /bin/sh -c 'python /usr/local/lib/python3.11/site-packages/pgadmin4/pgAdmin4.py'
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/pgadmin4/pgAdmin4.py", line 93, in <module>
    app = create_app()
          ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pgadmin4/pgadmin/__init__.py", line 270, in create_app
    fh = EnhancedRotatingFileHandler(config.LOG_FILE,
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pgadmin4/pgadmin/utils/enhanced_log_rotation.py", line 37, in __init__
    handlers.TimedRotatingFileHandler.__init__(self, filename=filename,
  File "/usr/local/lib/python3.11/logging/handlers.py", line 214, in __init__
    BaseRotatingHandler.__init__(self, filename, 'a', encoding=encoding,
  File "/usr/local/lib/python3.11/logging/handlers.py", line 58, in __init__
    logging.FileHandler.__init__(self, filename, mode=mode,
  File "/usr/local/lib/python3.11/logging/__init__.py", line 1181, in __init__
    StreamHandler.__init__(self, self._open())
                                 ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/logging/__init__.py", line 1213, in _open
    return open_func(self.baseFilename, self.mode,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/dev/stdout'

OK. What I think is going on here is because of the change of owner in the exec gosu pgadmin4 "$@".

And, as an aside, now that I've thought about it some more, it might not be appropriate to have quotes around the $@. We can come back to that.

So this is something I've never researched to understand it in detail but I think the container starts and fires off PID=1 which is the entry-point script. It's running as root and it has /dev/stdout and /dev/stderr attached.

If we're talking about a container like Mosquitto where the exec $@ starts the broker, it's still root and PID=1 at that point and stays that way until the broker itself downgrades its privileges. It's possible that the mechanism involves an intermediate "launcher" (for want of a better word) that continues to run as root and pipes stdout and stderr on behalf of the broker process which runs as non-root.

I don't know how gosu works but, taking it logically, gosu will start as root with PID=1 and be attached to stdout and stderr, but then gosu probably re-execs as the pgadmin4 user to run $@ and, at that point, it lacks permissions to do anything with stdout and stderr.

Emphasise: guesswork and speculation.

Paraphraser commented 1 year ago

Perhaps take a look at:

zerotier-docker entrypoint-router.sh

This is something I worked on a few months back. I don't claim credit for all of that script - I borrowed stuff from all over the place and it took a long time to get right.

I suggest starting at line 101:

define a file (the "tail-pipe") for inter-process communication and initialise it with a cat (note touch did not work)
start tail running in the background waiting for writes on the "tail-pipe", and keep a note of its PID so it can be cleaned-up later
use nohup to spin "$@" (with quotes so I guess you can ignore that thought I had earlier), with both the subordinate process' stdout and stderr redirected to the tail-pipe. Make a note of that PID too.
define a termination-handler function to clean up if the subordinate process exits or aborts, then set up to trap the expected errors (and I did check which ones actually happen: INT TERM and HUP).
suspend on the subordinate process.

What normally happens is the trap fires and the termination handler runs - the script finally exits (which is where the docker-compose clause to "restart unless stopped" will kick in to start things up again). Lines 143 onwards are defensive programming.

Now, this is only needed because of all the iptables rules you'll see in the earlier part of the script. If the zerotier daemon fails, those rules need to get zapped before the container restarts, otherwise they just double-up (weird but true). I make that point because, but for iptables, the zerotier daemon could be launched via exec $@. The zerotier daemon runs as root and doesn't downgrade its privileges.

But, it is possible that you could use this approach in conjunction with either nohup or gosu or both.

It's equally possible that this is all gross overkill and that this problem has been solved before, far more elegantly.

The last point I'll make: if it were me, I think I'd give up trying to run as ID=1000 and just live with running as root. As I've said before, you'd be in very good company:

    PID EUSER    RUSER    SUSER    FUSER    COMMAND
 314604 root     root     root     root     influxd
3316282 root     root     root     root     zerotier-one
3316319 root     root     root     root     portainer
3316461 root     root     root     root     grafana-server
3316506 1883     1883     1883     1883     mosquitto
3316677 root     root     root     root     node-red
3321115 999      999      999      999      pihole-FTL

gpongelli commented 1 year ago

Hi Phil, I've done local build and fixes, build is in progress.

Test made:

run with "user: 0", starts without error and the user launching pgadmin is pgadmin, and not root (so gosu works)
- in entrypoint.sh it passes into if branch, doing chmod and gosu call
run without "user: 0", starts without error and the user launching pgadmin is pgadmin
removing gosu tool, the above two test led to have root as launching user in first case
when container is recreated, the entire folder is cleared, loosing the sqlite file and any json exported for server connection

My docker-compose used for those tests:

services:
  pg_admin:
    container_name: pg_admin
#    user: "0"
    build: .
    restart: unless-stopped
    environment:
    - TZ=${TZ:-Etc/UTC}
    ports:
    - 5050:5050
    volumes:
    - ./volumes/pg_admin:/var/lib/pgadmin

EDIT: I've done some changes because, reading here, the folder that lives after container creations is /var/lib/pgadmin, and not /pgadmin4 . I've done some changes and exported server configuration remains there on mapped folder.

do your tests as usual, but I think I've reached the best result.

Paraphraser commented 1 year ago

Success!

Caveat:

I have only tried the ARMv8 image; and
without the user: '0' clause.

But, there are still some oddities that I'll go through.

What gets created in the persistent store?

$ tree -apug IOTstack/volumes/pgadmin4/
IOTstack/volumes/pgadmin4/
├── [drwxr-xr-x pi       staff   ]  azurecredentialcache
├── [drwxr-xr-x pi       staff   ]  config
│   └── [-rw------- pi       staff   ]  pgadmin4.db
└── [drwxr-xr-x pi       staff   ]  storage

3 directories, 1 file

Looks good. Anything in the logs?

$ docker logs pgadmin4 
+ export 'HOME=/var/lib/pgadmin'
+ PUID=1000
+ PGID=50
+ chown -Rc 1000:50 /var/lib/pgadmin
+ chmod o+w /dev/stdout
+ id -u
+ user=1000
+ '[' 1000 '=' 0 ]
+ exec python /usr/local/lib/python3.11/site-packages/pgadmin4/pgAdmin4.py
Starting pgAdmin 4. Please navigate to http://0.0.0.0:5050 in your browser.
 * Serving Flask app 'pgadmin' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off

Mostly OK but what's that chown -Rc 1000:50 /var/lib/pgadmin? That doesn't make sense in the context of:

    volumes:
    - ./volumes/pgadmin4:/pgadmin4

Let's explore inside the container:

$ docker exec -it pgadmin4 ash

What's the default working directory?

~ $ pwd
/var/lib/pgadmin
~ $ cd
~ $ pwd
/var/lib/pgadmin

So that's the default and also the home directory. This implies the Dockerfile creates the pgadmin4 user with this home directory. We're sort of back where we started with a mixture of pgadmin and pgadmin4.

I can't see any symlinks that explain this so…

Anything actually in this directory?

~ $ ls -al
total 12
drwxrwsr-x    2 pgadmin4 pgadmin4      4096 Feb  5 14:52 .
drwxr-xr-x    1 root     root          4096 Feb  5 05:26 ..
-rw-------    1 pgadmin4 pgadmin4        71 Feb  5 15:10 .ash_history

Not much. But what about the internal path we first thought of?

~ $ cd /pgadmin4/
/pgadmin4 $ ls -al
total 20
drwxr-xr-x    5 pgadmin4 pgadmin4      4096 Feb  4 20:28 .
drwxr-xr-x    1 root     root          4096 Feb  5 15:06 ..
drwxr-xr-x    2 pgadmin4 pgadmin4      4096 Feb  4 20:28 azurecredentialcache
drwxr-xr-x    2 pgadmin4 pgadmin4      4096 Feb  5 15:08 config
drwxr-xr-x    2 pgadmin4 pgadmin4      4096 Feb  4 20:28 storage

So that's there and clearly maps to the persistent store, as expected.

Let's look at the entrypoint script.

/pgadmin4 $ cat /entrypoint.sh 
#!/bin/ash
set -e -x

export HOME=/var/lib/pgadmin

# ref https://docs.docker.com/engine/reference/builder/#entrypoint
#trap "echo TRAPed signal" HUP INT QUIT TERM

# can be overridden in the service definition, 1000 and 50 comes from Dockerfile
PUID=${PUID:-1000}
PGID=${PGID:-50}

chown -Rc "$PUID:$PGID" "$HOME"

# beign able to write to /dev/stdout not only by root user
# https://github.com/moby/moby/issues/31243
chmod o+w /dev/stdout

# enforce ownership if running as root
user="$(id -u)"
if [ "$user" = '0' ]; then

    # Add call to gosu to drop from root user to pgadmin4 user
    exec gosu pgadmin4 "$@"
fi

exec $@
/pgadmin4 $ exit
$

So, that's clearly doing the recursive chown of /var/lib/pgadmin but (a) that has stuff all in it and (b) isn't the persistent store and (c) any changes won't persist anyway.

I haven't experimented with seeing if "something else" is enforcing chown on each launch (eg if I change ownership on pgadmin4.db, will it get fixed on next launch?). I thought I'd wait until what I'm thinking are "inconsistencies" turn out to be deliberate and necessary.

I have confirmed that anything I stick into pgadmin4.db does persist across container recreations so I'm not seeing your last dot point.

gpongelli commented 1 year ago

Hi Phil, I’ve already put the link to official documentation where:

/var/lib/pgadmin

This is the working directory in which pgAdmin stores session data, user files, configuration files, and it’s configuration database. Mapping this directory onto the host machine gives you an easy way to maintain configuration between invocations of the container.

it’s a folder that persist through container creation, I’ve tested it, and its location is not under my choices, so I cannot change it. is now clear the reason why I’ve moved to that folder?

have you tried the docker-compose file on my last comment? There is no more reference to /pgadmin4 folder because, on my tests, it get recreated each time .

now the only extra step I could do is add all the files/folders explained into official documentation as volumes to be mounted, but nothing more because it’s working.

Paraphraser commented 1 year ago

No. I missed that. I did not change the volumes statement.

Paraphraser commented 1 year ago

Hmmm. As soon as I do that the container goes into a restart loop

D65C5F60-0FCC-4948-8850-416DC5DAD644

Paraphraser commented 1 year ago

Well, now I have no idea why it "worked" before. I give up.

864F7D67-EFD7-480C-8E04-0B793B2A0F22

gpongelli commented 1 year ago

Hi Phil, New image available, tested on my rpi. Do your check, thanks.

Paraphraser commented 1 year ago

Success!

F044032F-C6FB-44F2-ADB4-1037699AAEEB

First attempt still had yesterday's internal path but, luckily, the command echo was still on in the entry point script so I could see that straight away.

As the old expression goes: LGTM

So, next step is the "latest" tag?

Paraphraser commented 1 year ago

Self repair works too:

8AE08B57-09F9-449E-A416-152C0E7A2513

gpongelli commented 1 year ago

So, next step is the "latest" tag?

From my side, yes, from yours create a PR to close this issue 😉

gpongelli commented 1 year ago

added the tags:

gpongelli/pgadmin4-arm:6.19-py3.11-armv7-latest
gpongelli/pgadmin4-arm:6.19-py3.11-armv8-latest

Paraphraser commented 1 year ago

Yup. Tomorrow.

Paraphraser commented 1 year ago

Those tags are too specific. They need to be more general:

gpongelli/pgadmin4-arm:latest-armv7 -> gpongelli/pgadmin4-arm:6.19-py3.11-armv7
gpongelli/pgadmin4-arm:latest-armv8 -> gpongelli/pgadmin4-arm:6.19-py3.11-armv8

Then, when you release a new version, say, 6.20, you have

gpongelli/pgadmin4-arm:6.19-py3.11-armv7
gpongelli/pgadmin4-arm:6.19-py3.11-armv8
gpongelli/pgadmin4-arm:latest-armv7 -> gpongelli/pgadmin4-arm:6.20-py3.11-armv7
gpongelli/pgadmin4-arm:latest-armv8 -> gpongelli/pgadmin4-arm:6.20-py3.11-armv8

Thus, latest always points to the most recent version but you maintain a back-history for anyone needing to still pull a specific version.

Make sense?

gpongelli commented 1 year ago

gpongelli/pgadmin4-arm:latest-armv7 -> gpongelli/pgadmin4-arm:6.19-py3.11-armv7

gpongelli/pgadmin4-arm:latest-armv8 -> gpongelli/pgadmin4-arm:6.19-py3.11-armv8

Done

Paraphraser commented 1 year ago

Perfect!

I think it might be true to say that neither of us expected such a long and winding road when you opened this issue. 🤓

Paraphraser commented 1 year ago

PRs #661 (master branch) #662 (old-menu branch) #663 (experimental branch) submitted.

Paraphraser commented 1 year ago

To give it a whirl:

$ PR=661
$ cd ~/IOTstack
$ git switch master
$ git pull origin master
$ git fetch origin pull/$PR/head:PR${PR}
$ git switch PR${PR}

Documentation in:

~/IOTstack/docs/Containers/PgAdmin4.md

Please let me know if you spot any issues or areas for improvement.

To revert after testing:

$ PR=661
$ cd ~/IOTstack
$ git switch master
$ git branch -D PR${PR}