Closed markusweigelt closed 1 year ago
@bertsky Here is the permission problem what i mentioned in our dev meeting last week. Do you have any idea?
I think unzipping the resource data.zip explains the rights of files and directories that are not root.
Yes.
So i can change them to root but i wonder why the files and directories run on the root user. Cause with our UID and GID implementations the files and directories should have the current user and group as owne.
Not necessarily: we do pass the current $(id -u)
and $(id -g)
values as $UID
and $GID
into Manager and Controller containers, but not into Kitodo.
I don't know how specifically Kitodo then proceeds to create all the files under APP_DATA
from _resources/data
, but likely it just runs as root and therefore does not care about the ownership of the latter.
docker inspect kitodo_production_ocrd-ocrd-manager-1
shows that env variables are not replaced with environment variables defined in Makefile.
Yes, that's odd. It works in my (also non-default) case:
"UID=1003",
"GID=100",
"UMASK=0002",
"CONTROLLER=sdvocr.slub-dresden.de:8022",
Perhaps the .EXPORT_ALL_VARIABLES
failed. Which version of make
are you running?
(Mine is 4.2.1)
Another possible explanation: you still have some prior definitions of these variables in your shell. Since in the Makefile we only define these as opt-in…
…any shell env value would override them.
I think unzipping the resource data.zip explains the rights of files and directories that are not root.
Yes.
So i can change them to root but i wonder why the files and directories run on the root user. Cause with our UID and GID implementations the files and directories should have the current user and group as owne.
Not necessarily: we do pass the current
$(id -u)
and$(id -g)
values as$UID
and$GID
into Manager and Controller containers, but not into Kitodo.
Oh yes that is the reason! If I use only Kitodo.Production without the OCR-D implementations i run also in the permission problems so therefore I will adapt the permissions analogous the OCR-D services to fix it for all usages.
I don't know how specifically Kitodo then proceeds to create all the files under
APP_DATA
from_resources/data
, but likely it just runs as root and therefore does not care about the ownership of the latter.
docker inspect kitodo_production_ocrd-ocrd-manager-1
shows that env variables are not replaced with environment variables defined in Makefile.Yes, that's odd. It works in my (also non-default) case:
"UID=1003", "GID=100", "UMASK=0002", "CONTROLLER=sdvocr.slub-dresden.de:8022",
Perhaps the
.EXPORT_ALL_VARIABLES
failed. Which version ofmake
are you running?(Mine is 4.2.1)
I use 4.2.1 too.
…any shell env value would override them.
For example, this could have happened if you ever sourced your .env
file.
Another possible explanation: you still have some prior definitions of these variables in your shell. Since in the Makefile we only define these as opt-in…
…any shell env value would override them.
Yes. In my these variables are not defined in shell before using make.
…any shell env value would override them.
For example, this could have happened if you ever sourced your
.env
file.
I don't include using source and i used new terminal too.
I don't include using source and i used new terminal too.
Could be in .bashrc though. So there is no $MANAGER_ENV_UID
?
Ah, you answered that one above already.
I think .EXPORT_ALL_VARIABLES
fails. So that will be the reason.
Cause in my local (atm) make "info" target, env values are defined by make correctly.
CONTROLLER_ENV_UID is 946828167
CONTROLLER_ENV_GID is 946800513
MANAGER_ENV_UID is 946828167
MANAGER_ENV_GID is 946800513
I think
.EXPORT_ALL_VARIABLES
fails. So that will be the reason.
It's hard to think how it actually could, though.
What does make help
show for these values?
It could still be some .bashrc thing. Your terminal's shell might be differently initialized than the bash which make is invoking...
I think
.EXPORT_ALL_VARIABLES
fails. So that will be the reason.It's hard to think how it actually could, though.
What does
make help
show for these values?It could still be some .bashrc thing. Your terminal's shell might be differently initialized than the bash which make is invoking...
Variables:
- CONTROLLER_ENV_UID user id to use on the OCR-D Controller (default: 946828167)
- CONTROLLER_ENV_GID group id to use on the OCR-D Controller (default: 946800513)
- MANAGER_ENV_UID user id to use on the OCR-D Manager (default: 946828167)
- MANAGER_ENV_GID group id to use on the OCR-D Manager (default: 946800513)
- MODE if 'managed', also starts/stops OCR-D Controller here (default: managed)
- in addition, all variables defined in .env can be overridden via shell or make
Same. I overlooked that make help
prints these variables too so the make info
is not necessary anymore. I have not made any adjustments to the bash.rc. Beside that when i printed all my env over printenv
an these envs are not defined.
Do you happen to have a $BASH_ENV
, or some Manager related defs in ~/.profile
or ~/.bash_profile
?
Also, for diagnostics, could you add the following recipe lines to help
(after the EOF
line, but before endef
:
echo UID=$$MANAGER_ENV_UID GID=$$MANAGER_ENV_GID
Do you happen to have a
$BASH_ENV
, or some Manager related defs in~/.profile
or~/.bash_profile
?No
~/.bash_profile
does not exist and~/.profile
has not such variable.Also, for diagnostics, could you add the following recipe lines to
help
(after theEOF
line, but beforeendef
:echo UID=$$MANAGER_ENV_UID GID=$$MANAGER_ENV_GID
Change results in following output:
...
Variables:
- CONTROLLER_ENV_UID user id to use on the OCR-D Controller (default: 946828167)
- CONTROLLER_ENV_GID group id to use on the OCR-D Controller (default: 946800513)
- MANAGER_ENV_UID user id to use on the OCR-D Manager (default: 946828167)
- MANAGER_ENV_GID group id to use on the OCR-D Manager (default: 946800513)
- MODE if 'managed', also starts/stops OCR-D Controller here (default: managed)
- in addition, all variables defined in .env can be overridden via shell or make
UID=946828167 GID=946800513
Ok, so the problem is not on make's side, the variables do get exported.
You are absolutely sure the Container was started afresh (not from a stopped earlier instance)?
(Try:
docker stop kitodo_production_ocrd_ocrd-manager_1
docker container rm kitodo_production_ocrd_ocrd-manager_1
make start
docker inspect kitodo_production_ocrd_ocrd-manager_1 | grep -e UID -e GID
)
Yes i'm absolutely sure that the Container is fresh. I checkout fpr test purposes your PR and run a new setup an for with explicite docker-compose build --no-cache.
In addition I have also executed the script. I only had to replace the container name because it is different for me.
After running application i got following result:
weigelt@LDV163:~/Work/OCR-D/kitodo_production_ocrd$ docker stop kitodo_production_ocrd-ocrd-manager-1
kitodo_production_ocrd-ocrd-manager-1
weigelt@LDV163:~/Work/OCR-D/kitodo_production_ocrd$ docker container rm kitodo_production_ocrd-ocrd-manager-1
kitodo_production_ocrd-ocrd-manager-1
weigelt@LDV163:~/Work/OCR-D/kitodo_production_ocrd$ make start
docker-compose up -d
[+] Running 8/8
⠿ Container kitodo_production_ocrd-dozzle-1 Running 0.0s
⠿ Container kitodo_production_ocrd-kitodo-mq-1 Running 0.0s
⠿ Container kitodo_production_ocrd-ocrd-controller-1 Running 0.0s
⠿ Container kitodo_production_ocrd-ocrd-monitor-1 Running 0.0s
⠿ Container kitodo_production_ocrd-kitodo-es-1 Running 0.0s
⠿ Container kitodo_production_ocrd-ocrd-manager-1 Started 0.4s
⠿ Container kitodo_production_ocrd-kitodo-db-1 Running 0.0s
⠿ Container kitodo_production_ocrd-kitodo-app-1 Running 0.0s
weigelt@LDV163:~/Work/OCR-D/kitodo_production_ocrd$ docker inspect kitodo_production_ocrd-ocrd-manager-1 | grep -e UID -e GID
"GID=1001",
"UID=1001",
weigelt@LDV163:~/Work/OCR-D/kitodo_production_ocrd$ make help
Targets:
- prepare: create directories, SSH key files and examples
- build: `docker-compose build` all images
- start: `docker-compose up` all containers (in detached mode)
- down: `docker-compose down` all containers (i.e. stop and remove)
- stop: `docker-compose stop` all containers (i.e. only stop)
- config: dump all the composed files
- status: list running containers
Variables:
- CONTROLLER_ENV_UID user id to use on the OCR-D Controller (default: 946828167)
- CONTROLLER_ENV_GID group id to use on the OCR-D Controller (default: 946800513)
- MANAGER_ENV_UID user id to use on the OCR-D Manager (default: 946828167)
- MANAGER_ENV_GID group id to use on the OCR-D Manager (default: 946800513)
- MODE if 'managed', also starts/stops OCR-D Controller here (default: managed)
- in addition, all variables defined in .env can be overridden via shell or make
UID=946828167 GID=946800513
Mysterious!
What versions do you use? (I have docker-compose 1.29 und Docker 20.10. But the environment
key has been there for a long time, and so has the general variable interpolation mechanism. Thus I doubt versions are the problem.)
Yes it is! My versions are Docker Compose Version 2.9.0 and Docker Version 20.10.12. So i will add the user to the kitodo-production-docker repo and check if same problem occurs there too.
Ooof. I thought the latest is 1.29 (since this is the most recent version on PyPI, and the most recent docker-compose
for Debian is also 1.29 for unstable and even 1.21 for stable/buster).
I can get docker-compose-plugin
though for my distro (via Docker apt repo), which brings docker compose
(mind the gap) 2.6. The two in parallel don't even conflict. Very strange.
Could you elaborate where you got your compose from (and which you are referring to, docker-compose --version
or docker compose version
)?
I installed both from the https://download.docker.com/linux/ubuntu
source in my linux mint environment.
Okay, regardless how and why on your system the official Docker repo offers a more up to date version of docker-compose
than mine: Let's analyse what specifically broke in v2 with our environment
variable passing.
I think this warning (fixed here) might be the clue. Could you please try updating to either 2.10 or downgrading to 2.8?
Upgraded docker-compose
to 2.10.x and now it works for me again.
docker inspect kitodo_production_ocrd_ocrd-manager_1 | grep -e UID -e GID
"UID=946828167",
"GID=946800513",
docker inspect kitodo_production_ocrd_ocrd-controller_1 | grep -e UID -e GID
"UID=946828167",
"GID=946800513",
I can get
docker-compose-plugin
though for my distro (via Docker apt repo), which bringsdocker compose
(mind the gap) 2.6. The two in parallel don't even conflict. Very strange.
Yes docker compose
and docker-compose
seemed to be different. I uninstalled docker-compose
and added a script docker-compose
under /usr/bin
witch include a compatibility call docker compose --compatibility "$@"
Seems to work with new Docker Compose version so i close this issue.
I am starting a project from scratch (make clean, make prepare, make build, make start) with normal user inside docker group.
Following error occurs in the OCR-D Manager when running the ocr script in Kitodo.Production.
today at 12:48:25Sep 1 10:48:24 ocrd-manager for_production.sh: insufficient permissions on /data volume
The data permissions and user group assignments are following in "kitodo" directory. For the OCR-D Manager only the "metadata" directory is mounted:
I think unzipping the resource data.zip explains the rights of files and directories that are not root. So i can change them to root but i wonder why the files and directories run on the root user. Cause with our UID and GID implementations the files and directories should have the current user and group as owne.
In my case current user id is "946828167" and group id "946800513" and for root user id is "0" group id "0".
When i go into the OCR-D Manager container UID and GID have the default value "1001" of
.env
file. So it seems env variables are not set correctly and probably root is then the fallback.When i print the environment variables of Makefile following output is generated:
docker inspect kitodo_production_ocrd-ocrd-manager-1
shows that env variables are not replaced with environment variables defined in Makefile.