Unable to run docker container as unprivelleged (with "user:" option)

skellycode commented 1 month ago

Describe the bug

When attempting to run windmill.dev with user: in the docker compose call, the following backtrace is produced:

Attaching to windmill
windmill  | 2024-08-03T20:40:16.646786Z  INFO src/main.rs:125: jemalloc enabled
windmill  | 2024-08-03T20:40:16.646804Z  INFO src/main.rs:196: Binary is in 'standalone' mode
windmill  | 2024-08-03T20:40:16.646810Z  INFO src/main.rs:267: Connecting to database...
windmill  | 2024-08-03T20:40:16.663825Z  INFO src/main.rs:269: Database connected
windmill  | 2024-08-03T20:40:16.664596Z  INFO src/main.rs:273: PostgreSQL version: PostgreSQL 16.3 (Debian 16.3-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit (windmill require PG >= 14)
windmill  | 2024-08-03T20:40:16.666040Z  INFO windmill-api/src/db.rs:77: Acquiring global PG lock for potential migration with pid: Some(190383)
windmill  | 2024-08-03T20:40:16.666572Z  INFO windmill-api/src/db.rs:97: Acquired global PG lock
windmill  | 2024-08-03T20:40:16.667785Z  INFO windmill-api/src/db.rs:112: Releasing PG lock
windmill  | 2024-08-03T20:40:16.668029Z  INFO windmill-api/src/db.rs:120: Released PG lock
windmill  | 2024-08-03T20:40:16.668052Z  INFO src/main.rs:311:
windmill  | ##############################
windmill  | Windmill Community Edition v1.371.4-1-g1a4732505
windmill  | ##############################
windmill  | 2024-08-03T20:40:16.668070Z  INFO src/main.rs:711: config: MODE: standalone, BASE_URL: example.com, GO_PATH: /usr/local/go/bin/go, PATH: /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin, HOME: /
windmill  | 2024-08-03T20:40:16.670576Z  INFO windmill-common/src/worker.rs:497: Loading config from WORKER_GROUP: default
windmill  | 2024-08-03T20:40:16.671093Z  INFO src/monitor.rs:950: Reloading worker config...
windmill  | 2024-08-03T20:40:16.672391Z  INFO windmill-common/src/worker.rs:166: Loaded setting custom_tags, common: ["chromium"], per-workspace: {}
windmill  | 2024-08-03T20:40:16.673079Z  WARN windmill-api/src/oauth2_ee.rs:180: oauth.json not found, no OAuth clients loaded
windmill  | 2024-08-03T20:40:16.674247Z  WARN windmill-common/src/server.rs:77: SMTP not configured
windmill  | 2024-08-03T20:40:16.674253Z  INFO src/monitor.rs:907: Reloading server config...
windmill  | 2024-08-03T20:40:16.674435Z  INFO src/monitor.rs:731: Loaded setting retention_period_secs from db config: Number(2592000)
windmill  | 2024-08-03T20:40:16.674793Z  INFO src/monitor.rs:705: Loaded saml_metadata setting to None
windmill  | 2024-08-03T20:40:16.674954Z  INFO src/monitor.rs:705: Loaded scim_token setting to None
windmill  | 2024-08-03T20:40:16.675086Z  INFO src/monitor.rs:705: Loaded pip_extra_index_url setting to None
windmill  | 2024-08-03T20:40:16.675249Z  INFO src/monitor.rs:705: Loaded pip_index_url setting to None
windmill  | 2024-08-03T20:40:16.675443Z  INFO src/monitor.rs:705: Loaded npm_config_registry setting to None
windmill  | 2024-08-03T20:40:16.675636Z  INFO src/monitor.rs:705: Loaded bunfig_install_scopes setting to None
windmill  | 2024-08-03T20:40:16.704028Z  INFO windmill-api/src/embeddings.rs:222: Loading embedding model...
windmill  | 2024-08-03T20:40:16.704263Z  INFO src/main.rs:704: Successfully connected to pg listen
windmill  | 2024-08-03T20:40:16.734714Z  INFO windmill-api/src/lib.rs:365: server started on port=8000 and addr=0.0.0.0 instance=Ocbuu
windmill  | 2024-08-03T20:40:17.032245Z ERROR windmill-api/src/embeddings.rs:173: Failed to get config.json from hugging face: I/O error Permission denied (os error 13)
windmill  | 2024-08-03T20:40:17.032686Z ERROR windmill-api/src/embeddings.rs:613: Failed to initialize model instance: could not get config.json
windmill  | thread 'main' panicked at src/main.rs:790:14:
windmill  | could not create initial worker dir: Os { code: 13, kind: PermissionDenied, message: "Permission denied" }
windmill  | stack backtrace:
windmill  |    0:     0x56309c11ca55 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h1e1a1972118942ad
windmill  |    1:     0x56309c14d2ab - core::fmt::write::hc090a2ffd6b28c4a
windmill  |    2:     0x56309c117f1f - std::io::Write::write_fmt::h8898bac6ff039a23
windmill  |    3:     0x56309c11c82e - std::sys_common::backtrace::print::ha96650907276675e
windmill  |    4:     0x56309c11dca9 - std::panicking::default_hook::{{closure}}::h215c2a0a8346e0e0
windmill  |    5:     0x56309c11d9ed - std::panicking::default_hook::h207342be97478370
windmill  |    6:     0x56309c11e143 - std::panicking::rust_panic_with_hook::hac8bdceee1e4fe2c
windmill  |    7:     0x56309c11e024 - std::panicking::begin_panic_handler::{{closure}}::h00d785e82757ce3c
windmill  |    8:     0x56309c11cf19 - std::sys_common::backtrace::__rust_end_short_backtrace::h1628d957bcd06996
windmill  |    9:     0x56309c11dd57 - rust_begin_unwind
windmill  |   10:     0x5630955e9fd3 - core::panicking::panic_fmt::hdc63834ffaaefae5
windmill  |   11:     0x5630955ea486 - core::result::unwrap_failed::h82b551e0ff2b2176
windmill  |   12:     0x5630961c825b - windmill::windmill_main::{{closure}}::{{closure}}::h12b89e7fff7a7230
windmill  |   13:     0x563095ea441b - <futures_util::future::poll_fn::PollFn<F> as core::future::future::Future>::poll::he50395f72f8e67a7
windmill  |   14:     0x56309565a657 - windmill::windmill_main::{{closure}}::hdb2e0a23aa4b29e5
windmill  |   15:     0x56309562f0e9 - tokio::runtime::park::CachedParkThread::block_on::hfc7208a502545d5e
windmill  |   16:     0x563095e3a94f - tokio::runtime::context::runtime::enter_runtime::hfe3d9095ee3dd6d6
windmill  |   17:     0x56309597c99c - tokio::runtime::runtime::Runtime::block_on::h394cec4798ae5607
windmill  |   18:     0x563095ee746d - windmill::main::hd78c1289a1d9d719
windmill  |   19:     0x563095c4c163 - std::sys_common::backtrace::__rust_begin_short_backtrace::h04bff02b80bf23fa
windmill  |   20:     0x563095c248fd - std::rt::lang_start::{{closure}}::hb1f6a40ebc40ef91
windmill  |   21:     0x56309c10e760 - std::rt::lang_start_internal::h3ed4fe7b2f419135
windmill  |   22:     0x563095ee7be5 - main
windmill  |   23:     0x7f4e3f27824a - <unknown>
windmill  |   24:     0x7f4e3f278305 - __libc_start_main
windmill  |   25:     0x5630955ead91 - _start
windmill  |   26:                0x0 - <unknown>
windmill exited with code 101

To reproduce

include user: 1000:1000 (substitute for relavant UID/GID) in docker-compose.yml

Expected behavior

The ability to run windmill as an unprivileged container

Screenshots

No response

Browser information

No response

Application version

No response

Additional Context

No response

TheDan64 commented 1 month ago

Just ran into this as well, I think the worker node is trying to create /tmp/windmill/cache_nomount, but it can't do that without root perms since /tmp/windmill is most likely owned by root. Since /tmp/windmill/cache and /tmp/windmill/logs come mounted from the host, they're likely owned by the non-root user and therefore have no issue read/writing to them

pdonorio commented 3 weeks ago

Hitting this problem too.

As a note I'm checking on existing pods, the folders are there with correct permissions:

I have no name!@windmill-workers-6b96cfdb69-bm5sw:/usr/src/app$ ls -l /tmp/windmill
total 0
drwxr-xr-x 11 1000 1000 115 Aug 27 06:48 cache
drwxr-xr-x  4 1000 1000  31 Aug 27 06:48 cache_nomount
drwxr-xr-x  2 1000 1000   6 Aug 27 06:48 logs
drwxr-xr-x  3 1000 1000  50 Aug 27 06:48 wk-bm5sw-nrvwz

I have no name!@windmill-workers-6b96cfdb69-bm5sw:/usr/src/app$ ls -ld /tmp/windmill
drwxr-xr-x 6 1000 1000 95 Aug 27 06:48 /tmp/windmill

This is true at least until version 1.380.0 (helm chart 2.0.253)

It must have changed how the main /tmp/windmill folder is created, possibly it's now done by root and then creating subfolders is denied for user 1000.

rubenfiszel commented 3 weeks ago

This is solved on latest and on next release 1.386.0. /tmp/windmill was created in Dockerfile and not chmod correctly

windmill-labs / windmill