Bug: Unable to create directory structure for SURREAL_OBJECT_CACHE

[dvv]

Describe the bug

Instance goes down with:

surrealdb-1  | thread 'surrealdb-worker' panicked at core/src/obs/mod.rs:58:22:
surrealdb-1  | Unable to create directory structure for SURREAL_OBJECT_CACHE: Os { code: 13, kind: PermissionDenied, message: "Permission denied" }

Steps to reproduce

  surrealdb:
    image: surrealdb/surrealdb:nightly
    user: 1001:1001
    environment:
      RUST_BACKTRACE: full
    volumes:
      - ./surrealdb:/data
    command: start --auth --user root --pass root --log trace file:/data/surrealdb.db

DEFINE TOKEN web_token ON SCOPE web TYPE JWKS VALUE "https://idm.example.com/oauth2/openid/surrealdb/public_key.jwk";

Attempt to

GET /key/user
Authorization: Bearer ...

Expected behaviour

Not go down, honor the auth token

SurrealDB version

1.3.0+20240216.485a224 for linux on x86_64

Contact Details

github

Is there an existing issue for this?

• [X] I have searched the existing issues

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[gguillemas]

Hi @dvv! Thank you for opening this issue, I was not aware of this behavior when using our Docker container.

The issue that you are facing is related to the object storage used for caching JWKS objects defaulting to the working directory from which SurrealDB is launched. In order to make this work with your settings, you will need to change the locations of the object storage to a directory where the container is allowed to write, such as the mounted volume.

You can modify your Docker Compose as follows:

  surrealdb:
    image: surrealdb/surrealdb:nightly
    user: 1001:1001
    environment:
      RUST_BACKTRACE: full
      SURREAL_OBJECT_CACHE: file:/data/cache
      SURREAL_OBJECT_STORE: file:/data/store
    volumes:
      - ./surrealdb:/data
    command: start --auth --user root --pass root --log trace file:/data/surrealdb.db

However, you issue has made us think about whether or not it is necessary to use the generic object storage (which was introduced to store and cache machine learning models) for JWKS objects. It seems that it may introduce more issues than it solves considering that it defaults to the filesystem (it works in memory for WASM builds), that its configuration applies to all SurrealDB features that rely on it and that JWKS objects usually have a small footprint.

I will be looking into replacing it for a simpler cache method for JWKS objects or adapting the generic object storage so that different features can use different storage backends, which will allow JWKS objects to always be cached in memory.

Thank you again for taking the time to report this! Let me know if the proposed workaround addresses your issue.

[dvanmali]

A directory mounting issue may occur after following the suggestion from @gguillemas

Failed to store JWKS object in local cache: 'Object Store error: Generic LocalFileSystem error: Unable to create dir /jwks: Permission denied (os error 13)'

If so, add the additional jwks volume.

    volumes:
      - ./surrealdb:/data
      - ./surrealdb/jwks:/jwks

[dvv]

@gguillemas @dvanmali it works now, thank you! I believe it should be fixed as I see no reason to expose all these internals ($SURREALOBJECT*, /jwks) to a vanilla user by default.

[dvanmali]

@gguillemas are there any implications to making SURREAL_OBJECT_CACHE and SURREAL_OBJECT_STORE default to those file locations?

[gguillemas]

@dvanmali Any dedicated locations where SurrealDB will be able to manipulate files freely should be fine. However, take into account that this location will be shared by all SurrealDB features that use the obs cache (e.g. machine learning for storing models, which can get quite large), which will store their objects on a different path than jwks. For jwks, keep in mind that ability of any process to write to that location means the ability to effectively add cryptographic secrets and keys that will be trusted by SurrealDB, which is another reason why I would like to move this cache to memory very soon, as I think there is no actual reason to have it in the filesystem. Read access should generally be fine when the JWKS objects are public.

As stated, I would like to move this specific cache to memory so that this issue is no longer a concern in the short term.

[gguillemas]

I just wanted to let you know that, in 1.4.0-beta.1 and onwards, JWKS objects are now cached in memory and should not cause errors related to filesystem permissions anymore, regardless of where SurrealDB runs.

[dvv]

@gguillemas Thank you a lot!

[dvv]

Well done! Thank you!