Could not decode tenant token, JSON error: UUID parsing failed

[reefland]

• [ x] I have read the FAQ.

Describe the bug I had 4 cocktails to add, 3 went fine, when I started the 4th, I was unable to select ingredients to add. No versions were upgraded.

I tried:

1. Logout / Clear Cache / Login
2. Select Bar
3. Try to add ingredient again.
4. Restarted all containers and went back to step 1.

Message in Meilisearch:

2024-12-03T13:21:32.223086Z WARN HTTP request{method=POST host="bar.[redacted]" route=/multi-search query_parameters= user_agent=Mozilla/5.0 (X11; Linux x86_64; rv:131.0) Gecko/20100101 Firefox/131.0 status_code=403 error=Could not decode tenant token, JSON error: UUID parsing failed: invalid length: expected length 32 for simple format, found 0 at line 1 column 15.}: tracing_actix_web::middleware: Error encountered while processing the incoming HTTP request: ResponseError { code: 403, message: "Could not decode tenant token, JSON error: UUID parsing failed: invalid length: expected length 32 for simple format, found 0 at line 1 column 15.", error_code: "invalid_api_key", error_type: "auth", error_link: "https://docs.meilisearch.com/errors#invalid_api_key" }

To Reproduce Try to add ingredient and the list is empty.

Versions: Salt Rim: v3.5.0 Bar Assistant: v4.2.0 Meilisearch: 1.11.3

Additional context, log outputs... Inside Meilisearch Container:

/meili_data $ echo $MEILISEARCH_KEY
Uk2EipeE9YI6cfrEmOQ3QaiYqfATdOI1Gl4

/meili_data $ echo $MEILI_MASTER_KEY
Uk2EipeE9YI6cfrEmOQ3QaiYqfATdOI1Gl4

Inside Bar Assistant Container:

~/cocktails$ echo $MEILISEARCH_KEY
Uk2EipeE9YI6cfrEmOQ3QaiYqfATdOI1Gl4

~/cocktails$ echo $MEILI_MASTER_KEY
Uk2EipeE9YI6cfrEmOQ3QaiYqfATdOI1Gl4

Inside Salt-Rim:

/# echo $MEILISEARCH_KEY
Uk2EipeE9YI6cfrEmOQ3QaiYqfATdOI1Gl4

/# echo $MEILI_MASTER_KEY
Uk2EipeE9YI6cfrEmOQ3QaiYqfATdOI1Gl4

[karlomikus]

I noticed the same.

After API starts the first time all ENV variables used by Laravel config get cached. So when the setup meilisearch commands sets new keys, the API still reads old ones from cache.

I'll have a fix today.

[karlomikus]

Also a bit off topic, but you don't have to pass meilisearch keys to salt-rim via env variables. It's not a big deal, but from security pov it's probably a good idea not to pass them.

Client gets search tokens via API either way.

[reefland]

Was just showing they all matched. Good to know not needed. Easily changeable.

[reefland]

I just installed the new version:

• Salt Rim: v3.5.0
• Bar Assistant: v4.2.1
• Meilisearch: 1.11.3

Same message:

2024-12-03T18:48:52.711542Z WARN HTTP request{method=POST host="bar.[redacted]" route=/multi-search query_parameters= user_agent=Mozilla/5.0 (X11; Linux x86_64; rv:133.0) Gecko/20100101 Firefox/133.0 status_code=403 error=Could not decode tenant token, JSON error: UUID parsing failed: invalid length: expected length 32 for simple format, found 0 at line 1 column 15.}: tracing_actix_web::middleware: Error encountered while processing the incoming HTTP request: ResponseError { code: 403, message: "Could not decode tenant token, JSON error: UUID parsing failed: invalid length: expected length 32 for simple format, found 0 at line 1 column 15.", error_code: "invalid_api_key", error_type: "auth", error_link: "https://docs.meilisearch.com/errors#invalid_api_key" }

Only Bar Assistant was restarted after the upgrade.

[karlomikus]

Can you check the following:

Get current bar search token: docker compose exec bar-assistant sqlite3 storage/bar-assistant/database.ba3.sqlite "select search_token from bars where id = 1" or whatever bar id you use.

You should get something like: eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJhcGlLZXlVaWQiOiI2OTMzNDQ2Yi1lZDQxLTRhY2ItYWU0Zi1hYjM3NWZlZmRkNTciLCJzZWFyY2hSdWxlcyI6eyIqIjp7ImZpbHRlciI6ImJhcl9pZCA9IDEifX19.pyt8TdFQhx2tVmVyqqdzF8dP4E1GCqezwm42-E5JX20

Check decoded payload of that token on something like jwt.io.

It should contain apiKeyUid that is the same as the one in ENV: docker compose exec bar-assistant cat .env | grep MEILISEARCH_API_KEY_UID. For example: MEILISEARCH_API_KEY_UID=6933446b-ed41-4acb-ae4f-ab375fefdd57

If it's the same and search is still not working then it's an issue with the salt-rim.

[reefland]

Is MEILISEARCH_API_KEY_UID new? I don't have that defined anywhere.

From the Bar Assistant container:

$ sqlite3 ./storage/bar-assistant/database.ba3.sqlite "select search_token from bars where id = 1"

eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJhcGlLZXlVaWQiOiIiLCJzZWFyY2hSdWxlcyI6eyIqIjp7ImZpbHRlciI6ImJhcl9pZCA9IDEifX19.mAcn1gj4dwnqG2AyTpzyBg-XSFZTvj2cR5vjPrBxM_4

Decoded Value:

{
  "apiKeyUid": "",
  "searchRules": {
    "*": {
      "filter": "bar_id = 1"
    }
  }
}

Scan ENV for matching variables:

$ env | grep MEIL
MEILI_MASTER_KEY=[redacted]
MEILISEARCH_KEY=[redacted]
MEILISEARCH_HOST=[redacted]

I guess, null matches null if not defined. Does that count as being the same ?

[karlomikus]

MEILISEARCH_API_KEY_UID and MEILISEARCH_API_KEY get set by setup script, they are set directly in .env file.

Restart should fix that missing key. Or you can do it manually running this commands:

docker compose exec bar-assistant php artisan config:clear
docker compose exec bar-assistant php artisan bar:setup-meilisearch
docker compose exec bar-assistant php artisan config:cache

This should fill in the .env file and update the bars with the new values.

[reefland]

Can you expand on how they are "set by the setup script"? I'm running under Kubernetes and there is no concept of a .env file. I take values normally defined in .env and set them in the YAML of the respective pods for the containers ENV variables instead.

For example, in Bar Assistant I set:

        env:
          TZ: "America/New York"
          APP_DEBUG: true
          LOG_LEVEL: info
          PUID: 33
          PGID: 33
          APP_URL: "https://bar.[redacted]/bar"
          MEILISEARCH_HOST: http://bar-assistant-search.bar-assistant.svc.cluster.local:7700
          REDIS_HOST: dragonfly.dragonfly.svc.cluster.local
          REDIS_PASSWORD: null
          REDIS_PORT: 6379
          CACHE_DRIVER: redis
          SESSION_DRIVER: redis
          ALLOW_REGISTRATION: false

Variables like MEILI_MASTER_KEY and MEILISEARCH_KEY are stored in an external password vault and pulled from that at deployment time.

The pod for the Salt-Rim container gets it own variables defined. If two containers needs the same variable & value I can duplicate them, or put them in the common secret (or use YAML anchors).

I've been using Bar Assistant / Salt-Rim since September this hasn't been an issue. Trying to understand what recently changed so I can make the appropriate changes.

I'm thinking I could:

1. Run the php artisan bar:setup-meilisearch to create a .env within the bar-assistant container.
2. I can then cat .env file to get the two variable values set and then define MEILISEARCH_API_KEY_UID and MEILISEARCH_API_KEY as secrets pulled from the vault.
3. When I restart the container the generated .env files go away as they are outside the persistent storage area.
4. Then the question is which containers need these values? Seems like Bar-Assistant and Salt-Rim, unclear about Melisearch needing it or not.

As long as these values don't change that often I think it could work (I might have to update after a Melisearch upgrade for example). Any thoughts on a better way?