Lost all apps after upgrade (docker compose)

[MrApe]

Checklist

• [x] I have searched budibase discussions and github issues to check if my issue already exists

Hosting

• Self
  • Method: docker-compose (single image)
  • Budibase Version: 2.2.
  • App Version:

Describe the bug Lost all applications after Upgrading from version 2.2.1 to 2.2.19. All Settings, Accounts and configurations are still there. Even the App numbers on the "Users" page are correct but the app itself seems to have gone.

To Reproduce

docker compose down
docker compose pull
docker compose up

Expected behavior The app should still be there.

Additional context All stateful data is mounted into the container, so should be kept between container restarts. Previous upgrades have been successful with the steps above. What's the exact architecture of app storage? Maybe it is possible to recover them?

[shogunpurple]

@MrApe can you post your docker-compose.yaml file please?

[MrApe]

@shogunpurple sure! Thanks for helping!

version: "3"
services:
  budibase:
    image: budibase/budibase:latest
    restart: unless-stopped
    environment:
      SELF_HOSTED: 1
      MINIO_ACCESS_KEY: ${MINIO_ACCESS_KEY}
      MINIO_SECRET_KEY: ${MINIO_SECRET_KEY}
      INTERNAL_API_KEY: ${INTERNAL_API_KEY}
      JWT_SECRET: ${JWT_SECRET}
      ENABLE_ANALYTICS: "false"
      REDIS_PASSWORD: ${REDIS_PASSWORD}
      BB_ADMIN_USER_EMAIL: ${BB_ADMIN_USER_EMAIL}
      BB_ADMIN_USER_PASSWORD: ${BB_ADMIN_USER_PASSWORD}
      COUCHDB_PASSWORD: ${COUCHDB_PASSWORD}
      COUCHDB_USER: ${COUCHDB_USER}
      PLUGINS_DIR: ${PLUGINS_DIR}
    volumes:
      - /mnt/data/aero-apps-data/plugins:/plugins
      - /mnt/data/aero-apps-data/budibase:/data
    labels:
        [...traefik labels removed...]

[shogunpurple]

Thanks @MrApe - this all looks good, the volumes are set up correctly as far as I can see. When you run budibase, could you open the browser and tell me the result of the /applications call in the browser?

Do any apps come back for you at all?

Also long shot, but is there any chance that the data in the bind mount on your machine may have been affected?

[MrApe]

Thats the result:

[{_id: "app_metadata", _rev: "67-df7ebe3168ee938da573248ec2e635ea",…}]
 0 : {_id: "app_metadata", _rev: "67-df7ebe3168ee938da573248ec2e635ea",…}

Full JSON response

```json [ { "_id": "app_metadata", "_rev": "67-df7ebe3168ee938da573248ec2e635ea", "appId": "app_bdc3b996c80c472d8352c2d9dfea4090", "type": "app", "version": "2.2.3", "componentLibraries": [ "@budibase/standard-components" ], "name": "Eliteteam", "url": "/eliteteam", "instance": { "_id": "app_bdc3b996c80c472d8352c2d9dfea4090" }, "tenantId": "default", "updatedAt": "2023-01-17T20:07:31.604Z", "createdAt": "2022-11-24T12:45:43.423Z", "status": "published", "navigation": { "navigation": "Top", "title": "Eliteteam", "navWidth": "Max", "navBackground": "var(--spectrum-global-color-static-white)", "links": [ { "text": "Eliteteams", "url": "/eliteteams", "id": "iO0VkA9uf4", "roleId": null }, { "text": "Personen", "url": "/personen", "id": "c3RKKaepLJ", "roleId": null }, { "text": "Maßnahmen", "url": "/events", "id": "AjQ4hwcoF", "roleId": null }, { "text": "Abrechnungen", "url": "/abrechnungen" } ], "logoUrl": "/global/settings/logoUrl", "navTextColor": "var(--spectrum-global-color-gray-900)" }, "theme": "spectrum--lightest", "customTheme": { "buttonBorderRadius": "0", "primaryColor": "#d8222a", "primaryColorHover": "var(--spectrum-global-color-gray-900)" }, "updatedBy": "us_868204925fe545009c8cb0b8d335095d", "usedPlugins": [ { "_id": "plg_budibase-pdf", "name": "budibase-pdf", "version": "1.0.0", "jsUrl": "budibase-pdf/plugin.min.js", "hash": "4d518ae209737fa8ed3ca03651e48bdb5fc336ab" } ], "revertableVersion": "2.1.42" } ] ```

It reports reports one app, what is correct. But it does not show up and the app template library is shown.

[shogunpurple]

@MrApe the good news is that it seems like your data is still there - and you haven't lost your apps.

What happens if you go to the app URL directly?

<your-host>/builder/app/app_dev_bdc3b996c80c472d8352c2d9dfea4090

Does it work?

[MrApe]

OK, this was partially successful: The builder URL did not work. It just gave errors:

However, the production URL of the app worked. 🎉 The app is still there an fully functioning. It seems like the builder just doesn't know about it and lost reference for some reason.

The downside: I cannot change it anymore.

[shogunpurple]

Thanks @MrApe - at least the production app is up and running - although it looks like the dev app is not there and I'm not sure how that may have occurred, nor can i recreate with a similar setup. I think you can fix this by replicatiing the production app to the dev app ID.

Navigate to the directory with your .env file, then run the following commands (one by one): couchuser=$(cat .env | grep COUCH_DB_USER | sed 's/.*=//g')

couchpass=$(cat .env | grep COUCH_DB_PASSWORD | sed 's/.*=//g')

curl -X POST http://$couchuser:$couchpass@127.0.0.1:5984/_replicate -d '{"source":"app_bdc3b996c80c472d8352c2d9dfea4090", "target":"app_dev_bdc3b996c80c472d8352c2d9dfea4090"}' -H "Content-Type: application/json"

Can you then see your app in the app list, and access it?

[MrApe]

Thanks @shogunpurple for your help. I finally got it to work.

However, the solution was not so straight forward.

tldr; Couchdb didn't accept the .env-user credentials and thus replication failed. After supplying a new one, replication worked. Then changing the 'appId' in the 'app_metadata' document and deleting the 'deployments' document made the app appear again.

Details

I'm leaving this here, if someone ever has a similar problem.

The first try with the commands above did not work. Since I'm using the all in one image, couch is running in the container, so I first execed into it. The env vars are set there, so I used them directly.

# exec into container
docker exec -it aero-apps-budibase-1 bash

# check if couchdb is running
root@3e5536edabcc:/# curl http://127.0.0.1:5984
{"couchdb":"Welcome","version":"3.2.1","git_sha":"244d428af","uuid":"458081948727bf47f568173178dab77b","features":["search","access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}

# check the env variables (actual values removed)
root@3e5536edabcc:/# echo ${COUCHDB_USER}
[...]
root@3e5536edabcc:/# echo ${COUCHDB_PASSWORD}
[...]

root@3e5536edabcc:/# curl -X POST http://${COUCHDB_USER}:${COUCHDB_PASSWORD}@127.0.0.1:5984/_replicate -d '{"source":"app_bdc3b996c80c472d8352c2d9dfea4090", "target":"app_dev_bdc3b996c80c472d8352c2d9dfea4090"}' -H "Content-Type: application/json"
{"error":"unauthorized","reason":"Name or password is incorrect."}

Reading the budibase docs about couchdb again, you can use Fauxton to access couchdb using the url yourhost/db/_utils/#. This worked, but authentication failed here too. Obviously the couch admin user was not the one supplied by the env vars. Checking /opt/couchdb/etc/local.d/docker.ini actually showed another admin user than it should. So I decided to mount a custom .ini file to this dir. And finally, login to couch via Fauxton worked.

Now, I tried to use the web interface to replicate the db. The following error appeared in the docker logs:

couch_replicator_httpc: auth plugin initialization failed "https://.../db/app_bdc3b996c80c472d8352c2d9dfea4090/" {session_unexpected_result,"400","https://.../_session","..."}

Obviously, this is due to my reverse dns proxy (ref. https://github.com/apache/couchdb/issues/3986) so I added

[replicator]
auth_plugins = couch_replicator_auth_noop

to my custom .ini file. After restarting the container, the replication actually went through.

Finally, I used Fauxton to change the appId field of the document app_metadata to the _dev_-version and removed the deployments document completely. The first refresh of the app-list actually did nothing, but accessing the app directly using the url &HOST/builder/app/app_dev_bdc3b996c80c472d8352c2d9dfea4090 now worked. After that, "Exit to portal" from the hamburger menu braucht me right back to the app list and 🎉 my app was listed again.

Thanks again @shogunpurple ! You really made my Day! 👍

[shogunpurple]

Excellent work @MrApe - I was going to suggest fauxton but thought that script might have just done it automatically for you without having to perform the replication live in fauxton. Glad you got it resolved. I'm still not too sure how something like this may have been caused, but please do let us know if you see any other issues!