Cannot find module 'better-sqlite3' when running in Docker Container

[awanlin]

Expected Behavior

Backstage should start up

Actual Behavior

When trying to start up Backstage in a Docker container we get: Cannot find module 'better-sqlite3'

The fix is to move better-sqlite3": "^7.5.0 from the devDependencies section to the dependencies section in the \packages\backend\packages.json file.

Steps to Reproduce

1. Ran npx @backstage/create-app to create a instances of Backstage, my version in the backstage.json is 1.2.0
2. Then I added a Dockerfile to the root and used this, pretty much the same as from the multi-stage docker docks on backstage.io:

   
   # Stage 1 - Create yarn install skeleton layer
   FROM node:16-bullseye-slim AS packages

WORKDIR /app COPY package.json yarn.lock ./

COPY packages packages

Comment this out if you don't have any internal plugins

COPY plugins plugins

RUN find packages ! -name "package.json" -mindepth 2 -maxdepth 2 -exec rm -rf {} +

Stage 2 - Install dependencies and build packages

FROM node:16-bullseye-slim AS build

WORKDIR /app COPY --from=packages /app .

install sqlite3 dependencies

RUN apt-get update && \ apt-get install -y --no-install-recommends libsqlite3-dev python3 build-essential && \ yarn config set python /usr/bin/python3

RUN yarn install --frozen-lockfile --network-timeout 600000 && rm -rf "$(yarn cache dir)"

COPY . .

RUN yarn tsc RUN yarn --cwd packages/backend build

Stage 3 - Build the actual backend image and install production dependencies

FROM node:16-bullseye-slim

WORKDIR /app

install sqlite3 and TechDocs dependencies

RUN apt-get update && \ apt-get install -y --no-install-recommends libsqlite3-dev python3 build-essential && \ rm -rf /var/lib/apt/lists/* && \ yarn config set python /usr/bin/python3

Copy the install dependencies from the build stage and context

COPY --from=build /app/yarn.lock /app/package.json /app/packages/backend/dist/skeleton.tar.gz ./ RUN tar xzf skeleton.tar.gz && rm skeleton.tar.gz

RUN yarn install --frozen-lockfile --production --network-timeout 600000 && rm -rf "$(yarn cache dir)"

Copy the built packages from the build stage

COPY --from=build /app/packages/backend/dist/bundle.tar.gz . RUN tar xzf bundle.tar.gz && rm bundle.tar.gz

Copy any other files that we need at runtime

COPY app-config.yaml ./

CMD ["node", "packages/backend", "--config", "app-config.yaml"]

5. Then make sure your .dockerignore looks like this:
```text
.git
*.local.yaml
node_modules
packages/*/dist
packages/*/node_modules
plugins/*/dist
plugins/*/node_modules

6. From the root of your project run this to build the docker container: docker image build -t backstage .
7. Now run the container: docker run -it -p 7007:7007 backstage

Notice the container does not start up and you get this error: Cannot find module 'better-sqlite3'

Context

This prevents Backstage from running in a docker container based on the documentation on Backstage.io

Your Environment

Google Chrome Version 101.0.4951.64 (Official Build) (arm64)

OS:   Darwin 21.5.0 - darwin/x64
node: v14.18.1
yarn: 1.22.18
cli:  0.17.1 (installed)

Dependencies:
  @backstage/app-defaults                          1.0.2
  @backstage/backend-common                        0.13.5
  @backstage/backend-tasks                         0.3.1
  @backstage/catalog-client                        1.0.2
  @backstage/catalog-model                         1.0.2
  @backstage/cli-common                            0.1.9
  @backstage/cli                                   0.17.1
  @backstage/config-loader                         1.1.1
  @backstage/config                                1.0.1
  @backstage/core-app-api                          1.0.2
  @backstage/core-components                       0.9.4
  @backstage/core-plugin-api                       1.0.2
  @backstage/errors                                1.0.0
  @backstage/integration-react                     1.1.0
  @backstage/integration                           1.2.0
  @backstage/plugin-api-docs                       0.8.5
  @backstage/plugin-app-backend                    0.3.32
  @backstage/plugin-auth-backend                   0.14.0
  @backstage/plugin-auth-node                      0.2.1
  @backstage/plugin-catalog-backend                1.1.2
  @backstage/plugin-catalog-common                 1.0.2
  @backstage/plugin-catalog-graph                  0.2.17
  @backstage/plugin-catalog-import                 0.8.8
  @backstage/plugin-catalog-react                  1.1.0
  @backstage/plugin-catalog                        1.2.0
  @backstage/plugin-github-actions                 0.5.5
  @backstage/plugin-org                            0.5.5
  @backstage/plugin-permission-common              0.6.1
  @backstage/plugin-permission-node                0.6.1
  @backstage/plugin-permission-react               0.4.1
  @backstage/plugin-proxy-backend                  0.2.26
  @backstage/plugin-scaffolder-backend             1.2.0
  @backstage/plugin-scaffolder-common              1.1.0
  @backstage/plugin-scaffolder                     1.2.0
  @backstage/plugin-search-backend-module-pg       0.3.3
  @backstage/plugin-search-backend-node            0.6.1
  @backstage/plugin-search-backend                 0.5.2
  @backstage/plugin-search-common                  0.3.4
  @backstage/plugin-search-react                   0.2.0
  @backstage/plugin-search                         0.8.1
  @backstage/plugin-tech-radar                     0.5.12
  @backstage/plugin-techdocs-backend               1.1.1
  @backstage/plugin-techdocs-module-addons-contrib 1.0.0
  @backstage/plugin-techdocs-node                  1.1.1
  @backstage/plugin-techdocs-react                 1.0.0
  @backstage/plugin-techdocs                       1.1.1
  @backstage/plugin-user-settings                  0.4.4
  @backstage/release-manifests                     0.0.3
  @backstage/search-common                         0.3.4
  @backstage/test-utils                            1.1.0
  @backstage/theme                                 0.2.15
  @backstage/types                                 1.0.0
  @backstage/version-bridge                        1.0.1
✨  Done in 3.01s.

[Rugvip]

This is intended as of the v1.2 release. We removed SQLite from the Docker image as we now consider that to be the production deployment, which uses PostgreSQL. Having the SQLite dependency present in production means we both need to install additional dependencies from APT, as well as SQLite itself, both of which bloat the image and build process.

It's something you can ofc change in your own app though, either by adding back better-sqlite as a prod dependency, installing it separately, or perhaps having a separate development image that runs a full yarn install.

Happy to receive feedback on this change though, if you think there's something we can do that clarifies/simplifies things.

[awanlin]

Ok, that makes sense to me. I really just logged this as I ran into this when I was trying to help someone on Discord and this was the fastest way to get something up and running. I'm wondering if just adding a note about this on the Docker documentation might be helpful? I'm thinking for those doing a Backstage POC and just want something simple to deploy?

[jdotw]

As a relatively new user to Backstage, this added a sizeable "uhhh maybe this isn't ready for prime time" sentiment. If I follow the documentation about how to deploy Backstage, I expect it to work as documented. In this case, it does not.

[benjdlambert]

@jdotw what documentation are you referring to? We probably just missed something when rewriting these docs recently. :pray:

[derjust]

I can't talk for @jdotw nor @awanlin - but i think we all fell into the same trap:

While https://backstage.io/docs/deployment/docker mentions

Something that goes for all of these docker deployment strategies is that they are stateless, so for a production deployment you will want to set up and connect to an external PostgreSQL instance where the backend plugins can store their state, rather than using SQLite.

(Which makes sense as a statement but I think - in the spirit of *-as-code -there are valid production use cases that will not need/have a PostgreSQL - thus sticking with SQlite)

Further down in the document the multi-stage build installs SQlite dependencies

# install sqlite3 dependencies

and

# install sqlite3 dependencies, you can skip this if you don't use sqlite3 in the image

thus leaving the impression that the Dockerfile example there is 'ready-to-go' for SQlite deployments (and for PostgreSQL one can remove those two lines from the Dockerfile)

Not trying to complain here, just trying to help out as I think Backstage is a great tool though with some rough edges like this one :-)

[benjdlambert]

@derjust some times us as maintainers get a little lost amongst the weeds to be honest, and it's hard for us to pretend to be people that are potentially first time readers, so we would really appreciate some contributions on where you think we could improve the docs or whatever to make it easier to get setup and stop leading you down rabbit holes. 🤗

Like I said previously, we shuffled some docs around and tried to improve some bits, so it could be that we've missed some things, or they don't make sense anymore.

[markdavidmc0]

I just ran into the same issue and came across this thread. Thanks for the comments.

[ChrisJBurns]

I'm curious around how the local and production config is loaded. I used the create-app method of generating my backbase project and it gave me a backend Dockerfile that had a --config app-config.production.yaml flag at the end. I noticed when it got to Kubernetes, it weren't actually loading the config file and I noticed that I was getting the same error in this issue.

I've since copied the pg database config from the app-config.production.yaml file and put it inside the app-config.yaml file as that's the one that gets loaded in Kubernetes. I've even applied the hotfix to my package.json to put the sqlite dep in the standard deps instead devDeps, and yet I've still recieved the same error in the Kubernetes deployment.

But, when i run a yarn dev, it now starts with the app-config.yaml which is expected, but it tries to connect to a postgres database now. And because the sqlite config isn't in my app-config.local.yaml, it doesn't start. So I've had to manually add it into my local config. But the problem now, is because my local config isn't committed to Git. If someone else was to Clone the repo, they would get the same error when they run the yarn dev and will have to manually add the sqlite config into their local config yaml.

[markdavidmc0]

Interesting.

Let me try replicate. I still haven't figured this out yet myself and was hoping to start using Backstage for my company. The project has stalled a bit until I can conjure up the courage to give it another go 😅.

[ChrisJBurns]

I've played around with the Helm chart I'm using and it seemed to be that, that was causing the issue. It was overriding the command when the container started and it wasn't passing it the correct app-config.yaml.

[rezarajabii]

I've just encountered this issue trying to get Backstage up and running just to give it a try. Just wanted to say the documentation is still lacking this information

[lenichols]

Having the same issue.... anyone get around it?

[markdavidmc0]

Sorry, still no luck. I gave up on the project until there is better direction on this.

[awanlin]

@lenichols The fix is to move better-sqlite3": "^7.5.0 from the devDependencies section to the dependencies section in the \packages\backend\packages.json file.

@markdavidmc0 sorry to hear this has been an issue for you, the above fix does not resolve the problem for you? Could you provide more details then and I can try and help you out?

[Rugvip]

@awanlin thank you for chiming in with a fix!

For anyone having trouble, I'm curious what path you took to deploy Backstage, just to make sure we can add improvements where needed. Did you go through https://backstage.io/docs/getting-started/, https://backstage.spotify.com/learn/standing-up-backstage/, or something else?

[twanbeeren]

@lenichols The fix is to move better-sqlite3": "^7.5.0 from the devDependencies section to the dependencies section in the \packages\backend\packages.json file.

@markdavidmc0 sorry to hear this has been an issue for you, the above fix does not resolve the problem for you? Could you provide more details then and I can try and help you out?

This fix indeed works. Thank you for commenting. My app couldn't find the module, and then tried to install it with npm install, but only the yarn cli works for backstage.

[awanlin]

@Rugvip you're welcome, happy to help. I've created PR #13021 to add a note about this in the deployment/docker documentation. What do you think? I did fix the double "to" in the PR

[awanlin]

The note above has been replaced by a better solution via #13079