Dart app with Docker (AOT via scratch) cannot connect to MongoDB

[jdaibello]

Hi, good day.

I made a Dockerfile for running my Dart backend in containers (my objective is deploying the server by using Kubernetes on Digital Ocean) similar to the Dockerfile provided by the example in this repository, but my Dart server is not connecting to the MongoDB Atlas database (open for any connections) inside Docker (it can connect normally when running outside Docker).

My Dockerfile:

# Select amd64 image
FROM dart:latest as builder

RUN curl -fsSL https://deb.nodesource.com/setup_lts.x | bash - && \
apt-get install -y nodejs

# Sets the working directory to /app
WORKDIR /app

# Copies the current directory contents into the container at /app
COPY . .

# Create dart pubspec.yaml
RUN echo "name: backend" > pubspec.yaml && \
    echo "environment:" >> pubspec.yaml && \
    echo "  sdk: '>=2.18.0 <3.0.0'" >> pubspec.yaml

# Install any needed packages specified in pubspec.yaml
RUN dart pub add args
RUN dart pub add dotenv
RUN dart pub add json_annotation
RUN dart pub add orm
RUN dart pub add shelf --no-precompile
RUN dart pub add shelf_router
RUN dart pub add build_runner --dev
RUN dart pub add http --dev
RUN dart pub add json_serializable --dev
RUN dart pub add lints --dev --no-precompile
RUN dart pub add test --dev

# Install Prisma CLI
RUN npm install prisma

# Generate Prisma Client
RUN npx prisma generate &&\
    dart run build_runner build --delete-conflicting-outputs

# Dart compile executable
RUN dart compile exe bin/server.dart -o app-backend.exe

# Push schema to database
RUN npx prisma db push

# Set runtime image
FROM scratch

# Copy runtime dependencies
COPY --from=builder /runtime /
COPY --from=odroe/prisma-dart:amd64 /runtime /

# Copy executable
COPY --from=builder /app/app-backend.exe /app-backend.exe

# Copy Prisma Engine
COPY --from=builder /app/node_modules/prisma/query-engine-* /

# Run the executable
CMD ["/app-backend.exe"]

Obs: The command RUN npx prisma db push connect to MongoDB to push database changes.

My docker-compose.yaml:

version: '3.8'

services:
  app:
    build:
      context: .
    ports:
      - "8080:8080"
    container_name: soccer-stories-backend

My schema.prisma:

// This is your Prisma schema file,
// learn more about it in the docs: https://pris.ly/d/prisma-schema

generator client {
  provider = "dart run orm"
}

datasource db {
  provider = "mongodb"
  url      = env("DATABASE_URL")
}

model User {
  id String @id @default(auto()) @map("_id") @db.ObjectId
  email String @unique
  name String
  dateOfBirth DateTime?
}

My .env file:

# Environment variables declared in this file are automatically made available to Prisma.
# See the documentation for more detail: https://pris.ly/d/prisma-schema#accessing-environment-variables-from-the-schema

# Prisma supports the native connection string format for PostgreSQL, MySQL, SQLite, SQL Server, MongoDB and CockroachDB.
# See the documentation for all the connection string options: https://pris.ly/d/connection-strings

DATABASE_URL="mongodb+srv://xxxxxxxxxx:yyyyyyyyyy@mongodbcluster.gnuipvt.mongodb.net/soccer-stories?tls=true"
MONGODB_USERNAME="xxxxxxxxxx"
MONGODB_PASSWORD="yyyyyyyyyy"

And my bin/server.dart:

import 'dart:io';

import 'package:backend/src/generated/prisma/prisma_client.dart';
import 'package:dotenv/dotenv.dart';
import 'package:shelf/shelf.dart';
import 'package:shelf/shelf_io.dart';
import 'package:shelf_router/shelf_router.dart';
import 'package:orm/logger.dart';

// Configure routes.
final _router = Router()
  ..get('/', _rootHandler)
  ..get('/echo/<message>', _echoHandler);

Response _rootHandler(Request req) {
  return Response.ok('Hello, World!\n');
}

Response _echoHandler(Request request) {
  final message = request.params['message'];
  return Response.ok('$message\n');
}

Future<void> main(List<String> args) async {
  // Use any available host or container IP (usually `0.0.0.0`).
  final ip = InternetAddress.anyIPv4;

  // Configure a pipeline that logs requests.
  final handler = Pipeline().addMiddleware(logRequests()).addHandler(_router);

  // For running in containers, we respect the PORT environment variable.
  final port = int.parse(Platform.environment['PORT'] ?? '8080');
  final server = await serve(handler, ip, port);
  print('Server listening on port ${server.port}');

  var env = DotEnv(includePlatformEnvironment: true)..load();

  var dbUsername = env['MONGODB_USERNAME'];
  var dbPassword = env['MONGODB_PASSWORD'];

  final dbConnectionString =
      'mongodb+srv://$dbUsername:$dbPassword@mongodbcluster.gnuipvt.mongodb.net/soccer-stories?tls=true';

  final prisma = PrismaClient(
    stdout: Event.values,
    datasources: Datasources(db: dbConnectionString),
  );

  try {
    final user = await prisma.user.create(
      data: UserCreateInput(
        email: 'joao.${DateTime.now().millisecondsSinceEpoch}@gmail.com',
        name: 'João ${DateTime.now().millisecondsSinceEpoch}',
        dateOfBirth: DateTime(1999, 10, 29),
      ),
    );

    print(user.toJson());
  } finally {
    await prisma.$disconnect();
  }
}

Any help is appreciated.

[medz]

@jdaibello Thanks for asking, I'll test it when I have time, my guess is that the odroe/prisma-dart image is not properly providing the binary dependencies needed by MongoDB.

You should be able to tell what's missing from your Docker running logs, but I'd still follow the minimal example you provided to test it.

However, I can't guarantee a time, if you read my tweets you should know that our team is suffering from insufficient funds. Currently busy getting the team back on track, so I'll look into this as soon as I have as much free time as possible.

[medz]

@jdaibello Also, here's what I can offer to help troubleshoot this issue:

1. Build a dart via scratch image normally
2. Copy only your schema.prisma and binaries.
3. Check the package.json file of this repository and you will see DATABASE_URL=postgres://seven@localhost:5432/prisma-dart?schema=public <Prisma binary engine path> -m -g -r --datamodel -path example/prisma/schema.prisma -p 8080 code, replace URL with your desired database address.
4. Run it
5. You can view the running log through tools such as docker disktop, and the binary engine will prompt the missing dependencies, and then you can copy it from the builder image.

[jdaibello]

@medz I did your suggestion above, but the result in Docker logs is the same

My package.json

{
  "scripts": {
    "prisma:engine": "DATABASE_URL=mongodb+srv://username:password@mongodbcluster.gnuipvt.mongodb.net/soccer-stories node_modules/prisma/query-engine-darwin -m -g -r --datamodel-path prisma/schema.prisma -p 8080"
  },
  "prisma": {
    "schema": "prisma/schema.prisma"
  },
  "dependencies": {
    "prisma": "^4.13.0"
  }
}

My node_modules/prisma folder shows the query-engine-darwin-arm64file when I run the prisma generate command, because I'm using a MacBook with Apple Silicon (M2) chip, but inside Docker it runs alright without the -arm64 suffix.

I'll share the Docker logs with you below:

Server listening on port 8080
Calling http://127.0.0.1:39925/status (n=1)
Attempt 1/10 failed for status: Connection refused
Retrying after 81ms
Calling http://127.0.0.1:39925/status (n=2)
Attempt 2/10 failed for status: Connection refused
Retrying after 189ms
Calling http://127.0.0.1:39925/status (n=3)
Attempt 3/10 failed for status: Connection refused
Retrying after 324ms
Calling http://127.0.0.1:39925/status (n=4)
Attempt 4/10 failed for status: Connection refused
Retrying after 670ms
Calling http://127.0.0.1:39925/status (n=5)
Attempt 5/10 failed for status: Connection refused
Retrying after 1390ms
Calling http://127.0.0.1:39925/status (n=6)
Attempt 6/10 failed for status: Connection refused
Retrying after 3298ms
Calling http://127.0.0.1:39925/status (n=7)
Attempt 7/10 failed for status: Connection refused
Retrying after 7490ms
Calling http://127.0.0.1:39925/status (n=8)
Attempt 8/10 failed for status: Connection refused
Retrying after 12788ms
Calling http://127.0.0.1:39925/status (n=9)
Attempt 9/10 failed for status: Connection refused
Retrying after 28136ms
Calling http://127.0.0.1:39925/status (n=10)
Unhandled exception:
Connection refused
#0      IOClient.send (package:http/src/io_client.dart:94)
<asynchronous suspension>
#1      BaseClient._sendUnstreamed (package:http/src/base_client.dart:93)
<asynchronous suspension>
#2      _withClient (package:http/http.dart:166)
<asynchronous suspension>
#3      BinaryEngine._createEngineStatusValidator.<anonymous closure> (package:orm/binary_engine.dart:265)
<asynchronous suspension>
#4      _InternalRetry.call (package:orm/universal_engine.dart:298)
<asynchronous suspension>
#5      RetryOptions.retry (package:retry/retry.dart:131)
<asynchronous suspension>
#6      BinaryEngine._createProcess (package:orm/binary_engine.dart:189)
<asynchronous suspension>
#7      BinaryEngine.start (package:orm/binary_engine.dart:148)
<asynchronous suspension>
#8      UniversalEngine.request (package:orm/universal_engine.dart:59)
<asynchronous suspension>
#9      ModelDelegate._execute (package:orm/src/client/model_delegate.dart:40)
<asynchronous suspension>
#10     PrismaFluent.queryBuilder.<anonymous closure> (package:orm/src/client/prisma_fluent.dart:37)
<asynchronous suspension>

Note that my backend always connect to the database (almost of time not in the first try) if I run it using my physical machine, i.e. I only reproduced this behavior using Docker or Kubernetes

[jdaibello]

Do you know if it may happens with other databases inside Docker/Kubernetes? I prefer MongoDB (by MongoDB Atlas) because it has a free cluster in the cloud, but I can use another DB option. I'm just starting the development of my personal project.

[medz]

@jdaibello I see your log is running a Dart AOT program normally, I mean you should manually run this binary file in node_modules/prisma/prisma-engine-xxx. I think you copied it into scratch correctly.