🐛 BUG: `D1_ERROR` when using D1 bindings with `drizzle-orm/d1`

lamualfa commented 6 months ago

Which Cloudflare product(s) does this pertain to?

D1

What version(s) of the tool(s) are you using?

wrangler 2.20.1
drizzle-orm 0.29.1
typescript 4.9.5
itty-router 4.0.23
better-sqlite3 9.2.2

What version of Node are you using?

21

What operating system are you using?

Linux

Describe the Bug

The workers work well locally when using wrangler dev --local --persist. But it gives an error when running it on Cloudflare after deploying it using wrangler publish.

Please provide a link to a minimal reproduction

import type { DrizzleD1Database } from 'drizzle-orm/d1'
import { drizzle } from 'drizzle-orm/d1'
import { Router, status, IRequest as IttyRequest } from 'itty-router'

import { lakeContactTable, lakeContactProfileTable } from './schema'
import { Contact, ContactProfile, ContactType, accessToken } from './const'

export interface Env {
  DB: D1Database
}

interface Request extends IttyRequest {
  db: DrizzleD1Database
  payload: any
}

async function injectDb(req: Request, env: Env): Promise<void> {
  const db = drizzle(env.DB)
  req.db = db
}

async function parsePayload(req: Request) {
  try {
    req.payload = await req.json()
  } catch (error) {}
}

async function verifyRequest(req: Request): Promise<void | Response> {
  const unsafeAccessToken = req.query['accessToken']
  if (!unsafeAccessToken) {
    return status(401)
  }

  if (unsafeAccessToken !== accessToken) {
    return status(403)
  }
}

const router = Router<Request, [Env]>()

router.post(
  '/contacts',
  verifyRequest,
  parsePayload,
  injectDb,
  async (req: Request) => {
    try {
      const payload = req.payload
      if (!Array.isArray(payload)) {
        return status(400)
      }

      const contactTypes = Object.values(ContactType)
      const contacts = payload
        .filter(
          (v) =>
            v.phoneNumber &&
            typeof v.phoneNumber === 'string' &&
            v.type &&
            contactTypes.includes(v.type)
        )
        .map((v) => {
          return {
            phoneNumber: v.phoneNumber,
            type: v.type,
            countryCode: v.countryCode || null,
            publicName: v.publicName || null,
            about: v.about || null,
            imageUrl: v.imageUrl || null,
            longitude: v.longitude || null,
            latitude: v.latitude || null,
            address: v.address || null,
          } as Contact
        })

      if (contacts.length) {
        await req.db.insert(lakeContactTable).values(contacts)
      }

      return status(200)
    } catch (error) {
      console.error(error)
    }
  }
)

router.post(
  '/contact-profiles',
  verifyRequest,
  parsePayload,
  injectDb,
  async (req: Request) => {
    try {
      const payload = req.payload
      if (!Array.isArray(payload)) {
        return status(400)
      }

      const profiles = payload
        .filter(
          (v) =>
            v.sourcePhoneNumber &&
            v.phoneNumber &&
            typeof v.sourcePhoneNumber === 'string' &&
            typeof v.phoneNumber === 'string'
        )
        .map((v) => {
          return {
            sourcePhoneNumber: v.sourcePhoneNumber,
            phoneNumber: v.phoneNumber,
            privateName: v.privateName || null,
            privateShortName: v.privateShortName || null,
          } as ContactProfile
        })

      if (profiles.length) {
        await req.db.insert(lakeContactProfileTable).values(profiles)
      }

      return status(200)
    } catch (error) {
      console.error(error)
    }
  }
)

export default {
  fetch: router.handle,
}

Please provide any relevant error logs

{
  "outcome": "exception",
  "scriptName": "waser-lite-workers",
  "diagnosticsChannelEvents": [],
  "exceptions": [
    {
      "name": "TypeError",
      "message": "Incorrect type for Promise: the Promise did not resolve to 'Response'.",
      "timestamp": 1702114174748
    }
  ],
  "logs": [
    {
      "message": [
        "Error: D1_ERROR"
      ],
      "level": "error",
      "timestamp": 1702114174748
    }
  ],
  "eventTimestamp": 1702114174417,
  "event": {
    "request": {
      "url": "https://<redacted>",
      "method": "POST",
      "headers": {
        "accept": "*/*",
        "accept-encoding": "gzip",
        "cf-connecting-ip": "182.253.245.149",
        "cf-ipcountry": "ID",
        "cf-ray": "832c39760ec63f58",
        "cf-visitor": "{\"scheme\":\"https\"}",
        "connection": "Keep-Alive",
        "content-length": "19097",
        "content-type": "application/json",
        "host": "<redacted>",
        "postman-token": "REDACTED",
        "user-agent": "PostmanRuntime/7.35.0",
        "x-forwarded-proto": "https",
        "x-real-ip": "182.253.245.149"
      },
      "cf": {
        "clientTcpRtt": 18,
        "longitude": "106.84460",
        "latitude": "-6.21140",
        "tlsCipher": "AEAD-AES128-GCM-SHA256",
        "continent": "AS",
        "asn": 17451,
        "clientAcceptEncoding": "gzip, deflate, br",
        "country": "ID",
        "tlsClientAuth": {
          "certIssuerDNLegacy": "",
          "certIssuerSKI": "",
          "certSubjectDNRFC2253": "",
          "certSubjectDNLegacy": "",
          "certFingerprintSHA256": "",
          "certNotBefore": "",
          "certSKI": "",
          "certSerial": "",
          "certIssuerDN": "",
          "certVerified": "NONE",
          "certNotAfter": "",
          "certSubjectDN": "",
          "certPresented": "0",
          "certRevoked": "0",
          "certIssuerSerial": "",
          "certIssuerDNRFC2253": "",
          "certFingerprintSHA1": ""
        },
        "tlsExportedAuthenticator": {
          "clientFinished": "4bdc8184827ad4718913461be028bd31b833efde84ae3a80e035857a9c7bc5c4",
          "clientHandshake": "bcc3e9d9eadbd1f3dea331dca7cf9834a658208acf12fd245a90a046eca7db60",
          "serverHandshake": "c4c6be18a99b3441edb7c9f90e7a57a30bb7e80b6d6f1e6bc34d8f24fd5e8650",
          "serverFinished": "522069443af906896317adcf4953aba38b2c3dfe44309bc941d56176bd85d1d2"
        },
        "tlsVersion": "TLSv1.3",
        "city": "Jakarta",
        "timezone": "Asia/Jakarta",
        "colo": "SIN",
        "region": "Jakarta",
        "requestPriority": "",
        "verifiedBotCategory": "",
        "httpProtocol": "HTTP/1.1",
        "regionCode": "JK",
        "asOrganization": "Biznet Networks",
        "edgeRequestKeepAliveStatus": 1
      }
    },
    "response": {
      "status": 500
    }
  },
  "id": 0
}

mrbbot commented 6 months ago

Hey! 👋 Thanks for raising this. A couple things to try:

Have you applied migrations (if any) to your remote database?
Could you try replacing console.log(error) with console.log(error.stack)? That might include some more information. Also try logging error.cause?.stack.

lamualfa commented 6 months ago

@mrbbot

Have you applied migrations (if any) to your remote database?

Yes I have.

Could you try replacing console.log(error) with console.log(error.stack)? That might include some more information. Also try logging error.cause?.stack.

The error caught by try catch is undefined.

The interesting thing that I found was, that the insert many functions in drizzle will produce an error if I insert many rows in a single query. My current solution is to break it up into small rows and execute insert many functions many times.

I don't need to do that when running it locally using wrangler. The error only occurs in Cloudflare server. So I guess there's a limitation of the size of the query in single execution when using D1 in the Cloudflare server.

mrbbot commented 6 months ago

So I guess there's a limitation of the size of the query in single execution when using D1 in the Cloudflare server.

That's a great insight. 😃 It looks like there are some limits here we should be enforcing locally too: https://developers.cloudflare.com/d1/platform/limits/.

cloudflare / workers-sdk