🐛 BUG: `D1_ERROR` when using D1 bindings with `drizzle-orm/d1`

lamualfa commented 11 months ago

Which Cloudflare product(s) does this pertain to?

D1

What version(s) of the tool(s) are you using?

wrangler 2.20.1
drizzle-orm 0.29.1
typescript 4.9.5
itty-router 4.0.23
better-sqlite3 9.2.2

What version of Node are you using?

21

What operating system are you using?

Linux

Describe the Bug

The workers work well locally when using wrangler dev --local --persist. But it gives an error when running it on Cloudflare after deploying it using wrangler publish.

Please provide a link to a minimal reproduction

import type { DrizzleD1Database } from 'drizzle-orm/d1'
import { drizzle } from 'drizzle-orm/d1'
import { Router, status, IRequest as IttyRequest } from 'itty-router'

import { lakeContactTable, lakeContactProfileTable } from './schema'
import { Contact, ContactProfile, ContactType, accessToken } from './const'

export interface Env {
  DB: D1Database
}

interface Request extends IttyRequest {
  db: DrizzleD1Database
  payload: any
}

async function injectDb(req: Request, env: Env): Promise<void> {
  const db = drizzle(env.DB)
  req.db = db
}

async function parsePayload(req: Request) {
  try {
    req.payload = await req.json()
  } catch (error) {}
}

async function verifyRequest(req: Request): Promise<void | Response> {
  const unsafeAccessToken = req.query['accessToken']
  if (!unsafeAccessToken) {
    return status(401)
  }

  if (unsafeAccessToken !== accessToken) {
    return status(403)
  }
}

const router = Router<Request, [Env]>()

router.post(
  '/contacts',
  verifyRequest,
  parsePayload,
  injectDb,
  async (req: Request) => {
    try {
      const payload = req.payload
      if (!Array.isArray(payload)) {
        return status(400)
      }

      const contactTypes = Object.values(ContactType)
      const contacts = payload
        .filter(
          (v) =>
            v.phoneNumber &&
            typeof v.phoneNumber === 'string' &&
            v.type &&
            contactTypes.includes(v.type)
        )
        .map((v) => {
          return {
            phoneNumber: v.phoneNumber,
            type: v.type,
            countryCode: v.countryCode || null,
            publicName: v.publicName || null,
            about: v.about || null,
            imageUrl: v.imageUrl || null,
            longitude: v.longitude || null,
            latitude: v.latitude || null,
            address: v.address || null,
          } as Contact
        })

      if (contacts.length) {
        await req.db.insert(lakeContactTable).values(contacts)
      }

      return status(200)
    } catch (error) {
      console.error(error)
    }
  }
)

router.post(
  '/contact-profiles',
  verifyRequest,
  parsePayload,
  injectDb,
  async (req: Request) => {
    try {
      const payload = req.payload
      if (!Array.isArray(payload)) {
        return status(400)
      }

      const profiles = payload
        .filter(
          (v) =>
            v.sourcePhoneNumber &&
            v.phoneNumber &&
            typeof v.sourcePhoneNumber === 'string' &&
            typeof v.phoneNumber === 'string'
        )
        .map((v) => {
          return {
            sourcePhoneNumber: v.sourcePhoneNumber,
            phoneNumber: v.phoneNumber,
            privateName: v.privateName || null,
            privateShortName: v.privateShortName || null,
          } as ContactProfile
        })

      if (profiles.length) {
        await req.db.insert(lakeContactProfileTable).values(profiles)
      }

      return status(200)
    } catch (error) {
      console.error(error)
    }
  }
)

export default {
  fetch: router.handle,
}

Please provide any relevant error logs

{
  "outcome": "exception",
  "scriptName": "waser-lite-workers",
  "diagnosticsChannelEvents": [],
  "exceptions": [
    {
      "name": "TypeError",
      "message": "Incorrect type for Promise: the Promise did not resolve to 'Response'.",
      "timestamp": 1702114174748
    }
  ],
  "logs": [
    {
      "message": [
        "Error: D1_ERROR"
      ],
      "level": "error",
      "timestamp": 1702114174748
    }
  ],
  "eventTimestamp": 1702114174417,
  "event": {
    "request": {
      "url": "https://<redacted>",
      "method": "POST",
      "headers": {
        "accept": "*/*",
        "accept-encoding": "gzip",
        "cf-connecting-ip": "182.253.245.149",
        "cf-ipcountry": "ID",
        "cf-ray": "832c39760ec63f58",
        "cf-visitor": "{\"scheme\":\"https\"}",
        "connection": "Keep-Alive",
        "content-length": "19097",
        "content-type": "application/json",
        "host": "<redacted>",
        "postman-token": "REDACTED",
        "user-agent": "PostmanRuntime/7.35.0",
        "x-forwarded-proto": "https",
        "x-real-ip": "182.253.245.149"
      },
      "cf": {
        "clientTcpRtt": 18,
        "longitude": "106.84460",
        "latitude": "-6.21140",
        "tlsCipher": "AEAD-AES128-GCM-SHA256",
        "continent": "AS",
        "asn": 17451,
        "clientAcceptEncoding": "gzip, deflate, br",
        "country": "ID",
        "tlsClientAuth": {
          "certIssuerDNLegacy": "",
          "certIssuerSKI": "",
          "certSubjectDNRFC2253": "",
          "certSubjectDNLegacy": "",
          "certFingerprintSHA256": "",
          "certNotBefore": "",
          "certSKI": "",
          "certSerial": "",
          "certIssuerDN": "",
          "certVerified": "NONE",
          "certNotAfter": "",
          "certSubjectDN": "",
          "certPresented": "0",
          "certRevoked": "0",
          "certIssuerSerial": "",
          "certIssuerDNRFC2253": "",
          "certFingerprintSHA1": ""
        },
        "tlsExportedAuthenticator": {
          "clientFinished": "4bdc8184827ad4718913461be028bd31b833efde84ae3a80e035857a9c7bc5c4",
          "clientHandshake": "bcc3e9d9eadbd1f3dea331dca7cf9834a658208acf12fd245a90a046eca7db60",
          "serverHandshake": "c4c6be18a99b3441edb7c9f90e7a57a30bb7e80b6d6f1e6bc34d8f24fd5e8650",
          "serverFinished": "522069443af906896317adcf4953aba38b2c3dfe44309bc941d56176bd85d1d2"
        },
        "tlsVersion": "TLSv1.3",
        "city": "Jakarta",
        "timezone": "Asia/Jakarta",
        "colo": "SIN",
        "region": "Jakarta",
        "requestPriority": "",
        "verifiedBotCategory": "",
        "httpProtocol": "HTTP/1.1",
        "regionCode": "JK",
        "asOrganization": "Biznet Networks",
        "edgeRequestKeepAliveStatus": 1
      }
    },
    "response": {
      "status": 500
    }
  },
  "id": 0
}

mrbbot commented 11 months ago

Hey! 👋 Thanks for raising this. A couple things to try:

Have you applied migrations (if any) to your remote database?
Could you try replacing console.log(error) with console.log(error.stack)? That might include some more information. Also try logging error.cause?.stack.

lamualfa commented 11 months ago

@mrbbot

Have you applied migrations (if any) to your remote database?

Yes I have.

Could you try replacing console.log(error) with console.log(error.stack)? That might include some more information. Also try logging error.cause?.stack.

The error caught by try catch is undefined.

The interesting thing that I found was, that the insert many functions in drizzle will produce an error if I insert many rows in a single query. My current solution is to break it up into small rows and execute insert many functions many times.

I don't need to do that when running it locally using wrangler. The error only occurs in Cloudflare server. So I guess there's a limitation of the size of the query in single execution when using D1 in the Cloudflare server.

mrbbot commented 11 months ago

So I guess there's a limitation of the size of the query in single execution when using D1 in the Cloudflare server.

That's a great insight. 😃 It looks like there are some limits here we should be enforcing locally too: https://developers.cloudflare.com/d1/platform/limits/.

cloudflare / workers-sdk