search

vercel / ai

Build AI-powered applications with React, Svelte, Vue, and Solid
https://sdk.vercel.ai/docs
Other
9.56k stars 1.41k forks source link

Edge Runtime: convertToCoreMessages() fails to process base64 image data, unlike in local development #2616

Closed steveoon closed 1 month ago

steveoon commented 1 month ago

Description

When using the AI SDK in a Vercel Edge Runtime environment, there's an issue with handling image attachments in the convertToCoreMessages() function. This function, provided by the SDK, appears to fail when processing base64-encoded image data in the Edge Runtime, whereas it works correctly in local development and Node.js Runtime environments.

This issue occurs specifically when:

  1. Using the AI SDK (version 3.3.4) in a Next.js project
  2. Deployed to Vercel using Edge Runtime
  3. Sending messages with image attachments

The error message received in the Edge Runtime is: "Error converting messages to core messages: Invalid data content. Content string is not a base64-encoded media."

Importantly, this issue does not occur when using Node.js Runtime on Vercel or during local development.

Key Observations:

  1. In local development, convertToCoreMessages() successfully processes the image data and converts it to a Uint8Array:

    Message 8:
 Role: user
 Content:
   Part 0 (Text): Can you see this image...
   Part 1 (Image):
     Data: [Uint8Array]

    This log is from the local development environment in an API route, printed after image uploading.

  2. In the Vercel Edge Runtime, the function seems to throw an error before reaching this point, as we don't see this log output.

  3. The base64 data in experimental_attachments before the conversion appears correct.

  4. The error message suggests that the data:image/webp;base64, prefix is missing from the base64 string when the error occurs.

Steps to reproduce:

  1. Set up a Next.js project with AI SDK
  2. Implement a chat interface that allows image uploads
  3. Convert uploaded images to base64 and attach to messages
  4. Deploy to Vercel using Edge Runtime
  5. Attempt to send a message with an image attachment

Code example

/**
 * 将 Message 数组转换为 CoreMessage 数组
 * @param messages - 要转换的 Message 数组
 * @returns 转换后的 CoreMessage 数组
 * @throws 如果转换过程中出现错误则抛出异常
 */
export function convertMessagesToCoreMessages(messages: Message[]) {
  try {
    console.log(
      'Original messages:',
      JSON.stringify(
        messages,
        (key, value) => {
          if (key === 'experimental_attachments' && Array.isArray(value)) {
            return value.map((attachment) => ({
              ...attachment,
              url: attachment.url.substring(0, 100) + '...',
            }));
          }
          return value;
        },
        2
      )
    );

    const compatibleMessages = prepareCoreMessages(messages);

    console.log(
      'Compatible messages:',
      JSON.stringify(
        compatibleMessages,
        (key, value) => {
          if (key === 'experimental_attachments' && Array.isArray(value)) {
            return value.map((attachment) => ({
              ...attachment,
              url: attachment.url.substring(0, 100) + '...',
            }));
          }
          return value;
        },
        2
      )
    );

    // 比较 experimental_attachments
    messages.forEach((msg, index) => {
      if (msg.experimental_attachments) {
        console.log(
          `Message ${index} attachments:`,
          msg.experimental_attachments.map((att) => ({
            name: att.name,
            contentType: att.contentType,
            urlLength: att.url.length,
            urlType: typeof att.url,
          }))
        );
      }
    });

    compatibleMessages.forEach((msg, index) => {
      if (msg.experimental_attachments) {
        console.log(
          `Compatible message ${index} attachments:`,
          msg.experimental_attachments.map((att) => ({
            name: att.name,
            contentType: att.contentType,
            urlLength: att.url.length,
            urlType: typeof att.url,
          }))
        );
      }
    });

    return convertToCoreMessages(compatibleMessages);
  } catch (error) {
    console.error('Error details:', error);
    throw new Error(
      `Error converting messages to core messages: ${(error as Error).message}`
    );
  }
}

// API Route
const coreMessages = convertMessagesToCoreMessages(messages);
console.log('Core messages after conversion:');
    coreMessages.forEach((msg, index) => {
      console.log(`Message ${index}:`);
      console.log(`  Role: ${msg.role}`);

      if (typeof msg.content === 'string') {
        console.log(`  Content: ${msg.content.substring(0, 100)}...`);
      } else if (Array.isArray(msg.content)) {
        console.log('  Content:');
        msg.content.forEach((part, partIndex) => {
          if (part.type === 'text') {
            console.log(
              `    Part ${partIndex} (Text): ${part.text.substring(0, 50)}...`
            );
          } else if (part.type === 'image') {
            console.log(`    Part ${partIndex} (Image):`);
            if (typeof part.image === 'string') {
              console.log(`      Data: ${part.image.substring(0, 50)}...`);
            } else if (part.image instanceof URL) {
              console.log(`      URL: ${part.image.toString()}`);
            } else {
              console.log(`      Data: [${part.image.constructor.name}]`);
            }
            if (part.mimeType) {
              console.log(`      MIME Type: ${part.mimeType}`);
            }
          }
        });
      }

      if (msg.role === 'tool') {
        console.log(`  Tool Name: ${msg.content}`);
      }
    });
    const result = await streamText({
      model: whichModel,
      system: systemContent,
      maxTokens: maxTokens,
      temperature: 0.8,
      messages: coreMessages,
      ...

Additional context

Additional context:

We would appreciate any insights into why this conversion is failing specifically in the Edge Runtime and how we can ensure consistent behavior across different environments. Additionally, any guidance on how to properly handle or preprocess the base64 data to make it compatible with the Edge Runtime would be highly valuable.

The logs

[0] Message 4 attachments: [
[0]   { name: 'IMG_6958.jpg', contentType: 'image/jpeg', urlLength: 584167 }
[0] ]
[0] Message 8 attachments: [
[0]   {
[0]   name: 'iShot_2024-01-14_11.45.50.png',
[0]   contentType: 'image/png',
[0]   urlLength: 8654
[0] }
[0] ]
[0] Message 10 attachments: [
[0]   {
[0]   name: 'u=3676253094,2585937680&fm=253&fmt=auto&app=138&f=JPEG.webp',
[0]   contentType: 'image/webp',
[0]   urlLength: 17827
[0] }
[0] ]
[0] Compatible message 4 attachments: [
[0]   { name: 'IMG_6958.jpg', contentType: 'image/jpeg', urlLength: 584167 }
[0] ]
[0] Compatible message 8 attachments: [
[0]   {
[0]   name: 'iShot_2024-01-14_11.45.50.png',
[0]   contentType: 'image/png',
[0]   urlLength: 8654
[0] }
[0] ]
[0] Compatible message 10 attachments: [
[0]   {
[0]   name: 'u=3676253094,2585937680&fm=253&fmt=auto&app=138&f=JPEG.webp',
[0]   contentType: 'image/webp',
[0]   urlLength: 17827
[0] }
[0] ]

Original Messages

"role": "user",
[0]     "content": "再看看这个呢",
[0]     "experimental_attachments": [
[0]       {
[0]         "name": "u=3676253094,2585937680&fm=253&fmt=auto&app=138&f=JPEG.webp",
[0]         "contentType": "image/webp",
[0]         "url": "data:image/webp;base64,UklGRiA0AABXRUJQVlA4IBQ0AABQJAGdASr0AU0BPpFAm0slo7IspDRLekASCWVuZYDEvkinFc1tX..."
[0]       }
[0]     ]

package.json

"dependencies": {
    "@ai-sdk/anthropic": "^0.0.39",
    "@ai-sdk/openai": "^0.0.44",
    "ai": "^3.3.4",
    "next": "14.2.5",
steveoon commented 1 month ago

After analyzing error log, here are some findings:

  1. Error Type: AI_InvalidDataContentError: Invalid data content. Content string is not a base64-encoded media.

    This indicates that the system expects to receive base64-encoded media content, but the actual content received doesn't match this format.

  2. Error Location: The error stack shows that this issue occurs within the internal processing of the AI library, specifically involving Next.js route modules and Edge Functions.

  3. Root Cause: The deeper cause of the error is a TypeError suggesting an "Illegal invocation". This typically means a function was called with an incorrect context (this reference).

  4. Environment: The error is happening in a Cloudflare Workers environment, as evident from the URL link in the error message.

  5. Data Content: The error message ends with a long string that appears to be base64 encoded. This might be the actual content causing the error.

lgrammel commented 1 month ago

Could be a variant of this: https://github.com/vercel/ai/issues/1333

lgrammel commented 1 month ago

@steveoOn I suspect that we are using some function that is not available in the cloudflare workers environment, thus the type error. could you identify which call the type error happens on?

lgrammel commented 1 month ago

this is the problematic instruction:

CleanShot 2024-08-09 at 15 29 55

lgrammel commented 1 month ago

It might be the invocation with this.

@steveoOn can you try out the following 2 code pieces on cloudflare workers if you get a chance?

console.log(globalThis.atob("dGVzdA=="));
const atob = globalThis.atob;
console.log(atob("dGVzdA=="));

I suspect cloudflare does some extra checks for the this context. Thus the first code block would throw an error, and the 2nd would succeed.

steveoon commented 1 month ago
image

@lgrammel I've tried this on Cloudflare Worker, and shows all passed.

lgrammel commented 1 month ago

@steveoOn thanks. this is strange because the stack trace 

cause: TypeError: Illegal invocation: function called with incorrect `this` reference. See https://developers.cloudflare.com/workers/observability/errors/#illegal-invocation-errors for details.
    at (node_modules/.pnpm/@ai-sdk+provider-utils@1.0.9_zod@3.23.6/node_modules/@ai-sdk/provider-utils/dist/index.mjs:509:0)

points to exactly this location

lgrammel commented 1 month ago

double checked the location by going up the stack to the 2nd line of the cause (742):

CleanShot 2024-08-09 at 16 58 31

lgrammel commented 1 month ago

Therefore we have a confirmed location of the issue, but we were unable to reproduce it. The only difference that I'm aware of is that we use different data in the test.

@steveoOn can you use the data that caused the failure in the test that you just ran? also, are there any other differences between the worker env where the issue occurred and your test env (versions, settings, etc)?

steveoon commented 1 month ago

Therefore we have a confirmed location of the issue, but we were unable to reproduce it. The only difference that I'm aware of is that we use different data in the test.

@steveoOn can you use the data that caused the failure in the test that you just ran? also, are there any other differences between the worker env where the issue occurred and your test env (versions, settings, etc)?

@lgrammel I've deployed a testing Demo on Vercel: https://multi-modal-chatbot-demo.vercel.app

This is the Demo Github Repository: https://github.com/steveoOn/multi-modal-chatbot-demo.git

The testing Data:

[
  {
    "id": "1",
    "role": "user",
    "content": "describe image",
    "experimental_attachments": [
      {
        "name": "iShot_2024-01-14_11.45.50.png",
        "contentType": "image/png",
        "url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAcIAAAHCCAYAAAB8GMlFAAAZEElEQVR4nO3d3XoaSbIFUHy+ef9X1rmY1thtISkRWeSOiLUubTp/qypIA7t/vb29vd0AYKj/Oz0AADhJIQRgNIUQgNEUQgBGUwgBGE0hBGA0hRCA0RRCAEZTCAEYTSEEYDSFEIDRFEIARlMIARhNIQRgNIUQgNEUQgBGUwgBGE0hBGA0hRCA0RRCAEZTCAEYTSEEYDSFEIDRFEIARlMIARhNIQRgNIUQgNEUQgBGUwgBGE0hBGA0hRCA0RRCAEZTCAEYTSEEYDSFEIDRFEIARlMIARhNIQRgNIUQgNH+c3oAz/j169fpIVzq7e3t9BCATTyvcv16Kzj67hfU33Zt0d/r9my7q+2l97t7fLt1mceqLvN453mVr1QhnHZB/e2Zrfps7X7a5mp76f3uHt9uXeaxqss8bp5XpfaszGeE0y+q2xNr8NV/95M2V9tL73f3+HbrMo9VXeZxKzjeK1RagzKFEACuUKIQVnpncTVrAdnco79VWYsShRAArhJfCKu8o3ilR9fkqw+tf/KB9mp76f3uHt9uXeaxqsM8PK8+qrAm8YWQPe49SJ55uKy2l97v7vHt1mUeq7rMg1rifz5R4d3ECeHbBiN5Xt2X/rwqnSyzKm0T3CzAZzyvXq91IUy7oN69j2vCBfaILkk16f3u1iXp5/Q6p+3ruwnPq5afEb69vcVeVH+qMs5XuHeTPfNn+n2N3fM4Nd+T61zlOVBlnD/RrhBW3KiKY96pS1JNer+7dUn6ObnOFe/9imP+TrtCCACPaFUIK79TqTx24HGV7/nKY7+nVSEEgEe1KYQd3qF0mMNPdEmqSe93ty5JPyf6Tdi/Z3WYw7s2hZDauiTVpPe7W5ekn/R15lptkmXCp7Fs2nyhk2n3b5f5tv5B/aOe/ap0+mYDfXhe7aMQbvyt0Hs71S+w0wkbqSSo9O63ynXvebXf6M8If/36dckPZq9q9xXSk0xOkaDSu98K173n1XXGFsJTN2Gy9CSTUySo9O63wnXveXWtsYUQAG5TC+Er3/lMfpcFPM/z6nojCyEAvBtXCE+846nyLis9yeQUCSq9+02+7j2vXmNcIeRrEjbuk6DSu1/X/Wx+R8gHHgD3ra7L7tftpt/nXkc/ToQAjOZEOMipZJRTyR7pSSbcl35d0Y8T4RCnklFOJXukJ5lwX/p1RU8K4QCnklFOJXukJ5lwX/p1RV8KIQCjKYQAjKYQAjCaQjjAqWSUU8ke6Ukm3Jd+XdGXQjjEqWSUU8ke6Ukm3Jd+XdGT3xEOcioZ5VSyR3qSCfelX1f040QIwGhOhIN0SexITwqZlpDTZXzp1xXXcSIcoktiR3pSyLSEnC7jS7+uuNa4QnjiXd7pd5ZdEjvSk0KmJeR0GV/ydTXxeXXCuEIIAH8aWQhf+Y5n4rsrYB/Pq+uNLIQA8G5sITz14+oTuiR2pCeFTEvI6TK+9OvqNux5dcLYQnj7Z+Ov2Pyr2n1Gl8SO9KSQaQk5XcaXfl3dhj2vXu3XW/gKrH5ra8c0nv2G2CvHEL5tMJLn1XV9XckP6v+QvlkA7zyv9lEIB5Gw8Zz09UvfN+Mj1ejPCCeRsPGc9PVL3zfjI1mbQtjhor1qDh0SNk5KX7/0fTO+17X7Sh3m8K5NIQSAn2hVCCu/Q6k8duBxle/5ymO/p1UhBIBHtSuEFd+pXD3mTgkbJ6SvX/q+Gd/nPK8ytCuEt382qsJmvXKcnRI2Tkhfv/R9M77PeV6d1yZZ5itpU+w4J6Dnvd1xTn8b8YP6ru9igH48r15vRCHkv3YnZ2hPe9qjg5afEfLR7uQM7WlPe3QRXwi9K/vo0TXZnZyhPe1p7z7Pq48qrEl8IQSAK5UohBXeUbyKtYBs7tHfqqxFiUIIAFcpUwirvLO40k/XYHdyhva0p72ftzlFpTWI/0H9PdO+zbVri9K/bq497XVq77N2uytYUmoWwnfdL7DCWwP8xfMqV+lCCADPkiwzyOo//Zx63apT/Z6Svh+rXC+kKvNlGZ6zmpxx6s9Wner3lPT9WOV6IZlCOMBqcsap16061e8p6fuxyvVCOoUQgNEUQgBGUwgBGE0hHGA1OePU61ad6veU9P1Y5XohnUI4xL0bOunPVp3q95T0/VjleiGZH9QDMJoTIQCjlUqWSU9+SE/ESE8omdbebumJMafa2y39futyHbxSmRNhevJDeiJGekLJtPZ2S0+MOdXebkn3VlJCTvq+fadEIUxPfkhPxEhPKJnW3m7piTGn2tst/X7rch2cUKIQAsBVFEIARlMIARitRCFMT35IT8RITyiZ1t5u6Ykxp9rbLf1+63IdnFCiEN4KJD+kJ2KkJ5RMa2+39MSYU+3tlnRvJSXkpO/bdyTLADBamRMhAFyhVLLMqvRklGn97paeYCEpRHtXtLe73/Tr75XanQjTk1Gm9btbeoKFpBDtXdHe7n7Tr79Xa1UI05NRpvW7W3qChaQQ7V3R3u5+06+/E1oVQgB4lEIIwGgKIQCjtSqE6cko0/rdLT3BQlKI9q5ob3e/6dffCa0K4a1AMsq0fndLT7CQFKK9K9rb3W/69fdqkmUAGK3diRAAHlEqWSY9MWHa+Hb3e0qXBJBT/XZJKOkyvvR5JCpzIkxPTJg2vt39ntIlAeRUv10SSrqML30eqUoUwvTEhGnj293vKV0SQE712yWhpMv40ueRrEQhBICrKIQAjKYQAjBaiUKYnpgwbXy7+z2lSwLIqX67JJR0GV/6PJKVKIS3AokJ08a3u99TuiSAnOq3S0JJl/GlzyOVZBkARitzIgSAK7RMljnVr6SQrASa9PmuSt/f9HU5xTrXUeZE2CUBRFLIfV3WZbekvay4LqdY51pKFMIuCSCSQh7/byqty27p+5u+LqdY53pKFEIAuIpCCMBoCiEAo5UohF0SQCSFPP7fVFqX3dL3N31dTrHO9ZQohLdGCSCSQu7rsi67Je1lxXU5xTrXIlkGgNHKnAgB4Aqjk2XSExPSEyeoqUty0Kl+p92XE+Zb5kSYnmSyW3riBDV1SQ461e+0+3LKfEsUwvQkk93SEyeoqUty0Kl+p92Xk+ZbohACwFUUQgBGUwgBGK1EIUxPMtktPXGCmrokB53qd9p9OWm+JQrhrUCSyW7piRPU1CU56FS/0+7LKfOVLAPAaGVOhABwhVLJMqsmJCH8SQIIJ+2+37q8brf0RKDK92+7E+GUJIR3EkA4aff91uXPdktPBKp+/7YqhJOSEG4SQDhs9/3W5XW7pScCdbh/WxVCAHiUQgjAaAohAKO1KoSTkhBuEkA4bPf91uV1u6UnAnW4f1sVwtugJIR3EkA4aff91uXPdktPBKp+/0qWAWC0didCAHhEy2SZVemJCV36rZw4ccLp3149uj9d7qMuyS3p80009kSYnpjQpd/qiRN8rct91CW5JX2+qUYWwvTEhC79dkic4HNd7qMuyS3p8002shACwLvRnxFCgrR31xM+E4I/ORECMNrIQpiemNCl3w6JE3yuy33UJbklfb7JRhbCW4HEhC79Vk+c4Gtd7qMuyS3p800lWQYO++4zwr9v0Vd/pugRQXdjT4QAcPOt0R7SEzt2t+dbjVmmJbLslr5+EzgRFpee2LG7vSlJF1VMS2TZLX39pvAZYWGPfraU2u9qe6fme7X0B9DqCeWz/y79ekm/j06t3yROhACMphACMJpCCMBoCmFh6Ykdu9ublHRRwbRElt3S128ShbC49MSO3e1NSbqoYloiy27p6zeFb43CYVW/NQpdOBECMFqpZJkuCQxdki7gEaeSYDw3as73lcqcCLskMHRJuoBHnEqC8dyoOd9XK1EIv1rQP/9u9XW7+93d3u5+V53ql952X/enXrcq/bmRPt8TShRCALhKqc8Ige9V+mwGEjgRAjBaiULYJYGhS9IFPOJUEoznxnOv2z2+ZCUK4a1RAkOXpAt4xKkkGM+NmvN9NckycNjpb9Z5BDBdmRMhAFyh1LdGJSHUZJ3ppEvyzaoJ92+ZE6EkhJqsM50kpdxImtqnxGeE3y38o1PY3R73paxz2jva3Q+S0/OZYvV6PvW63VLu31cocyIEgCsohACMphACMFqJzwhvX/x79U+Hv7s97ktY51d/RvjqLxO4Zl9n9Xo+9brdEu7fVyhzIpSEUJN1ppOklBtJU/uUORHCTzkRAl8pcyIEgCuUSpbZbVqiQ/o8XrUfV6/rq3X7vVfa9XK19PtygrEnwmmJDunzSN8PXmPa9ZJ+X04xshB+teF//t3q63bb3W/6PNL3g9eYdr2k35eTjCyEAPBu9GeEcPvBZyVp75pXf3MG3OdECMBoIwvhV+/4//y71dfttrvf9Hmk7wevMe16Sb8vJxlZCG8DEx3S55G+H7zGtOsl/b6cQrIM7XX/zMwtDM8ZeyIEgFu1b42eSpxIT3hJT4jokgDCc9ITVLrcl+nPv0RlToSnEifSE17SEyK6JIDwnPQElS73ZfrzL1WJQngqcSI94SU9IaJLAgjPSU9Q6XJfpj//kpUohABwFYUQgNEUQgBGK/M7ws/+TXo1Z/Hq161Kb2+3U+v8kzFVkbK3jzh1nXa5z3f3m3hfnlTmRHgqcSI94SU9IaJLAgjPSU9Q6XJfpj//UpU5EcIuToTAn8qcCAHgCi2TZU61t5vxXcP/v2+vaQkl6cktXdb5lcqcCLskP6wyPiqYllCSntzSZZ1frUQh7JL8sMr4qGBaQkl6ckuXdT6hRCEEgKuU+owQrpD+maHPeOBaToQAjFaiEH71jvgn75Z3t7eb8VHB6nXQ5XrZPd9pz7VkJQrhrVHywyrjo4JpCSXpyS1d1vnVJMvAN179maFbEl6rzIkQAK5Q6lujXRIY0pMf0pNC0tdvmvT96JIE02WdE5U5EXZJYEhPfkhPCklfv2nS96NLEkyXdU5V4jPC7xb00Snsbi+931Wr45u+fo/e4I/+TvH0dbAqfR67r+dTz6Eu65yszIkQAK6gEAIwmkIIwGglPiO8ffHv0D8d/u720vtdtTq+yet39WdFKdfCioT9+Mru6/nUc6jLOqcqcyLsksCQnvyQnhSSvn7TpO9HlySYLuucqsyJEFI4EUIvZU6EAHAFyTIL7aW/blWXk0faPE73f1qX+3K39OeG58tvZU6E6YkOXRIs0nWZRxdd7svdkp4Rni/fK1EIv1rQnyz2anvpr1u1u71Tusyjiy735W7pzw3Pl49KFEIAuEqpzwghkc9aoDYnQgBGK1EIv3pH/JN3y6vtpb9u1e72Tukyjy663Je7pT83PF8+KlEIbwUSHbokWKTrMo8uutyXuyU9IzxfvidZBoDRypwIAeAKpb41KsHivvTxwSO63Efpz5f08b1SmROhBIv70scHj+hyH6U/X9LH92olCqEEi8fbThgfPKLLfZT+fEkf3wklCiEAXEUhBGA0hRCA0UoUQgkWj7edMD54RJf7KP35kj6+E0oUwpsEi0+ljw8e0eU+Sn++pI/v1STLADBamRMhAFxBssxCe11et2raPE61t1v6+Falz2Pa/Za+HzuUORF2SVZIT3RIGvMr5nGqvd3Sx7cqfR7T7rf0/dilRCHskqyQnugwbR6n2tstfXyr0ucx7X5L34+dShRCALiKQgjAaAohAKOVKIRdkhXSEx2mzeNUe7ulj29V+jym3W/p+7FTiUJ4a5SskJ7okDTmV8zjVHu7pY9vVfo8pt1v6fuxi2QZAEYrcyIEgCu0TJaZkITwpy7r0iURI30eXfpNTw7qcv1NUOZEKAnhvi7r0iURI30eXfpNTw7qcv1NUaIQSkK4r8u6dEnESJ9Hl37Tk4O6XH+TlCiEAHAVhRCA0RRCAEYrUQglIdzXZV26JGKkz6NLv+nJQV2uv0lKFMKbJIRPdVmXLokY6fPo0m96clCX628KyTIAjFbmRAgAVyiVLLPbtOSMVdPWJX2d0xNA0vd31bR15rexJ8JpyRmrpq1L+jqnJ4Ck7++qaevMv40shNOSM1ZNW5f0dU5PAEnf31XT1pmPRhZCAHinEAIwmkIIwGgjC+G05IxV09YlfZ3TE0DS93fVtHXmo5GF8DYwOWPVtHVJX+f0BJD0/V01bZ35N8kyAIw29kQIADfJMj0SSlalJ7ek95ueeJK+zrtNu9+s83XGngi7JJSsSk9uSe83PfEkfZ13m3a/WedrjSyEXRJKVqUnt6T3m554kr7Ou02736zz9UYWQgB4pxACMJpCCMBoIwthl4SSVenJLen9pieepK/zbtPuN+t8vZGF8NYooWRVenJLer/piSfp67zbtPvNOl9LsgwAo409EQLArVqyzLSkhi4JIKvS942aTj03pj2vKitzIpyW1NAlAWRV+r5R06nnxrTnVXUlCuG0pIYuCSCr0veNmk49N6Y9rzooUQgB4CoKIQCjKYQAjFaiEE5LauiSALIqfd+o6dRzY9rzqoMShfA2MKmhSwLIqvR9o6ZTz41pz6vqJMsAMFqZEyEAXKFUsgzPmZZ0kZ7M0yWhJL299H5Xpa9L+vp9xYlwiGlJF+nJPElpJBXXZbf0BJX0dUlfv+8ohANMS7pIT+bpklCS3l56v6vS1yV9/VYohACMphACMJpCCMBoCuEA05Iu0pN5uiSUpLeX3u+q9HVJX78VCuEQ05Iu0pN5ktJIKq7LbukJKunrkr5+35EsA8BopX9QX+WruT/lPQrA9UoWwu4F8N37PHcVxAkJEVdKX7/0ftOvq/SEnPT9razUP41OKYCfeWarPlu7v9tcfd006euX3m/6dbV7fOntpff7amW+LDO9CN6eWINJCRFXSF+/9H7Tr6v0hJz0/e2gTCEEgCuUKITd3n08w1oA7FWiEALAVeILoRPQR4+uyaSEiCukr196v+nXVXpCTvr+dhBfCNljSkLEVdLXL73f9OsqPSEnfX+ri//5hBPhfeHbBlBGyR/UPyqtaCjuADlaF8K0AvjufVypBTE9wSI9ySR9fKvS59EluWVVl/sjUcvPCN/e3kpsauI47xXnVxTs1X53/9lu6eNblT6P3f1O24/0+b5au0KYVlhWpIw5PcEiPckkfXyr0ufRJbllVZf7I1m7QggAj2hVCFNOVj9ReewAlbUqhADwqDaFsMOJ6vQc0hMs0pNM0se3Kn0eXZJbVnW5P5K1KYTskZ5gkZ5kkj6+Venz6JLcsqrL/ZGqTbJM+DSWTZsvwGmtf1D/qGe/Oqw4AdSjEG787cx7O6kF8VQSR5fX7TZtfOn7lj6PLuuXaPRnhL9+/brkB6RXtfuMU0kcXf5st2njS9+3pDEn/dmq6kk1YwvhqYfJCaeSOLq8brdp40vft/R5dFm/ZGMLIQDcphbCV75LqfKOCGCqkYUQAN6NK4QnTminT4Wnkji6vG63aeNL37f0eXRZv2TjCuFUp5I4uvzZbtPGl75vSWNO+rNV1ZNqxiXLnDqd7R5f+LYBlOFECMBokmX4ID3x5JRpCS+npK/z7va67Ef6PL7iRMi/pCeenDIt4eWU9HXe3V6X/Uifx3cUQv4nPfHklGkJL6ekr/Pu9rrsR/o8ViiEAIymEAIwmkIIwGgKIf+TnnhyyrSEl1PS13l3e132I30eKxRC/iU98eSUaQkvp6Sv8+72uuxH+jy+I1nmRSTLAGRyIgRgNMkyfHAqySQ9QSU9YaNyskcl6dfBqi7z2MGJkH85lWSSnqCSnrBRPdmjivTrYFWXeewyrhCeeDdT5R3UqSST9ASV9ISNDskeFaRfB6u6zGOncYUQAP40shC+8oRW5TQIMNXIQggA78YWwlM/0k12KskkPUElPWGjQ7JHBenXwaou89hpbCG8/bOZV2zoVe2+wqkkk/QElfSEjerJHlWkXwerusxjl3HJMjv6ShhD+LYBlOEH9X9QXADmUQj5oEvCyyldEjvsW9a+dUloSjT6M0I+6pLwckqXxA77lrVvSWlMSfu2S5tC2GFzTs+hS8LLKV0SO+zb/b9LTw5Kv3+TtSmEAPATrQph5XcplccOUFmrQggAj2pXCCuerFLG3CXh5ZQuiR327f7fpScHpd+/ydoVwts/hSWluHwlcZxdEl5O6ZLYYd+y9i0pjSlp33ZpkyzzlbQpdpwTQFUjflCfduoCIMeIQsh/pSeFnGovPSHH+LKkr0v6dZCo5WeEfJSeFHKqvfSEHOPLkr4u6ddBqvhCWOldxas8uibpSSGn2ktPyDG+LOnrkn4dJIsvhABwpRKF0KnwN2sBsFeJQggAVylTCJ2Efr4G6Ukhp9pLT8gxvizp65J+HSSL/0H9PVU+gN1l1xZ1+Zr27vbSv+ZufFnS1yX9OkhUshC+614QC28NQBmlf1CvUADwrDKfEQLAFRRCAEZTCAEYTSEEYDSFEIDRFEIARlMIARhNIQRgNIUQgNEUQgBGUwgBGE0hBGA0hRCA0RRCAEZTCAEYTSEEYDSFEIDRFEIARlMIARhNIQRgNIUQgNEUQgBGUwgBGE0hBGA0hRCA0RRCAEZTCAEYTSEEYDSFEIDRFEIARlMIARhNIQRgNIUQgNEUQgBGUwgBGE0hBGA0hRCA0RRCAEZTCAEY7f8B76TmAmbOw/YAAAAASUVORK5CYII="
      }
    ]
  }
]

The result:

image

It also got the same Error when I called this DEMO API hosting online from my local env in another project.

steveoon commented 1 month ago

The log form API hosting demo:

Error details:  AI_InvalidDataContentError: Invalid data content. Content string is not a base64-encoded media.
    at (node_modules/.pnpm/ai@3.3.5_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.37_typescript@5.5.4__zod@3.23.8/node_modules/ai/dist/index.mjs:744:0)
    at (node_modules/.pnpm/ai@3.3.5_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.37_typescript@5.5.4__zod@3.23.8/node_modules/ai/dist/index.mjs:3288:0)
    at (node_modules/.pnpm/ai@3.3.5_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.37_typescript@5.5.4__zod@3.23.8/node_modules/ai/dist/index.mjs:3330:0)
    at (lib/utils/index.ts:147:33)
    at (app/api/chat/route.ts:12:55)
    at (node_modules/.pnpm/next@14.2.5_@opentelemetry+api@1.9.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/next/dist/esm/server/future/route-modules/app-route/module.js:195:0)
    at (node_modules/.pnpm/next@14.2.5_@opentelemetry+api@1.9.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/next/dist/esm/server/future/route-modules/app-route/module.js:124:0)
    at (node_modules/.pnpm/next@14.2.5_@opentelemetry+api@1.9.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/next/dist/esm/server/future/route-modules/app-route/module.js:257:0)
    at (node_modules/.pnpm/next@14.2.5_@opentelemetry+api@1.9.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/next/dist/esm/server/web/edge-route-module-wrapper.js:81:0)
    at (node_modules/.pnpm/next@14.2.5_@opentelemetry+api@1.9.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/next/dist/esm/server/web/adapter.js:158:0) {
  name: 'AI_InvalidDataContentError',
  cause: TypeError: Illegal invocation: function called with incorrect `this` reference. See https://developers.cloudflare.com/workers/observability/errors/#illegal-invocation-errors for details.
    at (node_modules/.pnpm/@ai-sdk+provider-utils@1.0.9_zod@3.23.8/node_modules/@ai-sdk/provider-utils/dist/index.mjs:509:0)
    at (node_modules/.pnpm/ai@3.3.5_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.37_typescript@5.5.4__zod@3.23.8/node_modules/ai/dist/index.mjs:742:38)
    at (node_modules/.pnpm/ai@3.3.5_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.37_typescript@5.5.4__zod@3.23.8/node_modules/ai/dist/index.mjs:3288:0)
    at (node_modules/.pnpm/ai@3.3.5_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.37_typescript@5.5.4__zod@3.23.8/node_modules/ai/dist/index.mjs:3330:0)
    at (lib/utils/index.ts:147:33)
    at (app/api/chat/route.ts:12:55)
    at (node_modules/.pnpm/next@14.2.5_@opentelemetry+api@1.9.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/next/dist/esm/server/future/route-modules/app-route/module.js:195:0)
    at (node_modules/.pnpm/next@14.2.5_@opentelemetry+api@1.9.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/next/dist/esm/server/future/route-modules/app-route/module.js:124:0)
    at (node_modules/.pnpm/next@14.2.5_@opentelemetry+api@1.9.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/next/dist/esm/server/future/route-modules/app-route/module.js:257:0)
    at (node_modules/.pnpm/next@14.2.5_@opentelemetry+api@1.9.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/next/dist/esm/server/web/edge-route-module-wrapper.js:81:0),
  content: 'iVBORw0KGgoAAAANSUhEUgAABYQAAALwCAIAAACoXeZMAAAAAXNSR0IArs4c6QAAIABJREFUeJzs3XFYVPedL/7v1tnHTNZIpnMlZFIaqme1RUppwHHMwnKNkMGy5YYN/px6ebZBMWPEZjGBa3YZH+t12I2FKJsE40QqaR8fO1xx8dJ6nRNQl4VWMkI6ZdHG9GBNaaZG7+QUSjP69Nj+/vis33tyZhgQYWaQ9+vJk+dw5sw533NmnJnv53y+n++f/elPf2IAAAAAAAAAANHymVg3AAAAAAAAAADmFgQjAAAAAAAAACCqEIwAAAAAAAAAgKhCMAIAAAAAAAAAogrBCAAAAAAAAACIKgQjAAAAAAAAACCqEIwAAAAAAAAAgKhCMAIAAAAAAAAAogrBCAAAAAAAAACIKgQjAAAAAAAAACCqEIwAAAAAAAAAgKhCMAIAAAAAAAAAogrBCAAAAAAAAACIKgQjAAAAAAAAACCqEIwAAAAAAAAAgKhCMAIAAAAAAAAAogrBCAAAAAAAAACIKgQjAAAAAAAAACCqEIwAAAAAAAAAgKhCMAIAAAAAAAAAogrBCAAAAAAAAACIKgQjAAAAAAAAACCqEIwAAAAAAAAAgKhCMAIAAAAAAAAAogrBCAAAAAAAAACIKgQjAAAAAAAAACCqEIwAAAAAAAAAgKhCMAIAAAAAAAAAogrBCAAAAAAAAACIKgQjAAAAAAAAACCqEIwAAAAAAAAAgKhCMAIAAAAAAAAAogrBCAAAAAAAAACIKgQjAAAAAAAAACCqEIwAAAAAAAAAgKhCMAIAAAAAAAAAogrBCAAAAAAAAACIKl2sGxDG6Ojo1atXP/jgA0mSYt0WgDglCMKjjz6alJS0cOHCWLcFAAAAAADgzvzZn/70p1i34T+dO3fO5/P19vbGuiEAs4zFYsnIyFi1alWsGwIAAAAAADApcRGMGB0dfeONNy5evBjrhgDMYqmpqc899xwSJQAAAAAAIP7FPhhx7ty5gwcPhq63WCyCICxYsCD6TQKIc2NjY5IkhU0j2rJlC1IkAAAAAAAgzsU4GLF37151QkRqampxcTGGwQNMEhVYaWtr0/w72rFjRwxbBQAAAAAAEFksgxGanAjc0QWYMvxrAgAAAACAWSRmwYjR0dFvfetbtJyQkLBz585FixbFpCUA94br16/v2bNnZGSE/nzttdeQYQQAAAAAAPHpM7E68BtvvMGXEYkAuHuLFi3auXMn/1P9TwwAAAAAACCuxCYYce7cOT7EfcuWLYhEAEyLRYsWbdmyhZYvXrx47ty52LYHAAAAAAAgrNgEI3w+Hy2kpqZiZDvANFq1alVqaiot839oAAAAAAAAcSU2wQg+JWFxcXFMGgBwD+P/rMLO/QkAAAAAABBzMQhGjI6O8uWkpKToNwDg3qb+Z6X+5wYAAAAAABAnYhCMuHr1Kl9GtX+Aaaf+Z6X+5wYAAAAAABAnYhCMCAQCtGCxWKZ95zU1NVlZWR6PZ9r3HLXD+f3+rKysrKysadznPUNRFL/fL8vy1J7u8XiysrJqamqmt1VxiP/j+uCDD2LbEgAAAAAAgFAxm9ozLJvNlpWV1dLSEvrQ1q1bx+tGtrS0ZGVlWa3W6W0M9XuDweD07jaULMt+v3+mjzJb0AsdGouRZXnr1q0Wi6WoqCg/P99qtYZ9nzDGenp6rFZrNGNScUgQBFqQJCm2LQEAAAAAAAili3UDPmXNmjWSJLW1ta1fv169XlEUr9fLGBNFcffu3Trdp5rd1dXFGCspKZnexly5csVmszmdzoKCgvG2qaqqqqioMBgMd3Ogmpoar9fb19d3Nzu5N3g8HnqhNWRZttlsgUBAEISsrKyOjo5AIFBXVzc6Orp582a+maIoDQ0Nbrc7ik2OUwsWLIh1EyBKbt26NTQ0FAgEhoaGfve738W6OfeIhx9+ODMz02QyzZs3L9ZtAYA7wD8SMZ/UNBIE4Ytf/CI+EgFg2sVXMGL16tUul0uSpGAwqNfr+frBwUG+fOXKFX7Xl6niFCtWrJjexkzmlrLBYLjLSARjLGz3ew6SZdnhcIR96LnnngsEAjabraqqijFWUVHhdDpFUXS5XKtXr+bvh8LCwkAgYDabFy9ejJAE3PNu3bp15syZI0eOxLoh96a2tjbGWGpq6oYNG5KTk2PdHACYwK1bt37wgx90dHTEuiH3ID45l8Vi+Zu/+Rt8JALAdImvYATvVV66dCkjI4OvP3/+PGPMarWKotjf368ORvA4RVpamnpXwWDw0qVL58+fHxkZ2bBhg8lk0hyLb3DlypWcnJyVK1fysEIwGOzq6nrrrbcYY93d3Yyx9PT00D0wxnw+39WrV9WPKorS29s7NjbW3d2dkpKyYsWKtLQ0TSqH+ul0aowxGlOgycKQJKm/v39gYGDdunVh99PT0/Phhx8ODAwYDIa0tDT1WYTl9/sHBgbS09MTExMHBwc7OzsTEhJ4I2VZvnDhwqlTp3JycnJzc9XxIDq1K1euSJJEp6aOAvBLsWDBguzsbPWz6LwEQVBvHBaNwamurq6rq1Ovl2WZAkOVlZW0Rq/X19bWpqSkuFyus2fP8j0HAgHKZLmjARqSJNH+8/LyxnulAOLNrVu36uvrL168GOuG3OMuXrzocDhqamqWLl0a67YAwLhu3rzZ0NCAj8SZ1tvb29vbW19fv2jRoli3BQDuBXHX9bLZbG63u7OzUx2MaG1tZYxVVVWJoqgZxMHjFOpu5MjIyIsvvsgzDtxutyAIzc3NvHft9/uLior49qIoMsaMRiMtqG/Ri6IoiqLT6QwbjDh27Jj6UUmSKioqeIVOxpjL5TIajSdPngzby6Wn0zIdUR2MOHTokMvl4s0wGo179+7ll0VRlF27dvGnc01NTepLpzEwMOBwOGw22+XLl/n1cblcNptt06ZNNBSCX5CGhgYeWVAUpbS0VJ0tQm1rb2+nc09MTCwvL1evYYz19PQ4HA6j0XjixInxmkRaWlq8Xq/NZsvJydEEI9555x0

It seems to point to the same reason:

Error Type

Source of the Error

Cause of the Error

Key Points from the Log

  1. Invalid Data Content: The log indicates that the data being processed is not in the expected base64-encoded format.
  2. Illegal Invocation: This suggests that the code encountered an issue with the context of a function call, potentially due to asynchronous code or execution environment constraints.

Error Locations

Potential Root Causes

  1. Data Format Mismatch: The data passed to the AI processing tool might not meet the expected format, such as a string that isn’t properly base64 encoded.
  2. Context or Environment Issues: The function may be called in an incorrect context, possibly due to binding issues or limitations imposed by the execution environment (e.g., Cloudflare Workers).
steveoon commented 1 month ago

This is complete Cloudflare worker testing log:

{
  "truncated": false,
  "outcome": "ok",
  "scriptVersion": {
    "id": "40c59538-4294-41a5-a300-e048b1c2f82b"
  },
  "scriptName": "polished-math-96b3",
  "diagnosticsChannelEvents": [],
  "exceptions": [],
  "logs": [
    {
      "message": [
        "Testing global atob:"
      ],
      "level": "log",
      "timestamp": 1723384322695
    },
    {
      "message": [
        "test"
      ],
      "level": "log",
      "timestamp": 1723384322695
    },
    {
      "message": [
        "Testing local atob:"
      ],
      "level": "log",
      "timestamp": 1723384322695
    },
    {
      "message": [
        "test"
      ],
      "level": "log",
      "timestamp": 1723384322695
    },
    {
      "message": [
        "Testing convertBase64ToUint8Array:"
      ],
      "level": "log",
      "timestamp": 1723384322695
    },
    {
      "message": [
        "116,101,115,116"
      ],
      "level": "log",
      "timestamp": 1723384322695
    }
  ],
  "eventTimestamp": 1723384322695,
  "event": {
    "request": {
      "url": "https://polished-math-96b3.rsw87621.workers.dev/favicon.ico",
      "method": "GET",
      "headers": {
        "accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
        "accept-encoding": "gzip, br",
        "accept-language": "zh-CN,zh;q=0.9,en-US;q=0.8,en;q=0.7,ru;q=0.6,zh-TW;q=0.5",
        "cf-connecting-ip": "xxx.xxx.xxx.xxx",
        "cf-ipcountry": "US",
        "cf-ray": "xxx",
        "cf-visitor": "{\"scheme\":\"https\"}",
        "connection": "Keep-Alive",
        "cookie": "xxx",
        "host": "polished-math-96b3.rsw87621.workers.dev",
        "priority": "u=1, i",
        "referer": "https://polished-math-96b3.rsw87621.workers.dev/",
        "sec-ch-ua": "\"Not)A;Brand\";v=\"99\", \"Google Chrome\";v=\"127\", \"Chromium\";v=\"127\"",
        "sec-ch-ua-mobile": "?0",
        "sec-ch-ua-platform": "\"macOS\"",
        "sec-fetch-dest": "image",
        "sec-fetch-mode": "no-cors",
        "sec-fetch-site": "same-origin",
        "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36",
        "x-forwarded-proto": "https",
        "x-real-ip": "xxx.xxx.xxx.xxx"
      },
      "cf": {
        "clientTcpRtt": 1,
        "longitude": "-97.82200",
        "httpProtocol": "HTTP/2",
        "tlsCipher": "AEAD-AES128-GCM-SHA256",
        "continent": "NA",
        "asn": 21859,
        "clientAcceptEncoding": "gzip, deflate, br, zstd",
        "country": "US",
        "tlsClientAuth": {
          "certIssuerDNLegacy": "",
          "certIssuerSKI": "",
          "certSubjectDNRFC2253": "",
          "certSubjectDNLegacy": "",
          "certFingerprintSHA256": "",
          "certNotBefore": "",
          "certSKI": "",
          "certSerial": "",
          "certIssuerDN": "",
          "certVerified": "NONE",
          "certNotAfter": "",
          "certSubjectDN": "",
          "certPresented": "0",
          "certRevoked": "0",
          "certIssuerSerial": "",
          "certIssuerDNRFC2253": "",
          "certFingerprintSHA1": ""
        },
        "tlsExportedAuthenticator": {
          "clientFinished": "xxx",
          "clientHandshake": "xxx",
          "serverHandshake": "xxx",
          "serverFinished": "xxx"
        },
        "tlsVersion": "TLSv1.3",
        "colo": "LAX",
        "timezone": "America/Chicago",
        "verifiedBotCategory": "",
        "edgeRequestKeepAliveStatus": 1,
        "tlsClientRandom": "xxx",
        "tlsClientExtensionsSha1": "xxx",
        "tlsClientHelloLength": "2075",
        "asOrganization": "Zenlayer",
        "requestPriority": "weight=220;exclusive=1",
        "latitude": "37.75100"
      }
    },
    "response": {
      "status": 200
    }
  },
  "id": 1
}
steveoon commented 1 month ago

Hi,@lgrammel I try to copy convertToCoreMessageslogic and test if this issue occurs inside the Function:

function base64ToUint8Array(base64: string): Uint8Array {
  try {
    const parts = base64.split(',');
    if (parts.length !== 2) {
      throw new Error('Invalid base64 string format');
    }
    const binaryString = atob(parts[1]);
    const bytes = new Uint8Array(binaryString.length);
    for (let i = 0; i < binaryString.length; i++) {
      bytes[i] = binaryString.charCodeAt(i);
    }
    return bytes;
  } catch (error) {
    console.error('Error in base64ToUint8Array:', error);
    throw new Error(
      `Failed to convert base64 to Uint8Array: ${(error as Error).message}`
    );
  }
}

export function copyConvertToCoreMessages(
  messages: InputMessages
): CoreMessage[] {
  if (!Array.isArray(messages)) {
    throw new TypeError('Input must be an array of messages');
  }

  return messages.map((message, index): CoreMessage => {
    try {
      if (typeof message !== 'object' || message === null) {
        throw new TypeError(
          `Invalid message at index ${index}: message must be an object`
        );
      }

      if (!('role' in message) || typeof message.role !== 'string') {
        throw new TypeError(
          `Invalid message at index ${index}: message must have a 'role' property of type string`
        );
      }

      switch (message.role) {
        case 'system':
        case 'assistant':
          return {
            role: message.role,
            content: message.content ?? '',
          };
        case 'user':
          if (
            message.experimental_attachments &&
            Array.isArray(message.experimental_attachments) &&
            message.experimental_attachments.length > 0
          ) {
            const content: Array<TextPart | ImagePart> = [
              { type: 'text', text: message.content ?? '' },
            ];

            message.experimental_attachments.forEach(
              (attachment: Attachment, attachmentIndex: number) => {
                if (attachment?.contentType?.startsWith('image/')) {
                  try {
                    content.push({
                      type: 'image',
                      image: base64ToUint8Array(attachment.url),
                      mimeType: attachment.contentType,
                    });
                  } catch (error) {
                    console.error(
                      `Error processing attachment ${attachmentIndex} in message ${index}:`,
                      error
                    );
                    // Optionally, you can choose to skip this attachment or throw an error
                  }
                }
              }
            );

            return {
              role: 'user',
              content: content,
            };
          } else {
            return {
              role: 'user',
              content: message.content ?? '',
            };
          }
        default:
          throw new Error(`Unsupported message role: ${message.role}`);
      }
    } catch (error) {
      console.error(`Error processing message at index ${index}:`, error);
      throw new Error(
        `Failed to convert message at index ${index}: ${
          (error as Error).message
        }`
      );
    }
  });
}

To run this code all works fine on Local env. but Edge Runtime fail with those Error logs:

Error in aiChat: TypeError: Illegal invocation: function called with incorrect `this` reference. See https://developers.cloudflare.com/workers/observability/errors/#illegal-invocation-errors for details.
    at (node_modules/.pnpm/@ai-sdk+provider-utils@1.0.9_zod@3.23.6/node_modules/@ai-sdk/provider-utils/dist/index.mjs:517:0)
    at (node_modules/.pnpm/@ai-sdk+openai@0.0.44_zod@3.23.6/node_modules/@ai-sdk/openai/dist/index.mjs:52:169)
    at (node_modules/.pnpm/@ai-sdk+openai@0.0.44_zod@3.23.6/node_modules/@ai-sdk/openai/dist/index.mjs:42:0)
    at (node_modules/.pnpm/@ai-sdk+openai@0.0.44_zod@3.23.6/node_modules/@ai-sdk/openai/dist/index.mjs:257:0)
    at (node_modules/.pnpm/@ai-sdk+openai@0.0.44_zod@3.23.6/node_modules/@ai-sdk/openai/dist/index.mjs:383:29)
    at (node_modules/.pnpm/ai@3.3.4_openai@4.42.0_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.36_typescript@5.3.3__zod@3.23.6/node_modules/ai/dist/index.mjs:2870:0)
    at (node_modules/.pnpm/ai@3.3.4_openai@4.42.0_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.36_typescript@5.3.3__zod@3.23.6/node_modules/ai/dist/index.mjs:251:0)
    at (node_modules/.pnpm/ai@3.3.4_openai@4.42.0_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.36_typescript@5.3.3__zod@3.23.6/node_modules/ai/dist/index.mjs:180:0)
    at (node_modules/.pnpm/ai@3.3.4_openai@4.42.0_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.36_typescript@5.3.3__zod@3.23.6/node_modules/ai/dist/index.mjs:249:0)

Even try to not use atob also did not work on Edge:

function base64ToUint8Array(base64: string): Uint8Array {
  const base64Chars =
    'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';

  function decode(encoded: string): string {
    let decoded = '';
    encoded = encoded.replace(/[^A-Za-z0-9+/]/g, '');
    for (let i = 0; i < encoded.length; i += 4) {
      const v =
        (base64Chars.indexOf(encoded[i]) << 18) |
        (base64Chars.indexOf(encoded[i + 1]) << 12) |
        (base64Chars.indexOf(encoded[i + 2]) << 6) |
        base64Chars.indexOf(encoded[i + 3]);
      decoded += String.fromCharCode((v >> 16) & 255, (v >> 8) & 255, v & 255);
    }
    return decoded;
  }

  try {
    const parts = base64.split(',');
    if (parts.length !== 2) {
      throw new Error('Invalid base64 string format');
    }
    const binaryString = decode(parts[1]);
    const bytes = new Uint8Array(binaryString.length);
    for (let i = 0; i < binaryString.length; i++) {
      bytes[i] = binaryString.charCodeAt(i);
    }
    return bytes;
  } catch (error) {
    console.error('Error in base64ToUint8Array:', error);
    throw new Error(
      `Failed to convert base64 to Uint8Array: ${(error as Error).message}`
    );
  }
}

and got same Error logs:

Error in aiChat: TypeError: Illegal invocation: function called with incorrect `this` reference. See https://developers.cloudflare.com/workers/observability/errors/#illegal-invocation-errors for details.
    at (node_modules/.pnpm/@ai-sdk+provider-utils@1.0.9_zod@3.23.6/node_modules/@ai-sdk/provider-utils/dist/index.mjs:517:0)
    at (node_modules/.pnpm/@ai-sdk+anthropic@0.0.39_zod@3.23.6/node_modules/@ai-sdk/anthropic/dist/index.mjs:76:55)
    at (node_modules/.pnpm/@ai-sdk+anthropic@0.0.39_zod@3.23.6/node_modules/@ai-sdk/anthropic/dist/index.mjs:263:0)
    at (node_modules/.pnpm/@ai-sdk+anthropic@0.0.39_zod@3.23.6/node_modules/@ai-sdk/anthropic/dist/index.mjs:356:29)
    at (node_modules/.pnpm/ai@3.3.4_openai@4.42.0_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.36_typescript@5.3.3__zod@3.23.6/node_modules/ai/dist/index.mjs:2870:0)
    at (node_modules/.pnpm/ai@3.3.4_openai@4.42.0_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.36_typescript@5.3.3__zod@3.23.6/node_modules/ai/dist/index.mjs:251:0)
    at (node_modules/.pnpm/ai@3.3.4_openai@4.42.0_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.36_typescript@5.3.3__zod@3.23.6/node_modules/ai/dist/index.mjs:180:0)
    at (node_modules/.pnpm/ai@3.3.4_openai@4.42.0_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.36_typescript@5.3.3__zod@3.23.6/node_modules/ai/dist/index.mjs:249:0)
    at (node_modules/.pnpm/ai@3.3.4_openai@4.42.0_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.36_typescript@5.3.3__zod@3.23.6/node_modules/ai/dist/index.mjs:2842:0)
    at (node_modules/.pnpm/ai@3.3.4_openai@4.42.0_react@18.3.1_sswr@2.1.0_svelte@4.2.18__svelte@4.2.18_vue@3.4.36_typescript@5.3.3__zod@3.23.6/node_modules/ai/dist/index.mjs:83:0)

I suspect that some code in @ai-sdk+provider-utils causes this issue?

lgrammel commented 1 month ago

@steveoOn 

       "url": "data:image/jpeg;base64,/9j/2wBDAAYEBQYFBAYGBQYHBwYIChAKCgkJChQODwwQFxQYGBcUFhYaHSUfGhsjHBYWICwgIyYnK..."

This is not a valid image base64 (... at end). Where the 3 dots in the data that you used or did you just remove it in this issue report?

steveoon commented 1 month ago

@steveoOn

       "url": "data:image/jpeg;base64,/9j/2wBDAAYEBQYFBAYGBQYHBwYIChAKCgkJChQODwwQFxQYGBcUFhYaHSUfGhsjHBYWICwgIyYnK..."

This is not a valid image base64 (... at end). Where the 3 dots in the data that you used or did you just remove it in this issue report? @lgrammel 

[
{
"id": "1",
"role": "user",
"content": "1+4=?"
},
{
"id": "2",
"role": "assistant",
"content": "1 + 4 equals 5."
},
{
"id": "3",
"role": "user",
"content": "能看到图片吗",
"experimental_attachments": [
{
"name": "IMG_6957.jpg",
"contentType": "image/jpeg",
"url": "data:image/jpeg;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/wcAAgEBAUKFVv8AAAAASUVORK5CYII="
}
]
}
]

Try this test data, remove the ... at end.

The base64 image data not include ... at the end while it transfers through my code context.

you can also try this with the API endpoint.

 curl -X POST https://multi-modal-chatbot-demo.vercel.app/api/chat \
-H "Content-Type: application/json" \
-d '{
  "messages": [
  {
    "id": "1",
    "role": "user",
    "content": "1+4=?"
  },
  {
    "id": "2",
    "role": "assistant",
    "content": "1 + 4 equals 5."
  },
  {
    "id": "3",
    "role": "user",
    "content": "能看到图片吗",
    "experimental_attachments": [
      {
        "name": "IMG_6957.jpg",
        "contentType": "image/jpeg",
        "url": "data:image/jpeg;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/wcAAgEBAUKFVv8AAAAASUVORK5CYII="
      }
    ]
  }
]
}'
steveoon commented 1 month ago

@lgrammel

image
lgrammel commented 1 month ago

Just reproduced and verified myself. My initial assessment was correct. The bug is indeed in globalThis.atob

lgrammel commented 1 month ago

Proposed fix: https://github.com/vercel/ai/pull/2635

steveoon commented 1 month ago

Just reproduced and verified myself. My initial assessment was correct. The bug is indeed in globalThis.atob

I'm curious why I didn't catch this issue when testing on Cloudflare Worker before, it's weird 🤔

lgrammel commented 1 month ago

@steveoOn please try with the latest versions of the ai sdk libs (just released)

lgrammel commented 1 month ago

(my reproduction / fix was for a vercel edge deployment fwiw)

steveoon commented 1 month ago

The ai@3.3.6 test passed, and Edge Runtime works as expected.🎉🎉