nuxt-hub / core

Build full-stack applications with Nuxt on CloudFlare, with zero configuration.
https://hub.nuxt.com
Apache License 2.0
812 stars 36 forks source link

[CRITICAL NUXTHUB LIST BUG]: hubBlob.list() returns incorrect values, after uploading 150+ images #190

Open FutureExcited opened 5 days ago

FutureExcited commented 5 days ago

Describe the bug

Ater uploading about 120-150 images (or more, if they're small), the hubBlob actually STOPS showing almost 70% of newly uploaded images. And when it DOES show it, the total shown amount of Blobs stays THE SAME. Meaning, it shows new blobs by not showing the previous ones??

Also, in local production for some reason the total amount of shown blobs is ALWAYS capped at a 100.

Do note, that it works this way with specifying limit = 1000, and without specifying limit at all.

At first, I thought the problem was with uploading files, but no. The files are successfully uploaded to cloudflare, but AREN'T shown with hubBlob.list(). On R2 they're uploaded completely correctly: links work, all of the metadata is there.

Steps to reproduce

1) Check if it caps at a 100 on localhost - may be the fastest way. 2) On deployed website, upload 120-150 normal sized images. 3) Afterwards, try to upload more and observe hubBlob.list() behaviour.

DO NOTE I'm uploading ALL of the images/videos with customMetadata property. Tested on both nuxthub 0.6.x, and the newest 0.7.0 version.

Expected behavior

To see the exact same amount of files. as are uploaded to cloudflare r2.

I'm currently using cloudflare workers for my bucket instead of hubBlob, and everything works perfectly. In fact, this is the exact copy of expeted behaviour of hubBlob, but without the bug of not seeing newly added images.

Take a look at the code:


export interface Env {
  MY_BUCKET: R2Bucket;
}

interface BlobObject {
  pathname: string;
  contentType: string | undefined;
  size: number;
  uploadedAt: string;
  customMetadata?: Record<string, string> | undefined;
}

export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    const url = new URL(request.url);
    const prefix = url.searchParams.get('prefix') || '';

    const options = {
      prefix: prefix,
      limit: 1000,
      include: ['httpMetadata', 'customMetadata'],
    };

    let allObjects = [];
    let cursor: string | undefined;

    do {
      const listed = await env.MY_BUCKET.list({ ...options, cursor });
      allObjects.push(...listed.objects);
      cursor = listed.truncated ? listed.cursor : undefined;
    } while (cursor);

    const blobs: BlobObject[] = allObjects.map(obj => ({
      pathname: obj.key,
      contentType: obj.httpMetadata?.contentType,
      size: obj.size,
      uploadedAt: obj.uploaded.toISOString(),
      customMetadata: obj.customMetadata,
    }));

    // Sort blobs by upload date (newest first)
    blobs.sort((a, b) => new Date(b.uploadedAt).getTime() - new Date(a.uploadedAt).getTime());

    return new Response(JSON.stringify(blobs, null, 2), {
      headers: { 'Content-Type': 'application/json' },
    });
  },
};
Atinux commented 3 days ago

Hi @FutureExcited

We will look into it, this is weird as it works properly with customMetadata on https://draw.nuxt.dev