hubBlob signed url uploading from client

[alexcroox]

Usually when uploading files from the client I use signed urls to upload directly to the storage, removing the need/cost/resource of sending the entire file through the API. It also means you can upload large files without hitting API payload size limits etc.

Is that in the scope of this project or shall I continue using S3 SDK with signed urls to upload direct to R2?

Thanks!

[atinux]

Good question! You are referring to https://developers.cloudflare.com/r2/api/s3/presigned-urls/ ?

From what I see, we will need to have a way to get back the accountId & bucketName right?

If so, I guess this would be possible as long as you use remote storage locally or that your project is deployed with NuxtHub as we could send back the accountId and bucketName from the project bindings.

How would you like to see such usage? Something like this?

const { accountId, bucketName } = await hubBlob().infos()

This way you have everything you need to create the signed url using aws4fetch I guess.

[alexcroox]

Yes or even better if there was hubBlob().generatePresignedUrl() 😁

Looks like workers provide an example that doesn’t require adding AWS sdk as a dependency to offer this:

https://developers.cloudflare.com/r2/api/s3/presigned-urls/#presigned-url-alternative-with-workers

[atinux]

The example uses aws4fetch but seems quite minimal.

What options would you see in the signedUrl method?

Do you have an example on how you use it on client-side?

[alexcroox]

api side signing function example (there is a whole bunch of validation I do but this is the final function):

import { PutObjectCommand, S3Client } from '@aws-sdk/client-s3'
import { getSignedUrl } from '@aws-sdk/s3-request-presigner'

export async function getPresignedUploadUrl(key: string, mimeType: string, expiresSeconds: number = 600) {
  const { media } = useRuntimeConfig()

  const client = createClient()

  return await getSignedUrl(
    client,
    new PutObjectCommand({ Bucket: media.bucket, Key: key, ContentType: mimeType }),
    {
      expiresIn: expiresSeconds
    }
  )
}

client side (composable)

export function useFileService() {
  const { apiStatus, api } = useApi()

  async function getUploadUrl(file: File) {
    return await api<{ signedUploadUrl: string; fileId: string; path: string }>(`/file/v1/signed-url`, {
      method: 'POST',
      body: {
        name: file.name,
        sizeBytes: file.size,
        mimeType: file.type
      }
    })
  }

  async function uploadFile(signedUploadUrl: string, file: File) {
    return await $fetch(signedUploadUrl, {
      method: 'PUT',
      body: file,
      headers: {
        'Content-Type': file.type
      }
    })
  }

  return { apiStatus, getUploadUrl, uploadFile }
}

client side (component):

<script setup lang="ts">
const { apiStatus, getUploadUrl, uploadFile } = useFileService()

async function onFileChange(e: Event) {
  const input = e.target as HTMLInputElement

  if (!input.files?.length) {
    return
  }

  try {
    // Get a signed URL to upload the file directly to R2
    const { signedUploadUrl, fileId, path } = await getUploadUrl(input.files[0])

    if (!signedUploadUrl) {
      throw new Error(t('thereWasIssueUploadingFile'))
    }

    // Upload directly to R2
    await uploadFile(signedUploadUrl, input.files[0])

    model.value.avatar = { id: fileId, path }

    $log.debug('Profile form: Avatar uploaded')
  } catch (error: any) {
    $log.error('Profile form: Error getting upload URL for avatar', error)

    notifyError({
      title: error?.message || t('thereWasIssueUploadingFile')
    })
  }
}

[atinux]

Having progress with:

const { accountId, bucketName, accessKeyId, secretAccessKey } = await hubBlob().credentials()

It will generate temporary credentials using https://developers.cloudflare.com/api/operations/r2-create-temp-access-credentials

But I am struggling with CORS or SignatureDoesNotMatch when trying to use it with curl, any hints?

BTW, did you see that we support multipart uploads to bypass the 100mb limit?

[alexcroox]

Nice! Signature not matching errors are often because the file you signed doesn’t exactly match the file you are uploading. Usually when you create a signed url you tell S3/R2 the name, path, file size and mime type. If those don’t exactly match when you come to upload then that’s when I’ve seen the error before.

More often than not for me it’s been the file size.

[atinux]

I got something working but seems that the temporary credentials are not yet working, ideally I would like to avoid having to create new R2 token just for this!

[atinux]

Ohhhhhh yeeeaahh! I just made it working 🚀

It works only in development with remote storage or in production:

// api/blob/signed/[...pathname.get.ts]
import { AwsClient } from 'aws4fetch'

/*
** Create a signed url to upload a file to R2
** https://developers.cloudflare.com/r2/api/s3/presigned-urls/#presigned-url-alternative-with-workers
*/
export default eventHandler(async (event) => {
  const { pathname } = await getValidatedRouterParams(event, z.object({
    pathname: z.string().min(1)
  }).parse)
  const { accountId, bucketName, ...credentials } = await hubBlob().createCredentials()
  const client = new AwsClient(credentials)
  const endpoint = new URL(pathname, `https://${bucketName}.${accountId}.r2.cloudflarestorage.com`)

  const { url } = await client.sign(endpoint, {
    method: 'PUT',
    aws: { signQuery: true }
  })
  return url
})

Then on client-side:

<script setup lang="ts">
async function upload(file: File) {
  await $fetch(`/api/blob/signed/${file.name}`)
    .then(url => {
      return $fetch(url, {
        method: 'PUT',
        body: file
       })
    })
    .then(() => {
      window.alert(`File ${file.name} uploaded.`)
    })
    .catch((err) => {
      window.alert(`Failed to upload ${file.name}.`)
    })
}
</script>

Will open a PR and add the docs!

[alexcroox]

Great! I guess it makes sense to keep doing it manually with AWS sdk external to NuxtHub so you don’t have to bring in a new dependency to core.

[atinux]

Exactly, and I believe this example is simple enough and we keep this zero-config approach as not need to store any access token or other.

Currently improving so you can directly limit the temporary credentials generated:

const credentials = await hubBlob().createCredentials({
  // can only read or write specific object(s)
  permission: 'object-read-write',
  // valid only 10 minutes
  ttl: 600,
  // can only create this object, a new credential will have to be created for a new signed url
  pathnames: [pathname]
})
/*
{
  accountId: '...',
  bucketName: '...',
  accessKeyId: '...',
  secretAccessKey: '...',
  sessionToken: '...'
}
*/

[alexcroox]

Yeah love the generate credentials feature of this. Isn’t there a limit of 50 tokens/api keys per account though? Won’t this be creating a new token with every user upload? Fine if they are short lived but wondering at scale when you have 50+ users a minute uploading something.

[atinux]

Where did you see the rate limit about the temporary tokens per account?

You can leverage the cache storage with maxAge (respecting the same ttl) using defineCachedFunction if you want to re-use credentials, but this is already over-optimizing IMO, but this is possible!

[atinux]

Need to update to use aws4fetch as I hit the same error as https://github.com/aws/aws-sdk-js-v3/discussions/6284

[alexcroox]

https://developers.cloudflare.com/cloudflare-one/account-limits/

Service tokens count 50

ah aws4fetch looks great. Would be a relief to have a simpler package for just signing so I don’t get 7 renovate bot alerts a week for AWS SDK updates 😅

[atinux]

You are reffering to this endpoint which is not the same: https://developers.cloudflare.com/api-next/resources/zero_trust/subresources/access/subresources/service_tokens/

I did not find any rate limit documented for https://developers.cloudflare.com/api/operations/r2-create-temp-access-credentials

I updated the example above using aws4fetch