aws-amplify / amplify-js

A declarative JavaScript library for application development using cloud services.
https://docs.amplify.aws/lib/q/platform/js
Apache License 2.0
9.41k stars 2.11k forks source link

Storage.uploadData onProgress callback not functioning at all #13221

Closed ChristopherGabba closed 4 months ago

ChristopherGabba commented 5 months ago

Before opening, please confirm:

JavaScript Framework

React Native

Amplify APIs

Storage

Amplify Version

v6

Amplify Categories

storage

Backend

Amplify Gen 2 (Preview)

Environment information

``` System: OS: macOS 14.2.1 CPU: (10) arm64 Apple M2 Pro Memory: 917.92 MB / 16.00 GB Shell: 5.9 - /bin/zsh Binaries: Node: 20.7.0 - /opt/homebrew/bin/node npm: 10.1.0 - /opt/homebrew/bin/npm Watchman: 2023.09.04.00 - /opt/homebrew/bin/watchman Browsers: Safari: 17.2.1 npmPackages: @aws-amplify/react-native: ^1.0.12 => 1.0.17 @babel/core: ^7.20.0 => 7.23.9 @babel/plugin-proposal-export-namespace-from: ^7.18.9 => 7.18.9 @babel/plugin-transform-runtime: ^7.23.9 => 7.23.9 @babel/preset-env: ^7.20.0 => 7.23.9 @babel/runtime: ^7.20.0 => 7.23.9 @config-plugins/ffmpeg-kit-react-native: ^7.0.0 => 7.0.0 @expo-google-fonts/m-plus-1p: ^0.2.3 => 0.2.3 @expo-google-fonts/montserrat: ^0.2.3 => 0.2.3 @expo/config-plugins: ^7.8.0 => 7.8.0 (7.8.4, 5.0.4) @expo/metro-config: ~0.17.1 => 0.17.6 @gorhom/bottom-sheet: ^4.4.7 => 4.6.1 @likashefqet/react-native-image-zoom: ^2.1.1 => 2.2.0 @react-native-async-storage/async-storage: 1.21.0 => 1.21.0 @react-native-clipboard/clipboard: ^1.11.2 => 1.13.2 @react-native-community/cli-platform-ios: ^8.0.2 => 8.0.6 (12.3.6) @react-native-community/netinfo: 11.1.0 => 11.1.0 @react-navigation/bottom-tabs: ^6.3.2 => 6.5.12 @react-navigation/native: ^6.0.2 => 6.1.10 @react-navigation/native-stack: ^6.0.2 => 6.9.18 @rnx-kit/metro-config: ^1.3.5 => 1.3.14 @rnx-kit/metro-resolver-symlinks: ^0.1.26 => 0.1.35 @sentry/react-native: 5.19.1 => 5.19.1 @shopify/flash-list: 1.6.3 => 1.6.3 @types/i18n-js: 3.8.2 => 3.8.2 @types/jest: ^29.2.1 => 29.5.12 @types/lodash.filter: ^4.6.9 => 4.6.9 @types/react: ~18.2.14 => 18.2.57 @types/react-test-renderer: ^18.0.0 => 18.0.7 @typescript-eslint/eslint-plugin: ^5.59.0 => 5.62.0 @typescript-eslint/parser: ^5.59.0 => 5.62.0 HelloWorld: 0.0.1 apisauce: 2.1.5 => 2.1.5 aws-amplify: ^6.0.20 => 6.0.20 aws-amplify/adapter-core: undefined () aws-amplify/analytics: undefined () aws-amplify/analytics/kinesis: undefined () aws-amplify/analytics/kinesis-firehose: undefined () aws-amplify/analytics/personalize: undefined () aws-amplify/analytics/pinpoint: undefined () aws-amplify/api: undefined () aws-amplify/api/server: undefined () aws-amplify/auth: undefined () aws-amplify/auth/cognito: undefined () aws-amplify/auth/cognito/server: undefined () aws-amplify/auth/enable-oauth-listener: undefined () aws-amplify/auth/server: undefined () aws-amplify/datastore: undefined () aws-amplify/in-app-messaging: undefined () aws-amplify/in-app-messaging/pinpoint: undefined () aws-amplify/push-notifications: undefined () aws-amplify/push-notifications/pinpoint: undefined () aws-amplify/storage: undefined () aws-amplify/storage/s3: undefined () aws-amplify/storage/s3/server: undefined () aws-amplify/storage/server: undefined () aws-amplify/utils: undefined () axios: ^1.5.0 => 1.6.7 (0.21.4) babel-jest: ^29.2.1 => 29.7.0 babel-loader: 8.2.5 => 8.2.5 babel-plugin-root-import: ^6.6.0 => 6.6.0 cheerio: ^1.0.0-rc.12 => 1.0.0-rc.12 core-util-is: 1.0.1 date-fns: ^2.29.2 => 2.30.0 eslint: 8.17.0 => 8.17.0 eslint-config-prettier: 8.5.0 => 8.5.0 eslint-config-standard: 17.0.0 => 17.0.0 eslint-plugin-import: 2.26.0 => 2.26.0 eslint-plugin-n: ^15.0.0 => 15.7.0 eslint-plugin-node: 11.1.0 => 11.1.0 eslint-plugin-promise: 6.0.0 => 6.0.0 eslint-plugin-react: 7.30.0 => 7.30.0 eslint-plugin-react-native: 4.0.0 => 4.0.0 expo: ~50.0.14 => 50.0.14 expo-application: ~5.8.3 => 5.8.3 expo-av: ^13.10.5 => 13.10.5 expo-blur: ~12.9.2 => 12.9.2 expo-build-properties: ~0.11.1 => 0.11.1 expo-clipboard: ~5.0.1 => 5.0.1 expo-config-plugin-ios-share-extension: ^0.0.4 => 0.0.4 expo-constants: ~15.4.5 => 15.4.5 expo-contacts: ~12.8.2 => 12.8.2 expo-dev-client: ~3.3.11 => 3.3.11 expo-device: ~5.9.3 => 5.9.3 expo-file-system: ~16.0.6 => 16.0.8 expo-font: ~11.10.3 => 11.10.3 expo-haptics: ~12.8.1 => 12.8.1 expo-image: ~1.10.6 => 1.10.6 expo-image-picker: ~14.7.1 => 14.7.1 expo-linear-gradient: ~12.7.2 => 12.7.2 expo-linking: ~6.2.2 => 6.2.2 expo-localization: ~14.8.3 => 14.8.3 expo-media-library: ~15.9.1 => 15.9.1 expo-notifications: ~0.27.6 => 0.27.6 expo-secure-store: ~12.8.1 => 12.8.1 expo-splash-screen: ~0.26.4 => 0.26.4 expo-status-bar: ~1.11.1 => 1.11.1 expo-store-review: ~6.8.3 => 6.8.3 expo-updates: ~0.24.12 => 0.24.12 expo-video-thumbnails: ~7.9.0 => 7.9.0 fbjs-scripts: 3.0.1 => 3.0.1 ffmpeg-kit-react-native: ^6.0.2 => 6.0.2 i18n-js: 3.9.2 => 3.9.2 inherits: 2.0.1 isarray: 0.0.1 jest: ^29.2.1 => 29.7.0 jest-circus: 29 => 29.7.0 jest-environment-node: 29 => 29.7.0 jest-expo: ~50.0.4 => 50.0.4 jsdom: ^22.1.0 => 22.1.0 (20.0.3) jsdom-jscore-rn: ^0.1.8 => 0.1.8 libphonenumber-js: ^1.10.56 => 1.10.56 libphonenumber-js/build: undefined () libphonenumber-js/core: undefined () libphonenumber-js/max: undefined () libphonenumber-js/max/metadata: undefined () libphonenumber-js/min: undefined () libphonenumber-js/min/metadata: undefined () libphonenumber-js/mobile: undefined () libphonenumber-js/mobile/examples: undefined () libphonenumber-js/mobile/metadata: undefined () lodash: ^4.17.21 => 4.17.21 lodash.filter: ^4.6.0 => 4.6.0 lottie-react-native: 6.5.1 => 6.5.1 metro-config: ~0.80.6 => 0.80.6 metro-source-map: ^0.80.6 => 0.80.6 mobx: 6.6.0 => 6.6.0 mobx-react-lite: 3.4.0 => 3.4.0 mobx-state-tree: 5.1.5 => 5.1.5 mocha: ^10.2.0 => 10.3.0 nwmatcher: 1.4.3 patch-package: ^6.4.7 => 6.5.1 path-browserify: 0.0.0 postinstall-prepare: 1.0.1 => 1.0.1 prettier: 2.8.8 => 2.8.8 (3.2.5) query-string: ^7.0.1 => 7.1.3 (6.10.1) querystring: 0.2.0 react: 18.2.0 => 18.2.0 react-devtools-core: 4.24.7 => 4.24.7 (4.28.5) react-dom: 18.2.0 => 18.2.0 react-native: 0.73.6 => 0.73.6 react-native-blurhash: ^2.0.2 => 2.0.2 react-native-bootsplash: ^5.0.2 => 5.4.1 react-native-compressor: ^1.8.23 => 1.8.24 react-native-context-menu-view: ^1.15.0 => 1.15.0 react-native-device-info: ^10.12.0 => 10.12.0 react-native-dots-pagination: ^0.3.1 => 0.3.1 react-native-element-dropdown: ^2.9.0 => 2.10.1 react-native-fs: ^2.20.0 => 2.20.0 react-native-gesture-handler: ~2.14.0 => 2.14.1 react-native-get-random-values: ~1.8.0 => 1.8.0 react-native-mime-types: ^2.5.0 => 2.5.0 react-native-mmkv: ^2.11.0 => 2.12.1 react-native-pager-view: 6.2.3 => 6.2.3 react-native-reanimated: ~3.6.2 => 3.6.3 react-native-receive-sharing-intent: ^2.0.0 => 2.0.0 react-native-render-html: ^6.3.4 => 6.3.4 react-native-safe-area-context: 4.8.2 => 4.8.2 react-native-screens: ~3.29.0 => 3.29.0 react-native-static-safe-area-insets: ^2.2.0 => 2.2.0 react-native-touchable-scale: ^2.2.0 => 2.2.0 react-native-url-polyfill: ^2.0.0 => 2.0.0 react-native-video: ^6.0.0-beta.4 => 6.0.0-beta.5 react-native-vision-camera: ^4.0.0-beta.11 => 4.0.0-beta.11 react-native-volume-manager: ^1.10.0 => 1.10.0 react-native-web: ~0.19.6 => 0.19.10 react-native-webview: 13.6.4 => 13.6.4 react-native-youtube-iframe: ^2.3.0 => 2.3.0 react-test-renderer: 18.2.0 => 18.2.0 reactotron-core-client: ^2.8.10 => 2.9.2 (2.8.10) reactotron-mst: 3.1.4 => 3.1.4 reactotron-react-js: ^3.3.7 => 3.3.13 reactotron-react-native: 5.0.3 => 5.0.3 regenerator-runtime: ^0.13.4 => 0.13.11 (0.14.1) string_decoder: 0.10.31 ts-jest: 29 => 29.1.2 typescript: ^4.9.4 => 5.3.3 urlmaster: 0.2.15 uuid: ^9.0.1 => 9.0.1 (8.3.2, 7.0.3) npmGlobalPackages: @aws-amplify/cli-internal: 12.10.1 @aws-amplify/cli: 12.10.1 @react-native-community/netinfo: 9.4.1 eas-cli: 7.6.2 expo-cli: 6.3.10 firebase-tools: 11.24.1 n: 9.1.0 node-gyp: 10.0.1 node: 20.6.0 npm: 10.3.0 pod-install: 0.2.0 react-native-spinkit: 1.5.1 ```

Describe the bug

I'm using the onProgress callback from the upload data exactly as per the docs: Storage Docs

And I cannot get the onProgress callback to provide any feedback whatsoever to me. It's really important in my app to give the user some sort of percent complete feedback and this needs to be working.

Expected behavior

The uploadData function's onProgress callback should log what percent complete the upload is but nothing happens.

Reproduction steps

On iPhone 12, run the code per the docs:

import { uploadData } from 'aws-amplify/storage';

try {
  const result = uploadData({
    key: filename,
    data: file,
    options: {
      onProgress: ({ transferredBytes, totalBytes }) => {
        if (totalBytes) {
          console.log(
            `Upload progress ${
              Math.round((transferredBytes / totalBytes) * 100)
            } %`
          );
        }
      }
    }
  }).result;
  console.log('Key from Response: ', result.key);
} catch (error) {
  console.log('Error : ', error);
}

Code Snippet

I created a custom upload function so that I could get the file key:

export async function uploadToS3(input: Omit<UploadDataInput, "data"> & { filePath: string }): Promise<string> {
  //   await fetchAuthSession({ forceRefresh: true })
  const response = await fetch(input.filePath)
  const data = await response.blob()
  const uploadDataOutput = uploadData({ key: input.key, data, options: input?.options })
  const result = await uploadDataOutput.result
  const key = result.key
  const urlResponse = await getUrl({
    key,
    options: {
      accessLevel: input.options?.accessLevel,
    },
  })
  const fullUrl = urlResponse.url.toString()
  return stripUrlParams(fullUrl)
}

and here is it's usage:

await uploadToS3({
          filePath: preUploadTempThumbnailPath,
          key: `${newSharedMediaId}/thumbnailUrl.jpg`,
          options: {
            contentType: "image/jpg",
            onProgress: (event) =>{
              console.log("MAIN THUMBNAIL UPLOAD", event)
            }
          },
 })

You will get no console logs to update the upload status.

Log output

``` // Put your logs below this line ```

aws-exports.js

/* eslint-disable */
// WARNING: DO NOT EDIT. This file is automatically generated by AWS Amplify. It will be overwritten.

const awsmobile = {
    "aws_project_region": "us-east-1",
    "aws_appsync_graphqlEndpoint": *REDACTED*,
    "aws_appsync_region": "us-east-1",
    "aws_appsync_authenticationType": "API_KEY",
    "aws_appsync_apiKey": *REDACTED*,
    "aws_cognito_identity_pool_id": "us-east-1:a2d5d29b-d69d-46d6-81c3-c6dc71323225",
    "aws_cognito_region": "us-east-1",
    "aws_user_pools_id": *REDACTED*,
    "aws_user_pools_web_client_id": *REDACTED*,
    "oauth": {},
    "aws_cognito_username_attributes": [],
    "aws_cognito_social_providers": [],
    "aws_cognito_signup_attributes": [
        "GIVEN_NAME",
        "FAMILY_NAME",
        "BIRTHDATE",
        "PHONE_NUMBER"
    ],
    "aws_cognito_mfa_configuration": "OFF",
    "aws_cognito_mfa_types": [],
    "aws_cognito_password_protection_settings": {
        "passwordPolicyMinLength": 8,
        "passwordPolicyCharacters": [
            "REQUIRES_LOWERCASE",
            "REQUIRES_UPPERCASE",
            "REQUIRES_NUMBERS",
            "REQUIRES_SYMBOLS"
        ]
    },
    "aws_cognito_verification_mechanisms": [
        "PHONE_NUMBER"
    ],
    "aws_user_files_s3_bucket": "reelfeelmedia",
    "aws_user_files_s3_bucket_region": "us-east-1"
};

export default awsmobile;

Manual configuration

No response

Additional configuration

No response

Mobile Device

iPhone 12 Physical

Mobile Operating System

iOS17

Mobile Browser

No response

Mobile Browser Version

No response

Additional information and screenshots

No response

cwomack commented 5 months ago

Hello again, @ChristopherGabba and thanks for creating this issue. Can you share any errors you see (if any) and confirm if you're seeing the file actually get uploaded into the S3 bucket? Want to see if we can isolate this to the onProgress callback or if there's something happening with the uploadData() for you.

ashika112 commented 5 months ago

@ChristopherGabba How big of a file are you uploading?

ChristopherGabba commented 5 months ago

@cwomack The file has uploaded perfectly every time, everything seems to be working perfectly but the onProgress callback does not fire at all.

@ashika112 The files can be small from 0.1MB up to about 15MB (short <30s videos and thumbnails).

ChristopherGabba commented 5 months ago

The react-native-compressor package:

const result = await Video.compress(
  'https://example.com/video.mp4',
  {
    progressDivider: 10,  // allows you to adjust the percent interval on how often the callback fires
    downloadProgress: (progress) => {
      console.log('downloadProgress: ', progress);
    },
  },
  (progress) => {
    console.log('Compression Progress: ', progress);
  }
);

and the react-native-fs package

type DownloadFileOptions = {
  progressInterval?: number;
  progressDivider?: number;
  begin?: (res: DownloadBeginCallbackResult) => void; // Note: it is required when progress prop provided
  progress?: (res: DownloadProgressCallbackResult) => void;
  resumable?: () => void;    // only supported on iOS yet
  connectionTimeout?: number // only supported on Android yet
  readTimeout?: number       // supported on Android and iOS
  backgroundTimeout?: number // Maximum time (in milliseconds) to download an entire resource (iOS only, useful for timing out background downloads)
};

that both download files locally have great progress feedback callbacks that let you customize how often they fire.

In this case, when signal is bad the upload can take a while and it's important that I give the user some sort of status feedback within the UI. Without this onProgress working, I can't do that.

I really feel like this uploadData function should have a functioning onProgress callback as well as a lot of these customizations like uploadingTimeout, progressUpdateInterval, etc. At a minimum it definitely needs to work.

natuan62 commented 5 months ago

Hi @ChristopherGabba Have you have same issue like me on amplify v5 on build mode https://github.com/aws-amplify/amplify-js/issues/11863

ChristopherGabba commented 5 months ago

@natuan62 Thanks for referencing. I looked and there is no resumable flag in v6. Here are the options:

Screenshot 2024-04-08 at 5 22 51 AM
natuan62 commented 5 months ago

@ChristopherGabba I did test with vite5-vue3 + amplify v6. The onProgress callback still trigger work fine on both (run dev localhost and build code + run serve)

ChristopherGabba commented 5 months ago

@natuan62 Maybe it's working on the react framework and not iOS react-native...

cwomack commented 5 months ago

@ChristopherGabba, we've tried to reproduce and test this internally and aren't running into any issue when using React Native too (on latest, same version as yours it looks like). Using the code you provided yielded a proper upload and log that matched yours as well.

To try and pin the cause of the issue you're experiencing further, could you clarify what simulator/devices you're using? And do you have any console logs from other sources by chance? Thanks!

ChristopherGabba commented 5 months ago

@cwomack @ashika112

Here is an expo-snack that you can use to reproduce this perfectly. You will get no onProgress callback fires at all: https://snack.expo.dev/@christophergabba/expo-image-picker-s3---no-onprogress

This is on a physical iPhone using expo development build (and in a TestFlight build). Seeing this on multiple different iPhones (12, 13, 15). You'll need to configure amplify to get this connected but it's not firing at all in my mobile app.

cshfang commented 5 months ago

Hi @ChristopherGabba - is there any way you can try downgrading to Expo SDK 48?

I came across https://github.com/expo/expo/discussions/27448 in my investigation and I was able to

  1. Reproduce what you were seeing using Expo 50
  2. See the progress events again after downgrading to Expo 48

This appears to be an Expo issue with XHR events not being triggered.

cshfang commented 5 months ago

@ChristopherGabba can I also ask if you are using Expo Go?

I was only able to replicate this issue with Expo Go at SDK 49+. When running the same application using

$ npx expo run:ios

I think I was able to avoid the issue. I wanted to bring this up since Amplify v6 no longer supports Expo Go.

cshfang commented 5 months ago

I have opened the following issue against Expo: https://github.com/expo/expo/issues/28269

I believe this is not an issue with Amplify and would recommend that we go ahead and close this and continue to monitor the Expo issue instead. Please let us know if that sounds acceptable to you @ChristopherGabba!

ChristopherGabba commented 5 months ago

@cshfang @cwomack Man there's a lot of Chris's in this thread!

I'm using an Expo development build on the managed work process, so not using Expo Go.

Thanks very much for digging down and identifying the potential source, it is pretty interesting that this problem falls in the Expo ecosystem. On another note, I am using the react-native-fs and react-native-compressor packages and am doing some basic downloadFile and Image.compress methods in my app. They both have downloadProgress callbacks, and am seeing that they are not always getting called as well. I found other issues such as:

https://github.com/numandev1/react-native-compressor/issues/276

I'll try downgrading and I'll keep my eye on the expo issue you have created. Historically these expo issues remain open for quite some time in my experience, but it sounds like it's definitely not an issue with the amplify package so I'm good with closing this issue.

Thanks again! Chris

cwomack commented 4 months ago

Appreciate the follow ups and discussion on this thread, @ChristopherGabba. We'll close this issue out and encourage anyone that comes across this to follow the expo issue opened by @cshfang above.

ChristopherGabba commented 3 weeks ago

@cshfang @cwomack I would like to reopen this issue. Referencing the issue mentioned above with Expo Go, this is still happening in Expo Development Client in Expo 49-51. According to the issue above, this is not an expo issue #28269 in a development client, and expo is successfully providing XMLHttpRequestUploadEvents in a Development Build, but the uploadProgress callbacks aren't firing.