Amplify v6, Storage uploadData throws ECONNABORTED error consistently in React Native Testflight Production App

[ChristopherGabba]

Before opening, please confirm:

• [X] I have searched for duplicate or closed issues and discussions.
• [X] I have read the guide for submitting bug reports.
• [X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

JavaScript Framework

React Native

Amplify APIs

Storage

Amplify Version

v6

Amplify Categories

storage

Backend

Amplify CLI

Environment information

``` System: OS: macOS 14.2.1 CPU: (10) arm64 Apple M2 Pro Memory: 467.20 MB / 16.00 GB Shell: 5.9 - /bin/zsh Binaries: Node: 20.7.0 - /opt/homebrew/bin/node npm: 10.1.0 - /opt/homebrew/bin/npm Watchman: 2023.09.04.00 - /opt/homebrew/bin/watchman Browsers: Safari: 17.2.1 npmPackages: @aws-amplify/react-native: ^1.0.12 => 1.0.12 @babel/core: ^7.20.0 => 7.23.7 @babel/plugin-proposal-export-namespace-from: ^7.18.9 => 7.18.9 @babel/preset-env: ^7.20.0 => 7.23.8 @babel/runtime: ^7.20.0 => 7.23.8 @expo-google-fonts/m-plus-1p: ^0.2.3 => 0.2.3 @expo-google-fonts/montserrat: ^0.2.3 => 0.2.3 @expo-google-fonts/space-grotesk: ^0.2.2 => 0.2.3 @expo/config-plugins: ~7.2.2 => 7.2.5 (5.0.4) @expo/metro-config: ^0.10.7 => 0.10.7 @gorhom/bottom-sheet: ^4.4.7 => 4.6.0 @likashefqet/react-native-image-zoom: ^2.1.1 => 2.1.1 @react-native-async-storage/async-storage: ^1.18.2 => 1.21.0 @react-native-clipboard/clipboard: ^1.11.2 => 1.13.2 @react-native-community/cli-platform-ios: ^8.0.2 => 8.0.6 (11.3.7) @react-native-community/netinfo: ^9.3.10 => 9.5.0 @react-navigation/bottom-tabs: ^6.3.2 => 6.5.11 @react-navigation/native: ^6.0.2 => 6.1.9 @react-navigation/native-stack: ^6.0.2 => 6.9.17 @rnx-kit/metro-config: ^1.3.5 => 1.3.14 @rnx-kit/metro-resolver-symlinks: ^0.1.26 => 0.1.34 @sentry/react-native: 5.10.0 => 5.10.0 @shopify/flash-list: 1.4.3 => 1.4.3 @types/i18n-js: 3.8.2 => 3.8.2 @types/jest: ^29.2.1 => 29.5.11 @types/lodash.filter: ^4.6.9 => 4.6.9 @types/react: ~18.2.14 => 18.2.48 @types/react-test-renderer: ^18.0.0 => 18.0.7 @typescript-eslint/eslint-plugin: ^5.59.0 => 5.62.0 @typescript-eslint/parser: ^5.59.0 => 5.62.0 HelloWorld: 0.0.1 apisauce: 2.1.5 => 2.1.5 aws-amplify: ^6.0.12 => 6.0.12 aws-amplify/adapter-core: undefined () aws-amplify/analytics: undefined () aws-amplify/analytics/kinesis: undefined () aws-amplify/analytics/kinesis-firehose: undefined () aws-amplify/analytics/personalize: undefined () aws-amplify/analytics/pinpoint: undefined () aws-amplify/api: undefined () aws-amplify/api/server: undefined () aws-amplify/auth: undefined () aws-amplify/auth/cognito: undefined () aws-amplify/auth/cognito/server: undefined () aws-amplify/auth/enable-oauth-listener: undefined () aws-amplify/auth/server: undefined () aws-amplify/datastore: undefined () aws-amplify/in-app-messaging: undefined () aws-amplify/in-app-messaging/pinpoint: undefined () aws-amplify/push-notifications: undefined () aws-amplify/push-notifications/pinpoint: undefined () aws-amplify/storage: undefined () aws-amplify/storage/s3: undefined () aws-amplify/storage/s3/server: undefined () aws-amplify/storage/server: undefined () aws-amplify/utils: undefined () axios: ^1.5.0 => 1.6.5 (0.21.4) babel-jest: ^29.2.1 => 29.7.0 babel-loader: 8.2.5 => 8.2.5 babel-plugin-root-import: ^6.6.0 => 6.6.0 cheerio: ^1.0.0-rc.12 => 1.0.0-rc.12 core-util-is: 1.0.1 date-fns: ^2.29.2 => 2.30.0 eslint: 8.17.0 => 8.17.0 eslint-config-prettier: 8.5.0 => 8.5.0 eslint-config-standard: 17.0.0 => 17.0.0 eslint-plugin-import: 2.26.0 => 2.26.0 eslint-plugin-n: ^15.0.0 => 15.7.0 eslint-plugin-node: 11.1.0 => 11.1.0 eslint-plugin-promise: 6.0.0 => 6.0.0 eslint-plugin-react: 7.30.0 => 7.30.0 eslint-plugin-react-native: 4.0.0 => 4.0.0 expo: ^49.0.21 => 49.0.21 expo-application: ~5.3.0 => 5.3.1 expo-blur: ~12.4.1 => 12.4.1 expo-build-properties: ~0.8.3 => 0.8.3 expo-clipboard: ~4.3.1 => 4.3.1 expo-config-plugin-ios-share-extension: ^0.0.4 => 0.0.4 expo-constants: ~14.4.2 => 14.4.2 expo-contacts: ~12.2.0 => 12.2.0 expo-dev-client: ~2.4.11 => 2.4.12 expo-device: ~5.4.0 => 5.4.0 expo-file-system: ~15.4.4 => 15.4.5 expo-font: ~11.4.0 => 11.4.0 expo-image: ~1.3.5 => 1.3.5 expo-image-picker: ~14.3.2 => 14.3.2 expo-linear-gradient: ~12.3.0 => 12.3.0 expo-linking: ~5.0.2 => 5.0.2 expo-localization: ~14.3.0 => 14.3.0 expo-media-library: ~15.4.1 => 15.4.1 expo-notifications: ~0.20.1 => 0.20.1 expo-secure-store: ~12.3.1 => 12.3.1 expo-splash-screen: ^0.20.5 => 0.20.5 expo-status-bar: ~1.6.0 => 1.6.0 expo-store-review: ~6.4.0 => 6.4.0 expo-video-thumbnails: ~7.4.0 => 7.4.0 fbjs-scripts: 3.0.1 => 3.0.1 i18n-js: 3.9.2 => 3.9.2 inherits: 2.0.1 isarray: 0.0.1 jest: ^29.2.1 => 29.7.0 jest-circus: 29 => 29.7.0 jest-environment-node: 29 => 29.7.0 jest-expo: ^49.0.0 => 49.0.0 jsdom: ^22.1.0 => 22.1.0 (20.0.3) jsdom-jscore-rn: ^0.1.8 => 0.1.8 lodash: ^4.17.21 => 4.17.21 lodash.filter: ^4.6.0 => 4.6.0 lottie-react-native: ^5.1.6 => 5.1.6 metro-config: 0.76.8 => 0.76.8 metro-source-map: 0.75.1 => 0.75.1 (0.76.8) mobx: 6.6.0 => 6.6.0 mobx-react-lite: 3.4.0 => 3.4.0 mobx-state-tree: 5.1.5 => 5.1.5 mocha: ^10.2.0 => 10.2.0 nwmatcher: 1.4.3 patch-package: ^6.4.7 => 6.5.1 path-browserify: 0.0.0 postinstall-prepare: 1.0.1 => 1.0.1 prettier: 2.8.8 => 2.8.8 (3.2.4) query-string: ^7.0.1 => 7.1.3 (6.10.1) querystring: 0.2.0 react: 18.2.0 => 18.2.0 react-devtools-core: 4.24.7 => 4.24.7 (4.28.5) react-dom: 18.2.0 => 18.2.0 react-native: 0.72.6 => 0.72.6 react-native-bootsplash: ^5.0.2 => 5.2.2 react-native-compressor: ^1.8.23 => 1.8.23 react-native-device-info: ^10.12.0 => 10.12.0 react-native-dots-pagination: ^0.3.1 => 0.3.1 react-native-element-dropdown: ^2.9.0 => 2.10.1 react-native-fs: ^2.20.0 => 2.20.0 react-native-gesture-handler: ~2.12.0 => 2.12.1 react-native-get-random-values: ~1.9.0 => 1.9.0 react-native-mmkv: ^2.11.0 => 2.11.0 react-native-pager-view: 6.2.0 => 6.2.0 react-native-reanimated: ~3.3.0 => 3.3.0 react-native-receive-sharing-intent: ^2.0.0 => 2.0.0 react-native-render-html: ^6.3.4 => 6.3.4 react-native-safe-area-context: 4.6.3 => 4.6.3 react-native-screens: ~3.22.0 => 3.22.1 react-native-shimmer-placeholder: ^2.0.9 => 2.0.9 react-native-static-safe-area-insets: ^2.2.0 => 2.2.0 react-native-touchable-scale: ^2.2.0 => 2.2.0 react-native-url-polyfill: ^2.0.0 => 2.0.0 react-native-video: ^6.0.0-beta.4 => 6.0.0-beta.4 react-native-vision-camera: ^3.6.6 => 3.8.0 react-native-volume-manager: ^1.10.0 => 1.10.0 react-native-web: ~0.19.6 => 0.19.10 react-native-webview: 13.2.2 => 13.2.2 react-native-youtube-iframe: ^2.3.0 => 2.3.0 react-test-renderer: 18.2.0 => 18.2.0 reactotron-core-client: ^2.8.10 => 2.8.11 (2.8.10) reactotron-mst: 3.1.4 => 3.1.4 reactotron-react-js: ^3.3.7 => 3.3.9 reactotron-react-native: 5.0.3 => 5.0.3 regenerator-runtime: ^0.13.4 => 0.13.11 (0.14.1) sentry-expo: ~7.1.0 => 7.1.1 string_decoder: 0.10.31 ts-jest: 29 => 29.1.1 typescript: ^4.9.4 => 5.3.3 urlmaster: 0.2.15 npmGlobalPackages: @aws-amplify/cli-internal: 12.10.0 @aws-amplify/cli: 12.10.1 @react-native-community/netinfo: 9.4.1 eas-cli: 7.0.0 expo-cli: 6.3.10 firebase-tools: 11.24.1 n: 9.1.0 node: 20.6.0 npm: 10.3.0 pod-install: 0.1.39 react-native-spinkit: 1.5.1 ```

Describe the bug

Uploading media to S3 via the uploadData api occasionally fails upload and throws ECONNABORTED Network Error. Unable to reproduce exact replication scenario, but it is happening ~30% of the time in production - beta.

Expected behavior

The video should complete the upload successfully, which it does most of the time. Connected to good wifi connection, no bad signal, still receiving this error.

Reproduction steps

1. Install amplify v6 CLI via the instructions
2. Install expo-image-picker
3. Here is the usage:

   
   import * as ImagePicker from "expo-image-picker"

async function openPhotoLibrary() {

const { status } = await ImagePicker.requestMediaLibraryPermissionsAsync() if (status !== "granted") { alert("The app needs access to your local_phone_storage in order for you to use this feature." ) return }

const result = await ImagePicker.launchImageLibraryAsync({
  mediaTypes: ImagePicker.MediaTypeOptions.All,
  exif: false,
  videoQuality: ImagePicker.UIImagePickerControllerQualityType.Low,
  // allowsEditing: true,
  // aspect: [4, 3],
  videoMaxDuration: 29,
  quality: 0.1,
})

if (!result.canceled) {
  // success
  const { uri } = result.assets[0]

  const extension = getFileExtension(uri)
  const contentType = getContentType(uri)

  const mediaUri = await uploadToS3({
      filePath: uri,
      key: `originalUrl.${extension}`,
      options: {
        accessLevel: "guest",
        contentType: contentType + "/" + extension,
      },
   })
}

}

4. Here is a custom upload function that I have developed for reusability:
```typescript
import { uploadData, UploadDataInput, getUrl, } from "aws-amplify/storage"

export async function uploadToS3(input: Omit<UploadDataInput, "data"> & { filePath: string }): Promise<string> {
  try {
    const response = await fetch(input.filePath)
    const data = await response.blob()
    const uploadDataOutput = uploadData({ key: input.key, data, options: input?.options })
    const result = await uploadDataOutput.result
    const key = result.key
    const urlResponse = await getUrl({
      key,
      options: {
        accessLevel: input.options?.accessLevel,
      },
    })
    const fullUrl = urlResponse.url.toString()
    return fullUrl.substring(0, fullUrl.indexOf("?"))
  } catch (error) {
    reportError({ error, method: "uploadToS3" }). // This is where the error is caught and displayed
    return ""
  }
}

Code Snippet

Most code is provided above in the reproduction steps.

Log output

Occuring in production copy of app in testflight Beta, so can't get logs. Refer to Sentry screenshot below.

aws-exports.js

Here is my aws-exports.js: / eslint-disable / // WARNING: DO NOT EDIT. This file is automatically generated by AWS Amplify. It will be overwritten.

const awsmobile = {
    "aws_project_region": "us-east-1",
    "aws_appsync_graphqlEndpoint": "https://5tvnvesrgjfjdclupp5rzrh5gi.appsync-api.us-east-1.amazonaws.com/graphql",
    "aws_appsync_region": "us-east-1",
    "aws_appsync_authenticationType": "API_KEY",
    "aws_appsync_apiKey": *REDACTED*,
    "aws_cognito_identity_pool_id":  *REDACTED*,
    "aws_cognito_region": "us-east-1",
    "aws_user_pools_id": "us-east-1_eDAImGHL9",
    "aws_user_pools_web_client_id":  *REDACTED*,
    "oauth": {},
    "aws_cognito_username_attributes": [],
    "aws_cognito_social_providers": [],
    "aws_cognito_signup_attributes": [
        "GIVEN_NAME",
        "FAMILY_NAME",
        "BIRTHDATE",
        "PHONE_NUMBER"
    ],
    "aws_cognito_mfa_configuration": "OFF",
    "aws_cognito_mfa_types": [],
    "aws_cognito_password_protection_settings": {
        "passwordPolicyMinLength": 8,
        "passwordPolicyCharacters": [
            "REQUIRES_LOWERCASE",
            "REQUIRES_UPPERCASE",
            "REQUIRES_NUMBERS",
            "REQUIRES_SYMBOLS"
        ]
    },
    "aws_cognito_verification_mechanisms": [
        "PHONE_NUMBER"
    ],
    "aws_user_files_s3_bucket": "reelfeelmedia",
    "aws_user_files_s3_bucket_region": "us-east-1"
};

export default awsmobile;

Manual configuration

Here is my amplifyconfiguration.json:

{
  "aws_project_region": "us-east-1",
  "aws_appsync_graphqlEndpoint": "https://5tvnvesrgjfjdclupp5rzrh5gi.appsync-api.us-east-1.amazonaws.com/graphql",
  "aws_appsync_region": "us-east-1",
  "aws_appsync_authenticationType": "API_KEY",
  "aws_appsync_apiKey":  *REDACTED*,
  "aws_cognito_identity_pool_id":  *REDACTED*,
  "aws_cognito_region": "us-east-1",
  "aws_user_pools_id": "us-east-1_eDAImGHL9",
  "aws_user_pools_web_client_id": " *REDACTED*,
  "oauth": {},
  "aws_cognito_username_attributes": [],
  "aws_cognito_social_providers": [],
  "aws_cognito_signup_attributes": [
    "GIVEN_NAME",
    "FAMILY_NAME",
    "BIRTHDATE",
    "PHONE_NUMBER"
  ],
  "aws_cognito_mfa_configuration": "OFF",
  "aws_cognito_mfa_types": [],
  "aws_cognito_password_protection_settings": {
    "passwordPolicyMinLength": 8,
    "passwordPolicyCharacters": [
      "REQUIRES_LOWERCASE",
      "REQUIRES_UPPERCASE",
      "REQUIRES_NUMBERS",
      "REQUIRES_SYMBOLS"
    ]
  },
  "aws_cognito_verification_mechanisms": [
    "PHONE_NUMBER"
  ],
  "aws_user_files_s3_bucket":  *REDACTED*,
  "aws_user_files_s3_bucket_region": "us-east-1"
}

Additional configuration

Here is my configuration in App.tsx:

import { Amplify } from "aws-amplify"
import amplifyconfig from "./app/amplifyconfiguration.json"

const libraryOptions = {
  Storage: {
    S3: {
      prefixResolver: async ({
        accessLevel,
        targetIdentityId,
      }: {
        accessLevel: any
        targetIdentityId?: string
      }) => {
        if (accessLevel === "guest") {
          return "public/"
        } else if (accessLevel === "protected") {
          return `protected/${targetIdentityId}/`
        } else {
          return `private/${targetIdentityId}/`
        }
      },
    },
  },
}

Amplify.configure(amplifyconfig, libraryOptions)

Mobile Device

iPhone 13 Pro (Physical)

Mobile Operating System

iOS 17.2.1

Mobile Browser

Safari

Mobile Browser Version

Unsure

Additional information and screenshots

Sentry Error with a little more error details:

iOS error alert:

[nadetastic]

Hi @ChristopherGabba thank you for opening this issue.

Taking a look at this, it appears that this issue is intermittent, only affecting some users, and is not easily reproducible. Im working to test this on my end and one thought I had was possibly the size of the upload is causing an error where S3 aborts the connection. Are you testing with files that are larger than 100MB with no error? Im also working to verify the same, but it also seems that this issue may be specific to production builds.

[ChristopherGabba]

@nadetastic Just got the error again during the upload... here is where that error came:

async function sendResponse() {
  if (!response.sender) return

  setIsLoading(true)

  const client = generateClient()
  try {
    // Upload main phone media to S3
    const compressedVideo = await compressVideo(uri)
    const videoUri = await uploadToS3({ // caught the exception here
      filePath: compressedVideo,
      key: `${response.id}/videoUrl.mp4`,
      options: {
        accessLevel: "guest",
        contentType: "video/mp4",
      },
    })
    // Get the new result Uri and update dynamoDB

    const apiResponse = await client.graphql({
      query: updateResponse,
      variables: {
        input: {
          id: response.id,
          caption: response.caption,
          videoUri: videoUri
        },
      },
    })

    console.log("successfully updated response", apiResponse)
    await sendResponseNotification(response.sender?.pushToken, true, response?.caption)
    await showConfirmation("common.sent")
  } catch (error) {
    reportError({
      error,
    })
  }

Error: ECONNABORTED , Description: Network Error

It is a failure to upload to S3

[ChristopherGabba]

@nadetastic If you would like me to modify my error output in some way to gather more information during the ECONNABORTED I can to help try and solve it.

Feel free to modify my uploadToS3 function to try and see what step its occurring at. Also if there is a way to program this to try again if this occurs maybe two more times or something that would be awesome. Maybe modify to try two more times max to get a successful upload?

[nadetastic]

@ChristopherGabba thanks for the additional information - one question I have - is there a way for you to tell the size of the file being uploaded when this occurs? I'd like to try eliminate that if it is not the case. My thinking is something may be happening where s3 aborts the connection in the middle of a large file upload. And to confirm, you do not see the video in s3 when this error occurs?

Additionally, you can do some retry logic that loops up to 2 times or breaks if the call is succesful. That is one way to mitigate this, but I would still like to understand what the root cause is here.

[ChristopherGabba]

@nadetastic Sorry I should have answered you when you asked that the first time! My uploads are typically from my photo library or recording from a camera with a max recorded length of 30s and on relatively low quality. But we always perform a video compression before the upload to save S3 memory. I just ran a test and it was 2.5MB after compression. All of ours should be < 10MB if I had to guess.

Here was my log output prior to the S3 upload: compressed from 6.546267509460449 MB to 1.5085878372192383 MB

And to answer your last question, the video does not upload successfully, we do not see it in S3.

[ChristopherGabba]

@nadetastic What are the chances this is related to this miserable ExpiredToken error I was getting intermittently in V5? That error would come after the app sat in the background for a few hours then you immediately open it and try to upload a file to S3.

I'm not sure on the backend side of things what's different between V5 and V6, but we are seeing a similar pattern with this ECONNABORTED: Network Error where the app sits in the background for a while, you open it up and upload, and then it fires off ECONNABORTED. I upload things throughout the day for testing and the error only seems to come when the app is in the background for a while and then you try to upload something.

I changed my authentication patterns every which way but hell to try and fix that ExpiredToken error, to the point where I upgraded from V5 amplify to V6 and when I didn't get that error again I was so ecstatic. Then shortly into testing V6, we started getting this ECONNABORTED errors as a replacement...

TBH these two errors have kept me from releasing the app for over 3 weeks now. I can't go into production with this happening.

Maybe it's a combination of our S3 bucket policy and our authentication?

Reading about this error elsewhere on other platforms, here are a few helpful links:

• https://stackoverflow.com/questions/36949557/express-js-sendfile-returning-econnaborted
• https://stackoverflow.com/questions/58235297/error-write-econnaborted-on-deploying-react-app-using-azure-app-service-exten
• https://stackoverflow.com/questions/54936185/express-mongoose-jest-error-econnaborted
• https://stackoverflow.com/questions/76610312/unable-to-receive-a-json-response-after-a-post-request
• https://stackoverflow.com/questions/55419866/node-js-request-aborted
• https://stackoverflow.com/questions/50619928/get-notified-after-axios-timeout

It seems based off the various articles that I've read, this error is related to a few different causes:

• Port mismatch on various API calls
• Interrupting an upload early
• Missing CORS headers in upload post requests
• Timeout of the upload

Really just trying to help, sorry for the long message.

[nadetastic]

HI @ChristopherGabba I think that is a great point - especially considering that the app sits in the background for a while. One thing you can try to do is to call fetchAuthSession() before you make the upload which would refresh the session. Additionally, you can also add a Hub listener to see if token refresh failure also happens. I will try to reproduce with the same context.

[ChristopherGabba]

@nadetastic I am still experiencing this every now. I am now using the fetchAuthSession with a force refresh. When I try the upload again, most of the time it goes through on the second attempt. The issue is definitely still present so I don't want to close.

The kicker is that it occurs infrequently and usually after the app has been static for a while.

[nadetastic]

@ChristopherGabba thanks for the additional info, Is it possible for you to share a minimal sample app for me to reproduce and test with?

[ChristopherGabba]

@nadetastic We have been using our app in TestFlight now for about 2 months. During this time, it's pretty random. Every day or every other day one of the users gets the ECONNABORTED error when they attempt the S3 upload. We have done a decent job at trying to make a graceful error that allows the user to try and re-upload, and usually when they try again, it works. However, I have been unable to get this to fire on command, but it is happening for several different users on several different types of iPhones. It would seem that the best way to replicate this would be:

• Create a sample app with a sample S3 storage bucket
• Have an image picker that will upload a video to S3
• Leave the app in the background and a few times a day upload a random photo
• You could potentially try and force this error by throttling network, or removing network connection and then immediately after getting it back try to upload, and trying different combinations.

I think the reason why it doesn't happen on development builds is that I'm always closing and refreshing the development builds, so its not getting the time to disconnect or lose connection to the aws server / web socket.

[cwomack]

@ChristopherGabba, this appears to be an error tied to axios from what I can tell. Are you using that package by chance in any way and can you share any code that uses it if you are?

This comment from the apache/cordova-android repo may give some insight in how to debug this better as well.

[ChristopherGabba]

@cwomack The place where this error comes the most often is this function:

async function sendResponse() {
    if (!response.sender) return

    setIsLoading(true)

    const client = generateClient()
    try {
      // Upload main phone media to S3
      const [reactionVideoUrl, reactionThumbnailUrl] = await Promise.all([
        await uploadToS3({
          filePath: reactionVideo,
          key: `${response.sharedMedia.id}/rf-${response.id}/reactionVideoUrl.mp4`,
          options: {
            accessLevel: "guest",
            contentType: "video/mp4",
          },
        }),
        await uploadToS3({
          filePath: thumbnailUrl,
          key: `${response.sharedMedia.id}/rf-${response.id}/reactionThumbnailUrl.jpg`,
          options: {
            contentType: "image/jpg",
          },
        }),
      ])

      const response = await client.graphql({
        query: updateResponse,
        variables: {
          input: {
            id: response.id,
            // ....
          },
        },
      })

      console.log("successfully updated response", response)
      await sendResponseNotification(
        response.sender?.pushToken,
        true,
      )
      setConfirmation({ tx: "common.sent", durationMillis: 1000 })
      await new Promise((resolve) => setTimeout(resolve, 1000))
      setResponseSent(true)
      setIsLoading(false)
    } catch (error) {
      setIsLoading(false)
      setConfirmation({ tx: "errors.sendingResponseError" })
      reportCrash({
        error,
        method: "sendResponse",
        component: "RecordingScreen",
      })
    }
  }

The only method in here that is not an amplify related method is the sendResponseNotification. The only other methods are client.graphql and the amplify upload function. This method uses fetch:

  try {
    await fetch("https://exp.host/--/api/v2/push/send", {
      method: "POST",
      headers: {
        Accept: "application/json",
        "Accept-encoding": "gzip, deflate",
        "Content-Type": "application/json",
      },
      body: JSON.stringify(message),
    })
  } catch (error) {
    reportCrash({
      error,
      method: "sendReelFeelNotification",
    })
  }

I know it doesn't get to this point though because the other user does not get the notification, and it is the uploadToS3 that actually fails. Do any of these amplify functions use axios under the hood? I can't replicate this issue in the development environment. It only happens every now and then in the production environment.

[ChristopherGabba]

@cwomack @nadetastic Okay, I finally spent time hunting down the exact source (line of code) that is triggering this error in production. It's happening at least 5 times a day among my test users and it's been driving me up the wall.

I put try-catch blocks around every line in my uploadToS3 function. I also had my reportCrash function get the network state during the time, and the aws authentication state. Both appear to be good, although this does happen more often with poor signal.

export async function uploadToS3(
  input: Omit<UploadDataInput, "data"> & { filePath: string },
): Promise<string> {
  let response: Response
  let data: Blob
  let output: UploadDataOutput
  let result: any
  let key: any
  let urlResponse: any

  try {
    response = await fetch(input.filePath)
  } catch (error) {
    reportCrash({ error, component: "uploadToS3-CatchBlock1" })
    throw error
  }

  try {
    data = await response.blob()
  } catch (error) {
    reportCrash({ error, component: "uploadToS3-CatchBlock2" })
    throw error
  }
  try {
    output = uploadData({ key: input.key, data, options: input?.options })
  } catch (error) {
    reportCrash({ error, component: "uploadToS3-CatchBlock3" })
    throw error
  }
  try {
    result = await output.result // <------------------------- It's this line right here
    key = result.key
  } catch (error) {
    reportCrash({ error, component: "uploadToS3-CatchBlock4" }). // <-------- see error screenshot below
    throw error
  }

  try {
    urlResponse = await getUrl({
      key,
      options: {
        accessLevel: input.options?.accessLevel,
      },
    })
  } catch (error) {
    reportCrash({ error, component: "uploadToS3-CatchBlock5" })
    throw error
  }

  const fullUrl = urlResponse.url.toString()
  return stripUrlParams(fullUrl)
}

Here is what I can tell you so far:

1. It's happing in the TestFlight production build on iOS maybe one in every 10 - 20 uploads
2. Once it happens once, it tends to happen a few more times until you reset the app. Sometimes you retry the upload and it goes again on the next hit, sometimes your app gets ECONNABORTED constantly and you have to reset the whole app.
3. It is clearly originating from the function above in the amplify-js package in v6.
4. The internet being slow definitely seems to contribute to it's frequency. You can see in the screenshot above that the network state was poor during the ECONNABORTED, but I have gotten it many times with decent internet.
5. Leaving the app in the background for a while and opening it up and uploading quickly seems to contribute to it.
6. It's happening on iPhone 12, 13, 14, and 15, to every user. One is in Colorado, one is in California, and two are in Houston.
7. If I push a new app version (user is on 1.0.0 and I push a 1.0.1) and they download and open the app and upload, it seems to happen more often right after the new version is downloaded.
8. Putting a fetchAuthSession({forceRefresh: true}) before calling this does not solve the problem.

This &$*@ has plagued us like a menace for the last 3 months and I will Venmo whoever solves this 50$ for beer.

If there is anything I can do, like you guys provide me with a patch file that has better logging for you using patch-package on the uploadData().resultfunction, I will gladly do it. There is something in that package throwing this error.

If you send me the background code of this uploadData function and I can use it in place of the uploadData function and break it down with try-catches even further. I will literally do anything to solve this lol...

Here are screenshots of the native alert I have displayed for the user:

[ChristopherGabba]

@cwomack a few more users got it today. Originating from the same function every time. The screenshots below show that it is coming from xhr-http-handler.

All the error messages show the user is still authenticated with cognito, neither token is expired (access or identity) and the user has a good internet connection.

[cshfang]

Hi @ChristopherGabba. I looked into the origin point of this error a bit and found it here

A couple issues with what we did and I'm talking to other engineers on what we should do to address:

1. The error code being returned here is incorrect/misleading - instead of 'ECONNABORTED' we really should be returning 'ERR_NETWORK'
2. When we replaced Axios in our library, we ported over some of the same rationale/logic when we were implementing our XHR handler and did not make use of the underlying event. We are evaluating if/how we want to approach exposing this underlying event but, per the specs, there may not be much information there either other than "The fetch failed."

I will add items to our task queue to address/evaluate these 2 issues. But, in the meantime, if it's possible to patch this line in your testing, you can try to get more information from the XHR event itself directly.

[cshfang]

Following up my comment above. Ultimately, the error is coming from XMLHttpRequest itself. Inspecting the error under the hood may help to glean some insight as to exactly what the network error is but is itself a low level construct in the runtime that the Amplify library would not actually have any control over. We can keep the issue open as we do need to fix the associated error code itself but the root cause does not appear to be something that can addressed at the library level.

[ChristopherGabba]

@cshfang First of all, thank you so much for digging deep and trying to find the source of this, I am very grateful. I patched the line below:

-        xhr.addEventListener('error', () => {
-            const networkError = buildHandlerError(constants_1.NETWORK_ERROR_MESSAGE, constants_1.NETWORK_ERROR_CODE);
+        xhr.addEventListener('error', (event) => {
+            const networkError = buildHandlerError(JSON.stringify(event, null, 4), constants_1.NETWORK_ERROR_CODE);

to capture the full error event. I do not know how this will end up being displayed, but I'll work a few different iterations if it comes out poorly. If you have any suggestions on the exact patch, I'll jump right on it.

Will get the error data here as soon as we get the error.

[cwomack]

@ChristopherGabba, appreciate you taking the time to help us get the context of what's happening here. It looks like the underlying issue being experienced isn't quite something that we'll be able to immediately fix from a bug standpoint, but wanted to let you know we're evaluating how to expose the underlying errors when this happens in a better format.

Given that the underlying xhr error is not technically a bug on the Amplify side, are you ok with us closing the issue at this point? To reiterate, we're doing what @cshfang mentioned above. However, at this time it seems there's no further debugging or action we can take at this time from the Amplify side.

[ChristopherGabba]

@cwomack Before closing, just so I have a better understanding, where exactly is this "bug" happening? Understanding I'm still trying to gather the raw error event message. Is it some sort of server rejection to the users upload? I just want to make sure that if this issue gets Closed, it's not forgotten. I really would like this bug to not happen again 😂. It definitely still needs a solution, so should I open another issue somewhere else?

[ChristopherGabba]

Okay guys, I just got the following error message for the raw event data:

Looks like the problem is isTrusted: false

[cshfang]

@cwomack Before closing, just so I have a better understanding, where exactly is this "bug" happening? Understanding I'm still trying to gather the raw error event message. Is it some sort of server rejection to the users upload? I just want to make sure that if this issue gets Closed, it's not forgotten. I really would like this bug to not happen again 😂. It definitely still needs a solution, so should I open another issue somewhere else?

Hey @ChristopherGabba. The error message is coming from the XMLHttpRequest object itself - as to the reason why, I honestly cannot speak to it as it is essentially a runtime implementation in ReactNative/Expo at that point. It's outside of the library control at that point and we can't really provide better insight into further debugging the issue.

The isTrusted property coming back from the XHR error message unfortunately tells us very little because successful events are going to contain that same property and value as well.

At this time, I would try to inspect the actual network logs to see if I can glean further insight as to what the issue is but I suspect it's a low level network connectivity resolution that is at fault here. Either searching through Expo issues or SO for similar issues and continuing to narrow down poor network conditions would be my approach. But, at this time, I'm afraid there's very little that we can do to the library code to narrow it down further.

[ChristopherGabba]

@cshfang @cwomack Thanks a bunch for digging deep. I'm okay with closing this issue, please keep me updated on any updates you guys do make in this area. Because I'm wading in some unknown waters here, how could I inspect the actual network logs? How is this different than what I've already done?

I did open the above issues with Expo and the React Native team to hopefully make progress in that area.

[cshfang]

Because I'm wading in some unknown waters here, how could I inspect the actual network logs? How is this different than what I've already done?

Hmm typically I would do this via Flipper, React Native Debugger or Reactotron. But it may require being able to replicate the issue locally somehow. I'm not sure if there is a way to configure those tools to log to some remote logging tool like CloudWatch.

Ultimately, I suspect the root issue is poor network conditions, maybe it would be possible to replicate locally using network link conditioner? This seems a tricky one to track down for sure.