search

aws-amplify / amplify-js

A declarative JavaScript library for application development using cloud services.
https://docs.amplify.aws/lib/q/platform/js
Apache License 2.0
9.41k stars 2.11k forks source link

Rest API/Lambda "No space left on device" network error #13721

Closed pjsandwich closed 1 week ago

pjsandwich commented 1 month ago

Before opening, please confirm:

JavaScript Framework

React Native

Amplify APIs

REST API, GraphQL API

Amplify Version

v5

Amplify Categories

auth, function, api

Backend

Amplify CLI

Environment information

``` # Put output below this line System: OS: Windows 10 10.0.19043 CPU: (8) x64 Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz Memory: 1.82 GB / 15.96 GB Binaries: Node: 20.3.1 - C:\Program Files\nodejs\node.EXE Yarn: 1.22.22 - C:\Program Files\nodejs\yarn.CMD npm: 9.6.7 - C:\Program Files\nodejs\npm.CMD pnpm: 8.15.1 - C:\Program Files\nodejs\pnpm.CMD Browsers: Chrome: 127.0.6533.119 Edge: Spartan (44.19041.1266.0), Chromium (127.0.2651.74) Internet Explorer: 11.0.19041.1566 npmPackages: @aws-amplify/api: ^6.0.10 => 6.0.42 (5.4.12) @aws-amplify/api/internals: undefined () @aws-amplify/api/server: undefined () @aws-amplify/auth: ^5.5.3 => 5.6.12 @babel/core: ^7.20.0 => 7.25.2 @expo/ngrok: ^4.1.0 => 4.1.3 @expo/vector-icons: ^13.0.0 => 13.0.0 @react-native-async-storage/async-storage: 1.18.2 => 1.18.2 @react-native-community/datetimepicker: 7.2.0 => 7.2.0 @react-native-community/netinfo: 9.3.10 => 9.3.10 @react-native-community/slider: 4.4.2 => 4.4.2 @react-native-masked-view/masked-view: 0.2.9 => 0.2.9 @react-navigation/bottom-tabs: ^6.3.1 => 6.6.1 @react-navigation/drawer: ^6.4.1 => 6.7.2 @react-navigation/material-bottom-tabs: ^6.2.1 => 6.2.29 @react-navigation/native: ^6.0.10 => 6.1.18 @react-navigation/native-stack: ^6.6.2 => 6.11.0 @types/d3-shape: ^3.1.6 => 3.1.6 @types/moment-duration-format: ^2.2.4 => 2.2.6 @types/react: ~18.2.14 => 18.2.79 @types/react-native: ^0.73.0 => 0.73.0 @typescript-eslint/eslint-plugin: ^6.17.0 => 6.21.0 @typescript-eslint/parser: ^6.17.0 => 6.21.0 HelloWorld: 0.0.1 amazon-cognito-identity-js: ^5.2.8 => 5.2.14 (6.3.13) amazon-cognito-identity-js/internals: undefined () aws-amplify: ^5.3.11 => 5.3.20 aws-amplify-react-native: ^6.0.4 => 6.0.8 aws-sdk: ^2.1194.0 => 2.1668.0 eslint: ^8.43.0 => 8.57.0 eslint-config-airbnb: latest => 19.0.4 eslint-config-prettier: ^8.8.0 => 8.10.0 eslint-import-resolver-typescript: ^3.6.1 => 3.6.1 eslint-plugin-import: ^2.25.3 => 2.29.1 eslint-plugin-jsx-a11y: ^6.5.1 => 6.9.0 eslint-plugin-react: latest => 7.35.0 eslint-plugin-react-hooks: ^4.3.0 => 4.6.2 eslint-plugin-react-native: ^4.0.0 => 4.1.0 expo: ^49.0.0 => 49.0.23 expo-apple-authentication: ~6.1.0 => 6.1.2 expo-blur: ~12.4.1 => 12.4.1 expo-constants: ~14.4.2 => 14.4.2 expo-dev-client: ~2.4.12 => 2.4.13 expo-device: ~5.4.0 => 5.4.0 expo-file-system: ~15.4.5 => 15.4.5 expo-font: ~11.4.0 => 11.4.0 expo-linear-gradient: ~12.3.0 => 12.3.0 expo-linking: ~5.0.2 => 5.0.2 expo-localization: ~14.3.0 => 14.3.0 expo-notifications: ~0.20.1 => 0.20.1 expo-status-bar: ~1.6.0 => 1.6.0 expo-web-browser: ~12.3.2 => 12.3.2 lodash: ^4.17.21 => 4.17.21 moment: ^2.29.3 => 2.30.1 moment-duration-format: ^2.3.2 => 2.3.2 moment-timezone: ^0.5.34 => 0.5.45 prettier: ^2.8.8 => 2.8.8 react: 18.2.0 => 18.2.0 react-dom: 18.2.0 => 18.2.0 react-native: 0.72.6 => 0.72.6 (0.74.5) react-native-calendars: ^1.1284.0 => 1.1306.0 react-native-chart-kit: ^6.12.0 => 6.12.0 react-native-gesture-handler: ~2.12.0 => 2.12.1 react-native-get-random-values: ~1.9.0 => 1.9.0 (1.11.0) react-native-google-fit: ^0.19.1 => 0.19.1 react-native-health: ^1.18.0 => 1.18.0 react-native-modal-datetime-picker: ^13.1.2 => 13.1.2 react-native-paper: ^5.12.1 => 5.12.5 react-native-reanimated: ~3.3.0 => 3.3.0 react-native-safe-area-context: 4.6.3 => 4.6.3 react-native-screens: ~3.22.0 => 3.22.1 react-native-select-dropdown: ^1.13.0 => 1.13.0 react-native-svg: 13.9.0 => 13.9.0 react-native-svg-charts: ^5.4.0 => 5.4.0 react-native-vector-icons: ^9.2.0 => 9.2.0 react-native-web: ~0.19.6 => 0.19.12 react-use: ^17.3.2 => 17.5.1 rn-placeholder: ^3.0.3 => 3.0.3 src: undefined () typescript: ^5.1.3 => 5.5.4 uuid: ^8.3.2 => 8.3.2 (3.4.0, 9.0.1, 8.0.0, 7.0.3) npmGlobalPackages: @aws-amplify/cli: 12.10.1 corepack: 0.18.0 eas-cli: 7.8.2 npm: 9.6.7 typescript: 5.1.6 yarn: 1.22.22 ```

Describe the bug

I have a function that writes batches of items to dynamo db. This is invoked via rest endpoint set up with amplify. Sometimes there are 30,000+ items that need to be written, I noticed that while AWS docs state 10MB can be transmitted, this doesn't seem to work, I've had to cap each API call 0.05 MB to work. While the API calls initially start to work after some number of requests I start to see the following network error, specifically odd is that there's no status code returned and the error code under response is "The operation couldn’t be completed. No space left on device".

{
    "message": "Network Error",
    "name": "AxiosError",
    "stack": "AxiosError: Network Error\n    at handleError (http://<IP_ADDRESS>:<PORT>/node_modules%5Cexpo%5CAppEntry.bundle//&platform=ios&dev=true&hot=false&lazy=true:<LINE_NUMBER>:<COLUMN_NUMBER>)\n    at call (native)\n    at dispatchEvent (http://<IP_ADDRESS>:<PORT>/node_modules%5Cexpo%5CAppEntry.bundle//&platform=ios&dev=true&hot=false&lazy=true:<LINE_NUMBER>:<COLUMN_NUMBER>)\n    at setReadyState (http://<IP_ADDRESS>:<PORT>/node_modules%5Cexpo%5CAppEntry.bundle//&platform=ios&dev=true&hot=false&lazy=true:<LINE_NUMBER>:<COLUMN_NUMBER>)\n    at __didCompleteResponse (http://<IP_ADDRESS>:<PORT>/node_modules%5Cexpo%5CAppEntry.bundle//&platform=ios&dev=true&hot=false&lazy=true:<LINE_NUMBER>:<COLUMN_NUMBER>)\n    at apply (native)\n    at anonymous (http://<IP_ADDRESS>:<PORT>/node_modules%5Cexpo%5CAppEntry.bundle//&platform=ios&dev=true&hot=false&lazy=true:<LINE_NUMBER>:<COLUMN_NUMBER>)\n    at apply (native)\n    at emit (http://<IP_ADDRESS>:<PORT>/node_modules%5Cexpo%5CAppEntry.bundle//&platform=ios&dev=true&hot=false&lazy=true:<LINE_NUMBER>:<COLUMN_NUMBER>)\n    at apply (native)\n    at __callFunction (http://<IP_ADDRESS>:<PORT>/node_modules%5Cexpo%5CAppEntry.bundle//&platform=ios&dev=true&hot=false&lazy=true:<LINE_NUMBER>:<COLUMN_NUMBER>)\n    at anonymous (http://<IP_ADDRESS>:<PORT>/node_modules%5Cexpo%5CAppEntry.bundle//&platform=ios&dev=true&hot=false&lazy=true:<LINE_NUMBER>:<COLUMN_NUMBER>)\n    at __guard (http://<IP_ADDRESS>:<PORT>/node_modules%5Cexpo%5CAppEntry.bundle//&platform=ios&dev=true&hot=false&lazy=true:<LINE_NUMBER>:<COLUMN_NUMBER>)\n    at callFunctionReturnFlushedQueue (http://<IP_ADDRESS>:<PORT>/node_modules%5Cexpo%5CAppEntry.bundle//&platform=ios&dev=true&hot=false&lazy=true:<LINE_NUMBER>:<COLUMN_NUMBER>)\n    at ?anon_0_ (http://<IP_ADDRESS>:<PORT>/node_modules%5Cexpo%5CAppEntry.bundle//&platform=ios&dev=true&hot=false&lazy=true:<LINE_NUMBER>:<COLUMN_NUMBER>)\n    at throw (native)\n    at asyncGeneratorStep (http://<IP_ADDRESS>:<PORT>/node_modules%5Cexpo%5CAppEntry.bundle//&platform=ios&dev=true&hot=false&lazy=true:<LINE_NUMBER>:<COLUMN_NUMBER>)\n    at _throw (http://<IP_ADDRESS>:<PORT>/node_modules%5Cexpo%5CAppEntry.bundle//&platform=ios&dev=true&hot=false&lazy=true:<LINE_NUMBER>:<COLUMN_NUMBER>)\n    at tryCallOne (/Users/<USERNAME>/react-native/packages/react-native/sdks/hermes/build_iphoneos/lib/InternalBytecode/InternalBytecode.js:<LINE_NUMBER>:<COLUMN_NUMBER>)\n    at anonymous (/Users/<USERNAME>/react-native/packages/react-native/sdks/hermes/build_iphoneos/lib/InternalBytecode/InternalBytecode.js:<LINE_NUMBER>:<COLUMN_NUMBER>)\n    at apply (native)\n    at anonymous (http://<IP_ADDRESS>:<PORT>/node_modules%5Cexpo%5CAppEntry.bundle//&platform=ios&dev=true&hot=false&lazy=true:<LINE_NUMBER>:<COLUMN_NUMBER>)\n    at _callTimer (http://<IP_ADDRESS>:<PORT>/node_modules%5Cexpo%5CAppEntry.bundle//&platform=ios&dev=true&hot=false&lazy=true:<LINE_NUMBER>:<COLUMN_NUMBER>)\n    at _callReactNativeMicrotasksPass (http://<IP_ADDRESS>:<PORT>/node_modules%5Cexpo%5CAppEntry.bundle//&platform=ios&dev=true&hot=false&lazy=true:<LINE_NUMBER>:<COLUMN_NUMBER>)\n    at callReactNativeMicrotasks (http://<IP_ADDRESS>:<PORT>/node_modules%5Cexpo%5CAppEntry.bundle//&platform=ios&dev=true&hot=false&lazy=true:<LINE_NUMBER>:<COLUMN_NUMBER>)\n    at __callReactNativeMicrotasks (http://<IP_ADDRESS>:<PORT>/node_modules%5Cexpo%5CAppEntry.bundle//&platform=ios&dev=true&hot=false&lazy=true:<LINE_NUMBER>:<COLUMN_NUMBER>)\n    at anonymous (http://<IP_ADDRESS>:<PORT>/node_modules%5Cexpo%5CAppEntry.bundle//&platform=ios&dev=true&hot=false&lazy=true:<LINE_NUMBER>:<COLUMN_NUMBER>)\n    at __guard (http://<IP_ADDRESS>:<PORT>/node_modules%5Cexpo%5CAppEntry.bundle//&platform=ios&dev=true&hot=false&lazy=true:<LINE_NUMBER>:<COLUMN_NUMBER>)\n    at flushedQueue (http://<IP_ADDRESS>:<PORT>/node_modules%5Cexpo%5CAppEntry.bundle//&platform=ios&dev=true&hot=false&lazy=true:<LINE_NUMBER>:<COLUMN_NUMBER>)\n    at callFunctionReturnFlushedQueue (http://<IP_ADDRESS>:<PORT>/node_modules%5Cexpo%5CAppEntry.bundle//&platform=ios&dev=true&hot=false&lazy=true:<LINE_NUMBER>:<COLUMN_NUMBER>)",
    "config": {
        "transitional": {
            "silentJSONParsing": true,
            "forcedJSONParsing": true,
            "clarifyTimeoutError": false
        },
        "adapter": [
            "xhr",
            "http",
            "fetch"
        ],
        "transformRequest": [
            null
        ],
        "transformResponse": [
            null
        ],
        "timeout": 0,
        "xsrfCookieName": "XSRF-TOKEN",
        "xsrfHeaderName": "X-XSRF-TOKEN",
        "maxContentLength": -1,
        "maxBodyLength": -1,
        "env": {},
        "headers": {
            "Accept": "application/json, text/plain, */*",
            "Content-Type": "application/json; charset=UTF-8",
            "x-amz-date": "<DATE>",
"Authorization": "AWS4-HMAC-SHA256 Credential=<CREDENTIAL>, SignedHeaders=content-type;host;x-amz-date;x-amz-security-token, Signature=<SIGNATURE>",
            "X-Amz-Security-Token": "<SECURITY_TOKEN>"
        },
        "method": "post",
        "url": "https://<API_ENDPOINT>/<path>",
        "host": "<API_ENDPOINT>",
        "path": "<path>",
        "data": "<data>",
    },
    "code": "ERR_NETWORK",
request: Object
DONE: 4
HEADERS_RECEIVED: 2
LOADING: 3
OPENED: 1
UNSENT: 0
onload: null
onloadstart: null
onprogress: null
onreadystatechange: null
readyState: 4
responseHeaders: undefined
status: 0
timeout: 0
upload: Object
withCredentials: true
_aborted: false
_cachedResponse: undefined
_hasError: true
_headers: Object
_incrementalEvents: false
_lowerCaseResponseHeaders: Object
_method: "POST"
_perfKey: "network_XMLHttpRequest_https://<ENDPOINT>/<PATH>"
_performanceLogger: Object
_requestId: null
_response: "The operation couldn’t be completed. No space left on device"
_responseType: ""
_sent: true
_subscriptions: Array(0)
_timedOut: false
_trackingName: "unknown"
_url: "https://<ENDPOINT>/<PATH>"
}

Expected behavior

When I make an API call I should be able to:

  1. Pass more than 0.05MB without seeing a request too large error (as AWS docs say it should be 10MB cap)
  2. Make repeated API calls without seeing a "The operation couldn’t be completed. No space left on device"

Reproduction steps

  1. create function to handle some data
  2. post request x number of times to invoke lambda, passing 0.05 MB chunks of data per call
  3. see after x number of calls "The operation couldn’t be completed. No space left on device" error returned.

Code Snippet

// Put your code below this line.

Log output

``` // Put your logs below this line ```

aws-exports.js

No response

Manual configuration

No response

Additional configuration

No response

Mobile Device

iPhone 13

Mobile Operating System

iOS 15.1

Mobile Browser

No response

Mobile Browser Version

No response

Additional information and screenshots

No response

david-mcafee commented 1 month ago

Hi @pjsandwich - could you give us a few more details on the environment you're testing in? Looks like you are using Expo - is this error occurring in a simulator? If so, which one? Have you tested on anything other than iOS?

Thank you!

pjsandwich commented 1 month ago

@david-mcafee , this is on a physical iOS 13 device. Though, I'm not entirely sure what the device has to do with the API network error

cwomack commented 1 month ago

@pjsandwich, are you able to reproduce this issue on other devices (or even on a simulator)? It looks like there's a handful of discussions on this (see here for example) and this may be tied to the device itself rather than an Amplify API. Still trying to find more information on this though.

pjsandwich commented 1 month ago

@cwomack Unfortunately no, this has dependencies on code being run on a physical device, but this leads to part of the discussion that I mentioned earlier, I do not see this error when trying a larger chunk size and have a suspicion that because there are many requests being made and they only error out after a few hundred, sending less requests should help this, however, while AWS docs state that 10MB is the cap for api gateway, and 500MB is the cap for a lambda function, sending anything more than 0.05MB yields a "request too large" error

cwomack commented 3 weeks ago

@pjsandwich, it seems like this error isn't going to be something we'd cause or throw from the Amplify side. Usually this "No space left on device" error is a result of having too little free space on your local drive or possible too much in your cache. Can you check your device storage to see if you're possibly getting maxed out there? Also recommend checking to see if you have a new version of iOS possibly downloaded and awaiting installation, which can eat up more GB's of storage than expected.

pjsandwich commented 1 week ago

Looks like it is an error on device, oddly though the api is structuring the error so that it declares it as an axios error, having nested the actual cause of the error as part of the err.originalError.request parameter... which is such an odd way to report it