search

facebook / hermes

A JavaScript engine optimized for running React Native.
https://hermesengine.dev/
MIT License
9.64k stars 618 forks source link

Property storage exceeds 196607 properties when creating a large sparse array #1070

Open XantreDev opened 1 year ago

XantreDev commented 1 year ago

Bug Description

I've used @react-three/fiber and tried to load GLTF model with useGLTF hook (it uses expo-asset which uses blueimp-md5 which throws with hermes).

Error log:

Possible Unhandled Promise Rejection (id: 0):
RangeError: Property storage exceeds 196607 properties
RangeError: Property storage exceeds 196607 properties
    at rstr2binl (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:1308:19)
    at rstrMD5 (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:1319:41)
    at rawMD5 (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:1353:21)
    at hexMD5 (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:1356:29)
    at md5 (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:1367:24)
    at ?anon_0_ (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:256954:56)
    at next (native)
    at asyncGeneratorStep (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:1456:26)
    at _next (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:1475:29)
    at anonymous (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:1480:14)
    at tryCallTwo (/root/react-native/ReactAndroid/hermes-engine/.cxx/Release/5e6w3h5p/arm64-v8a/lib/InternalBytecode/InternalBytecode.js:61:9)
    at doResolve (/root/react-native/ReactAndroid/hermes-engine/.cxx/Release/5e6w3h5p/arm64-v8a/lib/InternalBytecode/InternalBytecode.js:216:25)
    at Promise (/root/react-native/ReactAndroid/hermes-engine/.cxx/Release/5e6w3h5p/arm64-v8a/lib/InternalBytecode/InternalBytecode.js:82:14)
    at anonymous (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:1472:25)
    at apply (native)
    at _downloadAsyncUnmanagedEnv (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:256947:45)
    at ?anon_0_ (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:256919:40)
    at next (native)
    at asyncGeneratorStep (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:1456:26)
    at _next (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:1475:29)
    at anonymous (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:1480:14)
    at tryCallTwo (/root/react-native/ReactAndroid/hermes-engine/.cxx/Release/5e6w3h5p/arm64-v8a/lib/InternalBytecode/InternalBytecode.js:61:9)
    at doResolve (/root/react-native/ReactAndroid/hermes-engine/.cxx/Release/5e6w3h5p/arm64-v8a/lib/InternalBytecode/InternalBytecode.js:216:25)
    at Promise (/root/react-native/ReactAndroid/hermes-engine/.cxx/Release/5e6w3h5p/arm64-v8a/lib/InternalBytecode/InternalBytecode.js:82:14)
    at anonymous (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:1472:25)
    at apply (native)
    at downloadAsync (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:256912:32)
    at ?anon_0_ (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:259334:103)
    at next (native)
    at asyncGeneratorStep (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:1456:26)
    at _next (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:1475:29)
    at anonymous (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:1480:14)
    at tryCallTwo (/root/react-native/ReactAndroid/hermes-engine/.cxx/Release/5e6w3h5p/arm64-v8a/lib/InternalBytecode/InternalBytecode.js:61:9)
    at doResolve (/root/react-native/ReactAndroid/hermes-engine/.cxx/Release/5e6w3h5p/arm64-v8a/lib/InternalBytecode/InternalBytecode.js:216:25)
    at Promise (/root/react-native/ReactAndroid/hermes-engine/.cxx/Release/5e6w3h5p/arm64-v8a/lib/InternalBytecode/InternalBytecode.js:82:14)
    at anonymous (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:1472:25)
    at apply (native)
    at downloadAsync (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:259353:39)
    at anonymous (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:260924:34)
    at anonymous (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:478024:22)
    at tryCallTwo (/root/react-native/ReactAndroid/hermes-engine/.cxx/Release/5e6w3h5p/arm64-v8a/lib/InternalBytecode/InternalBytecode.js:61:9)
    at doResolve (/root/react-native/ReactAndroid/hermes-engine/.cxx/Release/5e6w3h5p/arm64-v8a/lib/InternalBytecode/InternalBytecode.js:216:25)
    at Promise (/root/react-native/ReactAndroid/hermes-engine/.cxx/Release/5e6w3h5p/arm64-v8a/lib/InternalBytecode/InternalBytecode.js:82:14)
    at loadBuffer (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:478023:27)
    at getDependency (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:477961:43)
    at loadBufferView (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:478033:34)
    at anonymous (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:477957:64)
    at _invokeOne (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:477915:28)
    at getDependency (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:477956:43)
    at loadAccessor (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:478054:53)
    at getDependency (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:477953:45)
    at assignAttributeAccessor (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:478868:34)
    at addPrimitiveAttributes (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:478875:43)
    at loadGeometries (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:478413:55)
    at loadMesh (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:478437:43)
    at anonymous (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:477949:52)
    at _invokeOne (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:477915:28)
    at getDependency (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:477948:43)
    at createNodeMesh (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:478652:36)
    at anonymous (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:478707:58)
    at _invokeOne (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:477915:28)
    at _loadNodeShallow (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:478706:44)
    at loadNode (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:478671:50)
    at anonymous (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:477944:52)
    at _invokeOne (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:477915:28)
    at getDependency (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:477943:43)
    at loadScene (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:478780:44)
    at getDependency (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:477940:42)
    at anonymous (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:478005:40)
    at map (native)
    at getDependencies (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:478004:46)
    at anonymous (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:477833:53)
    at tryCallOne (/root/react-native/ReactAndroid/hermes-engine/.cxx/Release/5e6w3h5p/arm64-v8a/lib/InternalBytecode/InternalBytecode.js:53:16)
    at anonymous (/root/react-native/ReactAndroid/hermes-engine/.cxx/Release/5e6w3h5p/arm64-v8a/lib/InternalBytecode/InternalBytecode.js:139:27)
    at apply (native)
    at anonymous (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:28884:26)
    at _callTimer (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:28803:17)
    at _callReactNativeMicrotasksPass (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:28833:17)
    at callReactNativeMicrotasks (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:28996:44)
    at __callReactNativeMicrotasks (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:4331:46)
    at anonymous (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:4143:45)
    at __guard (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:4315:15)
    at flushedQueue (http://localhost:8081/index.bundle?platform=android&dev=true&minify=false&app=dev.world.oone.driverapp&modulesOnly=false&runModule=true:4142:21)

This is the function: https://github.com/blueimp/JavaScript-MD5/blob/2c00da944e9f9ecc2348cf9ed26cd3808106aee2/js/md5.js#L257C48-L257C48 It fills array in unfamiliar way, that it not throws with small inputs, but throws with big ones

Hermes version: React Native version (if any): 0.71.7 OS version (if any): Android Platform (most likely one of arm64-v8a, armeabi-v7a, x86, x86_64): arm64-v8a / x86_64

Steps To Reproduce

Run this code inside hermes

var output = []
output[(6666145 >> 2) - 1] = undefined
for (i = 0; i < output.length; i += 1) {
   output[i] = 0
}

The Expected Behavior

To not recognize array as object and not throw

The Current Behavior

Trying to use array indices as object keys and throws

tmikov commented 1 year ago

Thank you for this bug report.

It is an interesting case. In the first iterations of the loop, the situation is like this: you have an element at index 1,666,535 and are adding an element at a small index around 0. Hermes detects this as a sparse array (because there is a lot of distance between 1,666,535 and the start of the array) and adds the element as an object property instead of an indexed array element, in order to conserve memory. At a certain point it runs out of property storage space.

This must be fixed but we need to think what the best way is.

tmikov commented 1 year ago

To further clarify the current Hermes behavior. It allows us to support cases like this, which would normally run out of memory trying to allocate many gigabytes of data:

a = [1];
a[4294967294] = 1;
XantreDev commented 1 year ago

Thank you for this bug report.

It is an interesting case. In the first iterations of the loop, the situation is like this: you have an element at index 1,666,535 and are adding an element at a small index around 0. Hermes detects this as a sparse array (because there is a lot of distance between 1,666,535 and the start of the array) and adds the element as an object property instead of an indexed array element, in order to conserve memory. At a certain point it runs out of property storage space.

This must be fixed but we need to think what the best way is.

I think a good way to determine it: is check at some amount of elements, that each of them is number (in case of array) and allocate memory for that array

crackhack8266 commented 9 months ago

Any solution ?

XantreDev commented 9 months ago

Patched with

var output = (new Array(6666145 >> 2))).fill(0)
crackhack8266 commented 9 months ago

Patched with


var output = (new Array(6666145 >> 2))).fill(0)

It works!!! Thanks

senamusic commented 4 months ago

Remendado com

var output = (new Array(6666145 >> 2))).fill(0)

Funciona!!! Obrigado

In which file do I put this code?

XantreDev commented 4 months ago

These 4 lines

var output = []
output[(input.length >> 2) - 1] = undefined
for (i = 0; i < output.length; i += 1) {
   output[i] = 0
}

https://github.com/blueimp/JavaScript-MD5/blob/2c00da944e9f9ecc2348cf9ed26cd3808106aee2/js/md5.js#L256-L260 To this one

var output = (new Array(6666145 >> 2))).fill(0)
tmikov commented 4 months ago

FWIW, we have a plan for fixing this. But everyone, please keep in mind that creating and using sparse arrays is almost never a good idea

XantreDev commented 4 months ago

FWIW, we have a plan for fixing this. But everyone, please keep in mind that creating and using sparse arrays is almost never a good idea

(new Array(6666145 >> 2))).fill(0)still creates a holley array, is not it? What is the most optimal way to create and fill arrays in js?

tmikov commented 4 months ago

(new Array(6666145 >> 2)).fill(0) creates a dense array. It is pretty efficient.

danyalutsevich commented 2 months ago

i have same issue using hemes with "react-native": "0.74.2" and "expo": "~51.0.14"