search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
14.91k stars 4.63k forks source link

[browser] RuntimeError: memory access out of bounds #96546

Closed pavelsavara closed 1 week ago

pavelsavara commented 8 months ago

Error Blob

{
  "ErrorMessage": "RuntimeError: memory access out of bounds",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Reproduction Steps

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=515167 Log

[20:26:27] warn: worker sent an error! http://127.0.0.1:39745/_framework/dotnet.native.worker.js:1: Uncaught RuntimeError: memory access out of bounds
[20:26:27] fail: MONO_WASM: Error
                     at Ze (http://127.0.0.1:39745/_framework/dotnet.js:3:19735)
                     at e (http://127.0.0.1:39745/_framework/dotnet.js:3:33729)
                     at http://127.0.0.1:39745/_framework/dotnet.js:3:33874
[20:26:27] warn: MONO_WASM: mono_wasm_exit failed unwind

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=515167 Error message validated: RuntimeError: memory access out of bounds Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 1/5/2024 12:55:21 PM UTC

Report

Build Definition Test Pull Request
774241 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#106329
774196 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#106317
773979 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#106013
773665 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#106166
773561 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#106169
773348 dotnet/runtime WasmTestOnV8-ST-System.Text.Encodings.Web.Tests.WorkItemExecution
773213 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#106166
773113 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution
772876 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#106271
772314 dotnet/runtime WasmTestOnChrome-ST-System.Text.Encodings.Web.Tests.WorkItemExecution
772028 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#106037
771854 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#106226
771814 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution
771612 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#106169
771470 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#106208
770671 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution
770319 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#106164
770038 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#106153
770007 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#106150
769899 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#106145
769813 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#106111
769640 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105941
769674 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105559
769572 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105866
769545 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#106123
769451 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#106111
769299 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#106108
769093 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105663
768949 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#106091
768676 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#104393
768667 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#106053
768671 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#106080
763399 dotnet/runtime WasmTestOnChrome-ST-System.Text.Encodings.Web.Tests.WorkItemExecution
768349 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#99596
768211 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#106062
768078 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#106030
767917 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105841
767877 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#106048
767709 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution
767356 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105841
766122 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105941
765139 dotnet/runtime WasmTestOnChrome-ST-System.Text.Encoding.Tests.WorkItemExecution
764961 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105910
764868 dotnet/runtime WasmTestOnV8-ST-System.Threading.Tasks.Dataflow.Tests.WorkItemExecution
764572 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105846
764553 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution
763838 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105548
763775 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution
763718 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105814
763499 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105782
763419 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105776
763061 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105548
762675 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105764
762167 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105765
762078 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105709
761851 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105765
761368 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105464
761344 dotnet/runtime WasmTestOnV8-ST-System.Text.Encoding.Tests.WorkItemExecution
761211 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105654
760839 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105709
760586 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105709
760243 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105471
760209 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105691
760113 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105561
759876 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105579
759555 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution
759524 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105424
759413 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105643
759407 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105642
759349 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105636
758737 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105559
758686 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105489
758465 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105610
758407 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#104849
758343 dotnet/runtime WasmTestOnChrome-ST-System.Text.Encodings.Web.Tests.WorkItemExecution
758133 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105471
758115 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105469
757874 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution
757596 dotnet/runtime WasmTestOnV8-ST-System.Runtime.Tests.WorkItemExecution dotnet/runtime#105548

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 0 79
ghost commented 8 months ago

Tagging subscribers to 'arch-wasm': @lewing See info in area-owners.md if you want to be subscribed.

Issue Details
### Error Blob ```json { "ErrorMessage": "RuntimeError: memory access out of bounds", "BuildRetry": false, "ErrorPattern": "", "ExcludeConsoleLog": false } ``` ### Reproduction Steps Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=515167 [Log](https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-96504-merge-533efffc1d07453f84/WasmTestOnBrowser-System.Diagnostics.DiagnosticSource.Tests/1/console.50136c6b.log?helixlogtype=result) ``` [20:26:27] warn: worker sent an error! http://127.0.0.1:39745/_framework/dotnet.native.worker.js:1: Uncaught RuntimeError: memory access out of bounds [20:26:27] fail: MONO_WASM: Error at Ze (http://127.0.0.1:39745/_framework/dotnet.js:3:19735) at e (http://127.0.0.1:39745/_framework/dotnet.js:3:33729) at http://127.0.0.1:39745/_framework/dotnet.js:3:33874 [20:26:27] warn: MONO_WASM: mono_wasm_exit failed unwind```
Author: pavelsavara
Assignees: pavelsavara
Labels: `arch-wasm`, `blocking-clean-ci`, `area-VM-threading-mono`, `Known Build Error`, `os-browser`
Milestone: 9.0.0
pavelsavara commented 8 months ago

Another Build Log

[23:11:59] info: [2024-01-10T23:11:59.397Z] [PASS] System.Threading.ThreadPools.Tests.ThreadPoolTests.RunProcessorCountItemsInParallel
[23:11:59] info: [2024-01-10T23:11:59.400Z] [SKIP] System.Threading.ThreadPools.Tests.ThreadPoolTests.SetMinMaxThreadsTest_ChangedInDotNetCore
[23:12:00] warn: worker sent an error! http://127.0.0.1:33633/_framework/dotnet.native.worker.js:1: Uncaught RuntimeError: memory access out of bounds
[23:12:00] fail: MONO_WASM: Error
                     at Ze (http://127.0.0.1:33633/_framework/dotnet.js:3:19735)
                     at e (http://127.0.0.1:33633/_framework/dotnet.js:3:33729)
                     at http://127.0.0.1:33633/_framework/dotnet.js:3:33874
[23:12:00] info: WASM EXIT 1
[23:12:00] info: Waiting to flush log messages with a timeout of 120 secs ..
pavelsavara commented 7 months ago

Another Log Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=549038&view=logs&jobId=63c2d0c8-fec2-5788-81c8-f3ac95e8841f

[11:44:47] fail: [0x066a4300-long 11:44:47.111] MONO_WASM: Unhandled error: {}
[11:44:47] fail: [0x066a4300-long 11:44:47.117] MONO_WASM: memory access out of bounds
                 RuntimeError: memory access out of bounds
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[8732]:0x217b29
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[2731]:0xd2e2b
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[2815]:0xd597a
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[2077]:0xaed21
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[324]:0x286c7
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[253]:0x25e62
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[244]:0x18884
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[283]:0x26af0
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[3319]:0xf9db8
                     at http://127.0.0.1:33073/_framework/dotnet.native.wasm:wasm-function[2587]:0xcbe17
[11:44:47] info: WASM EXIT 1
pavelsavara commented 6 months ago

Log

[08:55:30] info: Starting:    Microsoft.Extensions.Logging.Console.Tests.dll
[08:55:31] fail: [0x045f6788-emsc 08:55:31.650] MONO_WASM: preRunWorker() failed {}
[08:55:31] fail: [0x045f6788-emsc 08:55:31.652] MONO_WASM: memory access out of bounds
                 RuntimeError: memory access out of bounds
                     at http://127.0.0.1:44985/_framework/dotnet.native.wasm:wasm-function[649]:0x4a787
                     at http://127.0.0.1:44985/_framework/dotnet.native.wasm:wasm-function[650]:0x4a7d6
                     at http://127.0.0.1:44985/_framework/dotnet.native.wasm:wasm-function[652]:0x4a8ab
                     at http://127.0.0.1:44985/_framework/dotnet.native.wasm:wasm-function[530]:0x4793a
                     at e.<computed> (http://127.0.0.1:44985/_framework/dotnet.runtime.js:3:226093)
                     at http://127.0.0.1:44985/_framework/dotnet.runtime.js:3:101299
                     at Ga (http://127.0.0.1:44985/_framework/dotnet.runtime.js:3:101426)
                     at Ya (http://127.0.0.1:44985/_framework/dotnet.runtime.js:3:101713)
                     at http://127.0.0.1:44985/_framework/dotnet.runtime.js:3:212904
                     at iu (http://127.0.0.1:44985/_framework/dotnet.runtime.js:3:213061)
[08:55:31] info: [0x000dc84c-main 08:55:31.655] MONO_WASM: Dumping web worker info as seen by UI thread, it could be stale:
[08:55:31] info: 00 | 0x000dc84c-main: isRunning:false isAttached: true isEventLoop:false reuseCount:  1 - UI Thread
[08:55:31] info: 01 | 0x02ff8f90-norm: isRunning: true isAttached: true isEventLoop:false reuseCount:  1 - Finalizer
[08:55:31] info: 02 | 0x032a8030-pool: isRunning: true isAttached: true isEventLoop: true reuseCount:  1 - .NET TP Worker
[08:55:31] info: 03 | 0x035b0040-gate: isRunning: true isAttached: true isEventLoop:false reuseCount:  1 - .NET TP Gate
[08:55:31] info: 04 | 0x03dbea48-pool: isRunning: true isAttached: true isEventLoop: true reuseCount:  1 - .NET TP Worker
[08:55:31] info: 05 | 0x045f6788-emsc: isRunning: true isAttached:false isEventLoop:false reuseCount:  1 - pthread-assigned
[08:55:31] info: 06 | 0x045f6788-emsc: isRunning:false isAttached:false isEventLoop:false reuseCount:  1 - Console logger queue processing thread
[08:55:31] info: 07 | 0x045f6788-emsc: isRunning:false isAttached:false isEventLoop:false reuseCount:  1 - Console logger queue processing thread
[08:55:31] info: 08 | 0x045f6788-emsc: isRunning:false isAttached:false isEventLoop:false reuseCount:  1 - Console logger queue processing thread
[08:55:31] info: 09 | 0x045f6788-emsc: isRunning:false isAttached:false isEventLoop:false reuseCount:  1 - Console logger queue processing thread
[08:55:31] info: 10 | 0x045f6788-emsc: isRunning:false isAttached:false isEventLoop:false reuseCount:  1 - Console logger queue processing thread
[08:55:31] info: 11 | 0x045f6788-emsc: isRunning:false isAttached:false isEventLoop:false reuseCount:  1 - Console logger queue processing thread
[08:55:31] info: 12 | 0x045f6788-emsc: isRunning:false isAttached:false isEventLoop:false reuseCount:  1 - Console logger queue processing thread
[08:55:31] fail: [0x000dc84c-main 08:55:31.657] MONO_WASM: Uncaught RuntimeError: memory access out of bounds
                 Error
                     at Object.Ge [as mono_exit] (http://127.0.0.1:44985/_framework/dotnet.js:3:20140)
                     at ft.config.exitOnUnhandledError.e.onerror (http://127.0.0.1:44985/_framework/dotnet.runtime.js:3:38634)
[08:55:31] fail: [out of order message from the browser]: http://127.0.0.1:44985/_framework/dotnet.native.js 7:1295 Uncaught ErrorEvent: Uncaught RuntimeError: memory access out of bounds
pavelsavara commented 6 months ago

During thread creation Log Build

[18:39:28] fail: [--06290040-emsc 18:39:28.771] MONO_WASM: memory access out of bounds
                 RuntimeError: memory access out of bounds
                     at pthread_attr_init (http://127.0.0.1:34537/_framework/dotnet.native.wasm:wasm-function[8640]:0x2144c1)
                     at mono_thread_info_attach (http://127.0.0.1:34537/_framework/dotnet.native.wasm:wasm-function[786]:0x4ef10)
                     at start_wrapper (http://127.0.0.1:34537/_framework/dotnet.native.wasm:wasm-function[2762]:0xd6903)
                     at Object.invokeEntryPoint (http://127.0.0.1:34537/_framework/dotnet.native.js:8:69009)
                     at handleMessage (http://127.0.0.1:34537/_framework/dotnet.native.worker.js:1:2306)
pavelsavara commented 6 months ago

Related https://github.com/dotnet/runtime/issues/98630

pavelsavara commented 6 months ago

Log

[11:23:26] info: 001 | 0x03078030-pool: isRunning:false isAttached: true isEventLoop: true reuseCount:  1 - .NET TP Worker
[11:23:26] fail: [0x03078030-pool 11:23:26.696] MONO_WASM: memory access out of bounds
                 RuntimeError: memory access out of bounds
                     at dlmalloc (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[8788]:0x21d1e6)
                     at dlcalloc (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[8796]:0x21f96a)
                     at monoeg_g_calloc (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[598]:0x49521)
                     at monoeg_malloc0 (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[599]:0x49544)
                     at monoeg_g_array_new (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[569]:0x48b40)
                     at mono_interp_transform_method (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[438]:0x407b6)
                     at do_transform_method (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[255]:0x25481)
                     at mono_interp_exec_method (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[254]:0x251fb)
                     at interp_entry (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[327]:0x28391)
                     at interp_entry_static_2 (http://127.0.0.1:46791/_framework/dotnet.native.wasm:wasm-function[365]:0x29411)
lewing commented 6 months ago

cc @vargaz @steveisok

pavelsavara commented 6 months ago

local run with nice stack trace

fail: [0x05a6fef0-jsww 15:29:09.596] MONO_WASM: memory access out of bounds
  RuntimeError: memory access out of bounds
      at dlfree (dlfree (http://127.0.0.1:61068/_framework/dotnet.native.wasm:wasm-function[17333]:0x34eda4))
      at bound_fn (http://127.0.0.1:61068/_framework/dotnet.runtime.js:4146:24)
      at sync_bound_fn (http://127.0.0.1:61068/_framework/dotnet.runtime.js:3980:13)
      at JSImport_INTERNAL_http_wasm_supports_streaming_request (https://dotnet/JSImport/INTERNAL_http_wasm_supports_streaming_request:4:86)
      at mono_wasm_invoke_jsimport (http://127.0.0.1:61068/_framework/dotnet.runtime.js:3893:5)
      at do_icall (do_icall (http://127.0.0.1:61068/_framework/dotnet.native.wasm:wasm-function[231]:0x29de6))
      at do_icall_wrapper (do_icall_wrapper (http://127.0.0.1:61068/_framework/dotnet.native.wasm:wasm-function[151]:0x26971))
      at mono_interp_exec_method (mono_interp_exec_method (http://127.0.0.1:61068/_framework/dotnet.native.wasm:wasm-function[142]:0x150ac))
      at interp_runtime_invoke (interp_runtime_invoke (http://127.0.0.1:61068/_framework/dotnet.native.wasm:wasm-function[185]:0x27d50))
      at mono_jit_runtime_invoke (mono_jit_runtime_invoke (http://127.0.0.1:61068/_framework/dotnet.native.wasm:wasm-function[9174]:0x200a97))
pavelsavara commented 5 months ago

another local with great log


MONO_WASM: memory access out of bounds
RuntimeError: memory access out of bounds
    at copy_object_no_checks.1 (copy_object_no_checks.1 (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[2325]:0xa15c1))
    at simple_nursery_serial_scan_object (simple_nursery_serial_scan_object (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[2321]:0x9f935))
    at simple_nursery_serial_drain_gray_stack (simple_nursery_serial_drain_gray_stack (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[2324]:0xa1454))
    at sgen_drain_gray_stack (sgen_drain_gray_stack (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[1721]:0x7c5ca))
    at finish_gray_stack (finish_gray_stack (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[1828]:0x812be))
    at collect_nursery (collect_nursery (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[1744]:0x7d6ee))
    at sgen_perform_collection_inner (sgen_perform_collection_inner (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[1741]:0x7d11f))
    at sgen_perform_collection (sgen_perform_collection (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[1740]:0x7d001))
    at sgen_ensure_free_space (sgen_ensure_free_space (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[1739]:0x7cfac))
    at sgen_alloc_obj_nolock (sgen_alloc_obj_nolock (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[1669]:0x7af3e))```
pavelsavara commented 5 months ago

another local


RuntimeError: memory access out of bounds
    at dlfree (dlfree (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[17410]:0x34e69e))
    at bound_fn (https://localhost:65453/_framework/dotnet.runtime.js:4217:24)
    at async_bound_fn (https://localhost:65453/_framework/dotnet.runtime.js:4027:9)
    at JSImport_INTERNAL_dynamic_import (https://dotnet/JSImport/INTERNAL_dynamic_import:4:64)
    at mono_wasm_invoke_jsimport_MT (https://localhost:65453/_framework/dotnet.runtime.js:3948:5)
    at do_icall (do_icall (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[227]:0x29b34))
    at do_icall_wrapper (do_icall_wrapper (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[147]:0x266c6))
    at mono_interp_exec_method (mono_interp_exec_method (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[138]:0x150fd))
    at interp_runtime_invoke (interp_runtime_invoke (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[181]:0x27aa5))
    at mono_jit_runtime_invoke (mono_jit_runtime_invoke (https://localhost:65453/_framework/dotnet.native.wasm:wasm-function[9240]:0x2001ac))```
pavelsavara commented 5 months ago

Most of the problems are fixed by https://github.com/dotnet/runtime/pull/100304

There is new occurrence as fallout of OOM Log

[19:33:04] info: [FAIL] System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers(options: System.Text.Json.JsonWriterOptions, keyString: "mess><age")
[19:33:04] info: System.OutOfMemoryException : Out of memory
[19:33:04] info:    at System.GC.AllocateArray[Char](Int32 length, Boolean pinned)
[19:33:04] info:    at System.Text.StringBuilder.ExpandByABlock(Int32 minBlockCharCount)
[19:33:04] info:    at System.Text.StringBuilder.AppendWithExpansion(Char value)
[19:33:04] info:    at System.Text.StringBuilder.Append(Char value)
[19:33:04] info:    at System.IO.StringWriter.Write(Char value)
[19:33:04] info:    at Newtonsoft.Json.JsonTextWriter.WriteEnd(JsonToken token)
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.AutoCompleteClose(JsonContainerType type)
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.WriteEndArray()
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.WriteEnd(JsonContainerType type)
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.WriteEnd()
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.AutoCompleteAll()
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.Close()
[19:33:04] info:    at Newtonsoft.Json.JsonTextWriter.Close()
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.Dispose(Boolean disposing)
[19:33:04] info:    at Newtonsoft.Json.JsonWriter.System.IDisposable.Dispose()
[19:33:04] info:    at System.Text.Json.Tests.Utf8JsonWriterTests.GetExpectedLargeArrayOfStrings(Int32 length)
[19:33:04] info:    at System.Text.Json.Tests.Utf8JsonWriterTests.WriteJsonOnlyWritesToStreamOnDemand_FlushAsync()
[19:33:04] info: --- End of stack trace from previous location ---
[19:33:04] info:    at System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers(JsonWriterOptions options, String keyString)
[19:33:04] info:    at System.Object.InvokeStub_Utf8JsonWriterTests.WriteNumbers(Object , Span`1 )
[19:33:04] info:    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
[19:33:04] fail: [0x020cdfc0-dpty 19:33:04.706] MONO_WASM: Unhandled error: RuntimeError: memory access out of bounds
[19:33:04] fail: [0x020cdfc0-dpty 19:33:04.707] MONO_WASM: Unhandled error: RuntimeError: memory access out of bounds
[19:33:04] info: [0x020cdfc0-dpty 19:33:04.710] MONO_WASM: Dumping web worker info as seen by UI thread, it could be stale:
[19:33:04] info: 001 | 0x020cdfc0-dpty: isRunning: true isAttached: true isEventLoop:false reuseCount:  1 -
[19:33:04] fail: [0x020cdfc0-dpty 19:33:04.748] MONO_WASM: memory access out of bounds
                 RuntimeError: memory access out of bounds
                     at copy_object_no_checks.1 (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[1115]:0x637d7)
                     at simple_nursery_serial_scan_object (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[1113]:0x63003)
                     at simple_nursery_serial_drain_gray_stack (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[1110]:0x626d8)
                     at finish_gray_stack (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[955]:0x56d6a)
                     at collect_nursery (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[927]:0x5481b)
                     at sgen_perform_collection (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[925]:0x540da)
                     at sgen_ensure_free_space (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[924]:0x53f75)
                     at sgen_alloc_obj_nolock (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[911]:0x533bc)
                     at mono_gc_alloc_string (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[3174]:0xf0e73)
                     at mono_string_new_size_checked (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[2645]:0xd1efe)
                     at mono_interp_exec_method (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[257]:0x208b2)
                     at interp_entry (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[330]:0x286f9)
                     at interp_entry_static_0 (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[366]:0x29702)
                     at wasm_native_to_interp_System_Runtime_InteropServices_JavaScript_JSSynchronizationContext_BackgroundJobHandler (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[232]:0x1676e)
                     at mono_background_exec (http://127.0.0.1:35135/_framework/dotnet.native.wasm:wasm-function[836]:0x50e85)
                     at Os (http://127.0.0.1:35135/_framework/dotnet.runtime.js:3:69709)
                     at callUserCallback (http://127.0.0.1:35135/_framework/dotnet.native.js:8:102608)
                     at http://127.0.0.1:35135/_framework/dotnet.native.js:8:111163
[19:33:04] info: WASM EXIT 1
kg commented 5 months ago

This incorrectly ate https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-100801-merge-8c5cd227f4844ef98c/Methodical_d2/1/console.e95f9cb2.log?helixlogtype=result from a non-MT lane, which looks like it could be a real issue.

pavelsavara commented 5 months ago

This incorrectly ate https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-100801-merge-8c5cd227f4844ef98c/Methodical_d2/1/console.e95f9cb2.log?helixlogtype=result from a non-MT lane, which looks like it could be a real issue.

I already opened https://github.com/dotnet/runtime/issues/100757 for it

lewing commented 4 months ago

this appears to have been fixed with https://github.com/dotnet/runtime/pull/100907 closing so that new failures are caught

lewing commented 4 months ago

hit again in https://github.com/dotnet/runtime/pull/101179

pavelsavara commented 4 months ago

There are 4 more this morning, 3 of them are after OOM. The 4th is in GC (probably on the edge of OOM too) https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-101122-merge-f513e5fd0d4a45d08e/WasmTestOnBrowser-System.Text.Json.Tests/1/console.93cb60d2.log?helixlogtype=result

Perhaps some alloc failed and returned NULL and we are not checking it ?

pavelsavara commented 4 months ago

Alternatively, are we missing some memory fence which would synchronize the memory between WASM threads ?

We do issue .grow(0) in mono_wasm_synchronization_context_pump but sgen_perform_collection could be doing lot of work and the memory could grow on another thread in the meantime.

[09:03:07] fail: [0x020ff9e8-dpty 09:03:07.808] MONO_WASM: memory access out of bounds
                 RuntimeError: memory access out of bounds
                     at copy_object_no_checks.1 (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[1119]:0x677c9)
                     at simple_nursery_serial_scan_object (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[1117]:0x66fe9)
                     at simple_nursery_serial_drain_gray_stack (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[1114]:0x666be)
                     at finish_gray_stack (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[957]:0x5a74b)
                     at collect_nursery (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[930]:0x5813f)
                     at sgen_perform_collection (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[928]:0x579f4)
                     at sgen_ensure_free_space (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[927]:0x5788d)
                     at sgen_alloc_obj_nolock (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[914]:0x56d40)
                     at mono_gc_alloc_vector (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[3167]:0xfab94)
                     at mono_array_new_specific_internal (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[2710]:0xddfeb)
                     at mono_array_new_specific_checked (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[2711]:0xde022)
                     at mono_interp_exec_method (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[257]:0x21463)
                     at interp_entry (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[330]:0x297e8)
                     at interp_entry_static_0 (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[366]:0x2a830)
                     at wasm_native_to_interp_System_Runtime_InteropServices_JavaScript_JSSynchronizationContext_PumpHandler (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[232]:0x171f1)
                     at mono_wasm_synchronization_context_pump (http://127.0.0.1:45371/_framework/dotnet.native.wasm:wasm-function[206]:0x1556e)
kg commented 4 months ago

I think under normal circumstances we shouldn't need to .grow(0), the theory was that we specifically needed to do it if the debugger paused thread(s) while a grow operation happened, wasn't it?

Unchecked null pointer feels very plausible. It will fail silently and also corrupt the zero page (which will cause jiterp traces to break.)

pavelsavara commented 4 months ago

I saw Log

[13:54:11] fail: Error in mono_download_assets: RangeError: Start offset -1 is outside the bounds of the buffer
[13:54:11] fail: [--000eafbc-emsc 13:54:11.333] MONO_WASM: Start offset -1 is outside the bounds of the buffer
                 RangeError: Start offset -1 is outside the bounds of the buffer
                     at new Uint8Array (<anonymous>)
                     at http://127.0.0.1:41625/_framework/dotnet.runtime.js:3:77638
                     at Object.pa [as instantiate_asset] (http://127.0.0.1:41625/_framework/dotnet.runtime.js:3:77686)
                     at n (http://127.0.0.1:41625/_framework/dotnet.js:3:8629)
                     at async Promise.all (index 129)
[13:54:11] fail: [out of order message from the browser]: http://127.0.0.1:41625/_framework/dotnet.runtime.js 2:77637 Uncaught RangeError: Start offset -1 is outside the bounds of the buffer
[13:54:11] info: WASM EXIT 1

This translates to ICU asset data being loaded into memory via mono_wasm_load_bytes_into_heap_persistent on MT build in UI thread, while the mono is starting on deputy thread.

It seems that Module._sbrk returned -1 🤯

It seems that sbrk could be racing with other threads and loose. https://github.com/emscripten-core/emscripten/blob/1ca2f3141ec99110d6c5c5690dcd1c65d08d97e5/system/lib/libc/sbrk.c#L75

Also I'm not sure on what version of emscripten we are now W.R.T. https://github.com/emscripten-core/emscripten/pull/20793/files

We are just lucky it was called from JS and that we got nice JS exception. I bet the same could happen to our C code as well. Which would mean we could get RuntimeError: memory access out of bounds unless all our code is handling -1 and NULL returns from sbrk and malloc

kg commented 1 month ago

It looks like this was originally intended to match MT failures, but based on the recent matches in the automated list, it's matching ST failures. That's not good, memory access out of bounds on ST is more likely to be real bugs that we introduced in PRs and now it will be green.

pavelsavara commented 1 month ago

They are JS error, it will not capture the same issue if it happens on Windows target for example.

When access out of bounds happens in ST it only makes troubleshooting MT more difficult.

lewing commented 3 weeks ago

This has transitioned to ST only then dropped off after the alignment fix 🤞

ilonatommy commented 1 week ago

Not being hit after alignment PR: https://github.com/dotnet/runtime/pull/106313.