search

dart-lang / sdk

The Dart SDK, including the VM, JS and Wasm compilers, analysis, core libraries, and more.
https://dart.dev
BSD 3-Clause "New" or "Revised" License
10.2k stars 1.57k forks source link

../../runtime/vm/thread_pool.cc: 299: error: Could not start worker thread: result = 22. #46787

Open bubnenkoff opened 3 years ago

bubnenkoff commented 3 years ago 
../../runtime/vm/thread_pool.cc: 299: error: Could not start worker thread: result = 22.
version=2.14.0-363.0.dev (dev) (Wed Jul 28 23:43:25 2021 -0700) on "windows_x64"
pid=5636, thread=3800, isolate_group=main(00000085825B3A90), isolate=(nil)(0000000000000000)
isolate_instructions=7ff726c54e40, vm_instructions=7ff726c54e50
  pc 0x00007ff726e50052 fp 0x000000858523e080 Dart_IsPrecompiledRuntime+0x20f962
-- End of DumpStackTrace
[exit     : sp(0) fp(0x858523e7e0) pc(0)]
[dart     : sp(0x858523e7f0) fp(0x858523e830) pc(0x858be74189) dart:core__StringBase@0150898__substringUncheckedNative@0150898 ]
[dart     : sp(0x858523e840) fp(0x858523e888) pc(0x85a088a1b3) *dart:core__StringBase@0150898_substring ]
[dart     : sp(0x858523e898) fp(0x858523e8c8) pc(0x85a088b11e) *dart:convert__JsonStringParser@9003594_addSliceToString ]
[dart     : sp(0x858523e8d8) fp(0x858523e918) pc(0x858be7f79c) *dart:convert__ChunkedJsonParser@9003594_parseStringToBuffer ]
[dart     : sp(0x858523e928) fp(0x858523e960) pc(0x85a088e0d3) *dart:convert__ChunkedJsonParser@9003594_parseString ]
[dart     : sp(0x858523e970) fp(0x858523e9c8) pc(0x85a08917d1) *dart:convert__ChunkedJsonParser@9003594_parse ]
[dart     : sp(0x858523e9d8) fp(0x858523ea08) pc(0x85a08b7bdf) *dart:convert_::__parseJson@9003594 ]
[dart     : sp(0x858523ea18) fp(0x858523eb38) pc(0x85a08c618f) *package:dart_express/dart_express.dart_BodyParser_convertBodyToJson__async_op ]
[dart     : sp(0x858523eb48) fp(0x858523eb98) pc(0x85a0892aac) *dart:async__FutureListener@4048458_handleValue ]
[dart     : sp(0x858523eba8) fp(0x858523ec10) pc(0x85a0894246) *dart:async__Future@4048458__propagateToListeners@4048458_handleValueCallback ]
[dart     : sp(0x858523ec20) fp(0x858523ec78) pc(0x85a088f164) *dart:async__Future@4048458__propagateToListeners@4048458 ]
[dart     : sp(0x858523ec88) fp(0x858523ecc8) pc(0x85a08aefe6) *dart:async__Future@4048458__complete@4048458 ]
[dart     : sp(0x858523ecd8) fp(0x858523ed08) pc(0x85a08c8087) *dart:async_Stream_join_<anonymous closure> ]
[dart     : sp(0x858523ed18) fp(0x858523ed70) pc(0x85a08adb1c) *dart:async__RootZone@4048458_runGuarded ]
[dart     : sp(0x858523ed80) fp(0x858523edb0) pc(0x85a089de20) *dart:async__BufferingStreamSubscription@4048458__sendDone@4048458 ]
[dart     : sp(0x858523edc0) fp(0x858523ede8) pc(0x85a08dae52) *dart:convert__Utf8ConversionSink@9003594_close ]
[dart     : sp(0x858523edf8) fp(0x858523ee50) pc(0x85a08db17d) *dart:async__SinkTransformerStreamSubscription@4048458__handleDone@4048458 ]
[dart     : sp(0x858523ee60) fp(0x858523ee80) pc(0x85a08c84c8) *dart:async__SinkTransformerStreamSubscription@4048458__handleDone@4048458__handleDone@4048458 ]
[dart     : sp(0x858523ee90) fp(0x858523eee8) pc(0x85a08adb1c) *dart:async__RootZone@4048458_runGuarded ]
[dart     : sp(0x858523eef8) fp(0x858523ef28) pc(0x85a089de20) *dart:async__BufferingStreamSubscription@4048458__sendDone@4048458 ]
[dart     : sp(0x858523ef38) fp(0x858523ef68) pc(0x85a08c841f) *dart:async__ForwardingStreamSubscription@4048458__handleDone@4048458__handleDone@4048458 ]
[dart     : sp(0x858523ef78) fp(0x858523efd0) pc(0x85a08adb1c) *dart:async__RootZone@4048458_runGuarded ]
[dart     : sp(0x858523efe0) fp(0x858523f010) pc(0x85a089de20) *dart:async__BufferingStreamSubscription@4048458__sendDone@4048458 ]
[dart     : sp(0x858523f020) fp(0x858523f048) pc(0x85a08a458d) *dart:async__StreamController@4048458_close ]
[dart     : sp(0x858523f058) fp(0x858523f090) pc(0x85a08e99e3) *dart:_http__HttpParser@16463476__closeIncoming@16463476 ]
[dart     : sp(0x858523f0a0) fp(0x858523f130) pc(0x858be49847) *dart:_http__HttpParser@16463476__doParse@16463476 ]
[dart     : sp(0x858523f140) fp(0x858523f198) pc(0x85a08c94b8) *dart:_http__HttpParser@16463476__parse@16463476 ]
[dart     : sp(0x858523f1a8) fp(0x858523f1d0) pc(0x85a08b99b0) *dart:_http__HttpParser@16463476__onData@16463476__onData@16463476 ]
[dart     : sp(0x858523f1e0) fp(0x858523f260) pc(0x85a089f29d) *dart:async__RootZone@4048458_runUnaryGuarded ]
[dart     : sp(0x858523f270) fp(0x858523f2b0) pc(0x85a088a04e) *dart:async__BufferingStreamSubscription@4048458__sendData@4048458 ]
[dart     : sp(0x858523f2c0) fp(0x858523f300) pc(0x85a089987e) *dart:async__StreamController@4048458__add@4048458 ]
[dart     : sp(0x858523f310) fp(0x858523f338) pc(0x85a08ab676) *dart:io__Socket@14069316__onData@14069316 ]
[dart     : sp(0x858523f348) fp(0x858523f370) pc(0x85a08a790b) *dart:io__Socket@14069316__onData@14069316__onData@14069316 ]
[dart     : sp(0x858523f380) fp(0x858523f400) pc(0x85a089f29d) *dart:async__RootZone@4048458_runUnaryGuarded ]
[dart     : sp(0x858523f410) fp(0x858523f450) pc(0x85a088a04e) *dart:async__BufferingStreamSubscription@4048458__sendData@4048458 ]
[dart     : sp(0x858523f460) fp(0x858523f4a0) pc(0x85a089987e) *dart:async__StreamController@4048458__add@4048458 ]
[dart     : sp(0x858523f4b0) fp(0x858523f4e0) pc(0x85a08a94cd) *dart:io__RawSocket@14069316__RawSocket@14069316._<anonymous closure> ]
[dart     : sp(0x858523f4f0) fp(0x858523f5b0) pc(0x85a08b658e) *dart:core__Closure@0150898_dyn_call ]
[dart     : sp(0x858523f5c0) fp(0x858523f5e8) pc(0x85a08aec2b) *dart:io__NativeSocket@14069316_issueReadEvent_issue ]
[dart     : sp(0x858523f5f8) fp(0x858523f650) pc(0x85a08b3fe4) *dart:async_::__startMicrotaskLoop@4048458 ]
[dart     : sp(0x858523f660) fp(0x858523f678) pc(0x85a08a64c3) *dart:async_::__startMicrotaskLoop@4048458__startMicrotaskLoop@4048458 ] [dart     : sp(0x858523f688) fp(0x858523f6b0) pc(0x85a0898a0f) *dart:isolate__RawReceivePortImpl@1026248__handleMessage@1026248 ]
[entry    : sp(0x858523f6c0) fp(0x858523f7d8) pc(0x85824025a8)]
dart --version
Dart SDK version: 2.14.0-363.0.dev (dev) (Wed Jul 28 23:43:25 2021 -0700) on "windows_x64"

The server is getting SQL commands to handler and run the queries:

  app.post('/commands', (req, res) async {
    try {
        var r = await db!.commandFromParser(req.body).timeout(Duration(seconds: 120));
        return res.json({'status': "success", "data": r});
    }

    catch(e) {
      print("[ERROR] in commands handle: $e");
    }
  });    

  Future<dynamic> commandFromParser(Map body) async {
    try {
      // print("commandFromParser: ${body['sql']}");
      List<List<dynamic>> result = await connection.query(body['sql']).timeout(Duration(seconds: 120));
      return result;
    } on PostgreSQLException catch (e) {
      writeLog('commandFromParser', e.message);
    } catch (e) {
      print('commandFromParser exception on e: ${e}');
      writeLog('commandFromParser', e.toString());
    }
  }

deps:

 dependencies:
 path: ^1.7.0
 dart_express: ^0.5.3
 postgres_pool: ^2.1.2

The server getting sql update commands and execute complete them. About 60 000 per hour. After few hours of working I am getting this crush. Same was on Dart 2.13.

https://github.com/dart-lang/sdk/issues/37155

a-siva commented 3 years ago

../../runtime/vm/thread_pool.cc: 299: error: Could not start worker thread: result = 22.

Does your application use multiple isolates for handling the requests? I was wondering if the app ended up creating too many threads and at this point has run out of resources to create more threads.

bubnenkoff commented 3 years ago

Only if pg_pool start them, i dont think that dart express do it.

Пн, 2 авг. 2021 г. в 21:24, Siva @.***>:

../../runtime/vm/thread_pool.cc: 299: error: Could not start worker thread: result = 22.

Does your application use multiple isolates for handling the requests? I was wondering if the app ended up creating too many threads and at this point has run out of resources to create more threads.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/dart-lang/sdk/issues/46787#issuecomment-891238133, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABRWNFVXB5RMTRL5TC3KDX3T23PHPANCNFSM5BNFXRTQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

aam commented 3 years ago

Can you check number of threads being used by the dart server as it is running? (if on Linux you can use top, on Windows - Resource Monitor) See if it grows over time?

bubnenkoff commented 3 years ago

I am seeing only 4 instances of dart after 4 hours of work: изображение

aam commented 3 years ago

@bubnenkoff it would be interesting to see number of threads created by a dart process, because the error you saw might indicate that dart process created/accumulated too many of them. Resource Monitor(normally %systemroot%\system32\resmon.exe) should show that: image

bubnenkoff commented 3 years ago

Currently (after ~3 hours of working) it's seems that not too much: изображение

mraleph commented 3 years ago

@bubnenkoff can you also check memory usage and handle count numbers? Additionally I'd suggest to trace the application using procmon (https://docs.microsoft.com/en-us/sysinternals/downloads/procmon) focusing on thread creation events to see if you can glimps why underlying CreateThread fails.

AlexV525 commented 2 years ago

I'm also facing this on a specific Windows device. When it's running the compile command, you can see the sharp increase with the memory usage by a single dart.exe. IMG_20220102_192427

Platform and version: Windows 10 19042. Dart: 2.15.1 (stable).

praveeng2565 commented 1 year ago

I'm also facing this on a specific Windows device. When it's running the compile command, you can see the sharp increase with the memory usage by a single dart.exe. IMG_20220102_192427

Platform and version: Windows 10 19042. Dart: 2.15.1 (stable). Hi team,

Is there any update regarding this issue? currently Im also facing this issue.

FLUTTER STABLE : 3.10.6 DART : 3.0.6

mraleph commented 1 year ago

We have not been able to reproduce this issue or understand it's underlying cause - though we suspect it is an OOM situation.

If you are experiencing the issue consider helping us by recording a procmon trace and seeing if there any more information on the CreateThread failure in the trace.