search

opensumi / core

🚀 A framework helps you quickly build AI Native IDE products.
https://opensumi.com
MIT License
2.98k stars 382 forks source link

[BUG] 打开文件导致后端进程退出 #3119

Open sleepyyj opened 1 year ago

sleepyyj commented 1 year ago

描述你的问题（Describe the bug） 使用 ide-startup 测试，前端打开一个文件后，后端进程由于 OOM 退出。

<--- Last few GCs --->

[30947:0x130008000]   536403 ms: Mark-sweep 394.3 (434.6) -> 370.8 (410.2) MB, 71.1 / 0.0 ms  (+ 0.0 ms in 3 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 396 ms) (average mu = 0.999, current mu = 0.853) finalize incre[30947:0x130008000]   539844 ms: Mark-sweep 1714.8 (1754.3) -> 1546.8 (1586.2) MB, 4.0 / 0.0 ms  (+ 0.0 ms in 3 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 3118 ms) (average mu = 0.999, current mu = 0.999) finalize i

<--- JS stacktrace --->

 3: 0x1024e0000 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 4: 0x1024dff94 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 5: 0x1026636ac v8::internal::Heap::GarbageCollectionReasonToString(v8::internal::GarbageCollectionReason) [/usr/local/bin/node]
 6: 0x10286ffb8 v8::internal::Handle<v8::internal::NumberDictionary> v8::internal::HashTable<v8::internal::NumberDictionary, v8::internal::NumberDictionaryShape>::EnsureCapacity<v8::internal::LocalIsolate>(v8::internal::LocalIsolate*, v8::internal::Handle<v8::internal::NumberDictionary>, int, v8::internal::AllocationType) [/usr/local/bin/node]
 7: 0x102870830 v8::internal::Handle<v8::internal::NumberDictionary> v8::internal::Dictionary<v8::internal::NumberDictionary, v8::internal::NumberDictionaryShape>::Add<v8::internal::Isolate>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::NumberDictionary>, unsigned int, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyDetails, v8::internal::InternalIndex*) [/usr/local/bin/node]
 8: 0x1027b4908 v8::internal::(anonymous namespace)::DictionaryElementsAccessor::AddImpl(v8::internal::Handle<v8::internal::JSObject>, unsigned int, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, unsigned int) [/usr/local/bin/node]
 9: 0x1027b37dc v8::internal::(anonymous namespace)::ElementsAccessorBase<v8::internal::(anonymous namespace)::DictionaryElementsAccessor, v8::internal::(anonymous namespace)::ElementsKindTraits<(v8::internal::ElementsKind)12>>::Add(v8::internal::Handle<v8::internal::JSObject>, unsigned int, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, unsigned int) [/usr/local/bin/node]
10: 0x10283d850 v8::internal::JSObject::AddDataElement(v8::internal::Handle<v8::internal::JSObject>, unsigned int, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes) [/usr/local/bin/node]
11: 0x102880a04 v8::internal::Object::AddDataProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::Maybe<v8::internal::ShouldThrow>, v8::internal::StoreOrigin) [/usr/local/bin/node]
12: 0x10287f5c4 v8::internal::Object::SetProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::StoreOrigin, v8::Maybe<v8::internal::ShouldThrow>) [/usr/local/bin/node]
13: 0x10297f044 v8::internal::Runtime::SetObjectProperty(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::StoreOrigin, v8::Maybe<v8::internal::ShouldThrow>) [/usr/local/bin/node]
14: 0x1026fd498 v8::internal::Runtime_KeyedStoreIC_Slow(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node]
15: 0x102c8cbcc Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/usr/local/bin/node]
16: 0x1071cb2dc 
17: 0x102c1e64c Builtins_JSEntryTrampoline [/usr/local/bin/node]
18: 0x102c1e2e4 Builtins_JSEntry [/usr/local/bin/node]
19: 0x1025efe94 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/usr/local/bin/node]
20: 0x1025ef528 v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) [/usr/local/bin/node]
21: 0x1027554c4 v8::internal::JsonStringifier::ApplyToJsonFunction(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/usr/local/bin/node]
22: 0x10275c030 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/usr/local/bin/node]
23: 0x10275e74c v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/usr/local/bin/node]
24: 0x10275adf8 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/usr/local/bin/node]
25: 0x1027549bc v8::internal::JsonStringify(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/usr/local/bin/node]
26: 0x102569f40 v8::internal::Builtin_JsonStringify(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node]
27: 0x102c8cd0c Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit [/usr/local/bin/node]
28: 0x1073a9974 
29: 0x107394f78 
30: 0x10733a4c8 
31: 0x102cd4cf8 Builtins_PromiseFulfillReactionJob [/usr/local/bin/node]
32: 0x102c42234 Builtins_RunMicrotasks [/usr/local/bin/node]
33: 0x102c1e524 Builtins_JSRunMicrotasksEntry [/usr/local/bin/node]
34: 0x1025efe5c v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/usr/local/bin/node]
35: 0x1025f0290 v8::internal::(anonymous namespace)::InvokeWithTryCatch(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/usr/local/bin/node]
36: 0x1025f037c v8::internal::Execution::TryRunMicrotasks(v8::internal::Isolate*, v8::internal::MicrotaskQueue*, v8::internal::MaybeHandle<v8::internal::Object>*) [/usr/local/bin/node]
37: 0x102612fb8 v8::internal::MicrotaskQueue::RunMicrotasks(v8::internal::Isolate*) [/usr/local/bin/node]
38: 0x10261384c v8::internal::MicrotaskQueue::PerformCheckpoint(v8::Isolate*) [/usr/local/bin/node]
39: 0x1022dddb4 node::InternalCallbackScope::Close() [/usr/local/bin/node]
40: 0x1022de48c node::InternalMakeCallback(node::Environment*, v8::Local<v8::Object>, v8::Local<v8::Object>, v8::Local<v8::Function>, int, v8::Local<v8::Value>*, node::async_context) [/usr/local/bin/node]
41: 0x1022f34c8 node::AsyncWrap::MakeCallback(v8::Local<v8::Function>, int, v8::Local<v8::Value>*) [/usr/local/bin/node]
42: 0x1023949d0 node::fs::FSReqCallback::Resolve(v8::Local<v8::Value>) [/usr/local/bin/node]
43: 0x102395128 node::fs::AfterNoArgs(uv_fs_s*) [/usr/local/bin/node]
44: 0x102bfebd4 uv__work_done [/usr/local/bin/node]
45: 0x102c02370 uv__async_io [/usr/local/bin/node]
46: 0x102c140e8 uv__io_poll [/usr/local/bin/node]
47: 0x102c02800 uv_run [/usr/local/bin/node]
48: 0x1022deccc node::SpinEventLoop(node::Environment*) [/usr/local/bin/node]
49: 0x1023c9aec node::NodeMainInstance::Run(int*, node::Environment*) [/usr/local/bin/node]
50: 0x1023c97b8 node::NodeMainInstance::Run(node::EnvSerializeInfo const*) [/usr/local/bin/node]
51: 0x1023635c8 node::Start(int, char**) [/usr/local/bin/node]
52: 0x19bbfbf28 start [/usr/lib/dyld]

复现路径（To Reproduce） 1、启动 ide-startup 2、添加一个文件到 workspace，文件大小根据机器配置 20M~200M 不等 3、前端打开这个文件

预期表现（Expected behavior） 能正常打开文件

环境信息（Environment）

erha19 commented 1 year ago

@sleepyyj 是否测试过如 2.26，2.27 版本可不可以正常使用？看了一下目前集成的版本是比较旧的

sleepyyj commented 1 year ago

@sleepyyj 是否测试过如 2.26，2.27 版本可不可以正常使用？看了一下目前集成的版本是比较旧的

我测一下

sleepyyj commented 1 year ago

@erha19 2.26.8 和 2.27.0 都是一样的报错

erha19 commented 1 year ago

@sleepyyj 这里应该存在大文件打开时的内存泄漏问题，（即部分服务端环境下的机器确实无法承载大文件的语法高亮、缓存等）暂时你可以先通过设置 editor.largeFile 小一些来控制其展示，同时关闭大文件内的语法高亮来减少 OOM 问题，配置项如：

  'editor.languageFeatureEnabledMaxSize': {
    type: 'number',
    default: 4 * 1024 * 1024 * 1024, // 4096 MB
    description: '%editor.configuration.languageFeatureEnabledMaxSize%',
  },
  // 会同步到插件进程的最大文件尺寸, 必须大于等于 languageFeatureEnabledMaxSize
  'editor.docExtHostSyncMaxSize': {
    type: 'number',
    default: 4 * 1024 * 1024 * 1024, // 4096 MB
    description: '%editor.configuration.docExtHostSyncMaxSize%',
  },

相关配置项：

https://github.com/opensumi/core/blob/d10597dd539a36aa31012f091b16c71ed80ed252/packages/editor/src/browser/preference/schema.ts#L1474-L1484

降低这类配置会让宿主机器的表现会好一些

sleepyyj commented 1 year ago

@erha19 把配置大小改为 20MB 后，打开文件提示 The file is too large, continuing to open it may cause it to jam or crash.，点击 still open it。146MB 的文件无法打开，没有出现OOM，但是 readFile 消息没有响应。263MB 的文件依然 OOM。

erha19 commented 1 year ago

@sleepyyj 在 opensumi/core 中也复现了这个问题，本质原因是目前框架使用了 jsonRPC 这一通信结构，好处是在 WS 中能方便的进行问题调试，坏处就是像这里这种情况（或者说文件读取情况），读取文件获得的 Buffer 数据在经过 JSON.stringify 及 JSON.parse 过程会耗费大量的内存，导致内存泄漏问题。

该问题要彻底解决，需要通过对通信层协议进行改造来解决这一问题，在 架构改进 中有这块规划，但目前应该没有进度。

相关问题代码见：

https://github.com/opensumi/core/blob/73a869a50919a03768c1c707011ec28408950129/packages/connection/src/common/message.ts#L75

cc @bytemain