search

Last-Order / Minyami

A lovely video downloader for HLS videos
GNU General Public License v3.0
561 stars 37 forks source link

下载过程中出现内存溢出问题 #33

Closed LoveCany closed 5 years ago

LoveCany commented 5 years ago

Error Report: <--- Last few GCs --->

[34348:0000019CBBBCD6B0] 3563341 ms: Mark-sweep 1393.1 (1408.7) -> 1393.1 (1408.7) MB, 104.6 / 0.0 ms (average mu = 0.893, current mu = 0.006) last resort GC in old space requested [34348:0000019CBBBCD6B0] 3563456 ms: Mark-sweep 1393.1 (1408.7) -> 1393.1 (1408.7) MB, 115.0 / 0.0 ms (average mu = 0.806, current mu = 0.001) last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

0: ExitFrame [pc: 00000282D35DC5C1]

Security context: 0x01a14311e6e9 1: stringSlice(aka stringSlice) [000000A00E913391] [buffer.js:594] [bytecode=000000A00E955331 offset=91](this=0x027b951026f1 ,buf=0x0080fdb06781 ,encoding=0x01a14313e2e1 <String[4]: utf8>,start=0,end=2743805) 2: toString [00000142C7320429] [buffer.js:667] [bytecode=000000A00E954E21 offset=145](this=0x00...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory 1: 00007FF7D027C6AA v8::internal::GCIdleTimeHandler::GCIdleTimeHandler+4506 2: 00007FF7D0257416 node::MakeCallback+4534 3: 00007FF7D0257D90 node_module_register+2032 4: 00007FF7D057189E v8::internal::FatalProcessOutOfMemory+846 5: 00007FF7D05717CF v8::internal::FatalProcessOutOfMemory+639 6: 00007FF7D0757F94 v8::internal::Heap::MaxHeapGrowingFactor+9620 7: 00007FF7D075645B v8::internal::Heap::MaxHeapGrowingFactor+2651 8: 00007FF7D08802BB v8::internal::Factory::AllocateRawWithImmortalMap+59 9: 00007FF7D0882C2D v8::internal::Factory::NewRawOneByteString+77 10: 00007FF7D0881D99 v8::internal::Factory::NewStringFromOneByte+121 11: 00007FF7D0881E7A v8::internal::Factory::NewStringFromUtf8+106 12: 00007FF7D058D09B v8::String::NewFromUtf8+363 13: 00007FF7D0245561 node::Buffer::New+24225 14: 00007FF7D0784382 std::vector<v8::internal::compiler::MoveOperands __ptr64,v8::internal::ZoneAllocator<v8::internal::compiler::MoveOperands ptr64> >::_Umove+79442 15: 00007FF7D078580D std::vector<v8::internal::compiler::MoveOperands __ptr64,v8::internal::ZoneAllocator<v8::internal::compiler::MoveOperands __ptr64> >::_Umove+84701 16: 00007FF7D0784866 std::vector<v8::internal::compiler::MoveOperands * ptr64,v8::internal::ZoneAllocator<v8::internal::compiler::MoveOperands * __ptr64> >::_Umove+80694

怀疑由于直播过长（10H+）出现内存溢出问题，希望能在之后的版本对这一问题进行解决

Last-Order commented 5 years ago

comfirmed. will be fixed soon. thanks for your report.

Last-Order commented 5 years ago

试着修了下应该有缓解，再看看

LoveCany commented 5 years ago

相比稍有缓解，但依旧在距离较近的部位出现了内存溢出

[MINYAMI][INFO] Start fetching M3U8 file. [MINYAMI][INFO] M3U8 file fetched. [MINYAMI][INFO] Proccessing 43558 finished. (3467 / unknown | Avg Speed: 1.18 chunks/s or 5.89x) [MINYAMI][INFO] Start fetching M3U8 file. [MINYAMI][INFO] Proccessing 43559 finished. (3468 / unknown | Avg Speed: 1.18 chunks/s or 5.89x) [MINYAMI][INFO] M3U8 file fetched. [MINYAMI][INFO] Start fetching M3U8 file. [MINYAMI][INFO] Proccessing 43560 finished. (3469 / unknown | Avg Speed: 1.17 chunks/s or 5.88x) [MINYAMI][INFO] M3U8 file fetched. [MINYAMI][INFO] Start fetching M3U8 file.

<--- Last few GCs --->

[13400:0x2b4c240] 2961098 ms: Mark-sweep 1395.8 (1406.7) -> 1395.8 (1405.7) MB, 29.8 / 0.0 ms (average mu = 0.971, current mu = 0.050) last resort GC in old space requested [13400:0x2b4c240] 2961127 ms: Mark-sweep 1395.8 (1405.7) -> 1395.8 (1405.7) MB, 29.1 / 0.0 ms (average mu = 0.944, current mu = 0.001) last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

0: ExitFrame [pc: 0x1a32b45be1d]

Security context: 0x18bb28ec5861 1: stringSlice(aka stringSlice) [0x21ee57293429] [buffer.js:594] [bytecode=0x21ee572d24a1 offset=91](this=0x3ff883f026f1 ,buf=0x244c128b1a71 ,encoding=0x18bb28eda2c1 <String[4]: utf8>,start=0,end=2749294) 2: toString [0xac0bf762221] [buffer.js:667] [bytecode=0x21ee572d1f91 offset=145](this=0x244c128b1a71 <Uin...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory 1: 0x8fa060 node::Abort() [node] 2: 0x8fa0ac [node] 3: 0xaff7ee v8::Utils::ReportOOMFailure(v8::internal::Isolate, char const, bool) [node] 4: 0xaffa24 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate, char const, bool) [node] 5: 0xef3f32 [node] 6: 0xf0374f v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node] 7: 0xecb995 [node] 8: 0xed320a v8::internal::Factory::NewRawOneByteString(int, v8::internal::PretenureFlag) [node] 9: 0xed351b v8::internal::Factory::NewStringFromOneByte(v8::internal::Vector, v8::internal::PretenureFlag) [node] 10: 0xed3f6d v8::internal::Factory::NewStringFromUtf8(v8::internal::Vector, v8::internal::PretenureFlag) [node] 11: 0xb0d749 v8::String::NewFromUtf8(v8::Isolate, char const, v8::NewStringType, int) [node] 12: 0x9c16b8 node::StringBytes::Encode(v8::Isolate, char const, unsigned long, node::encoding, v8::Local*) [node] 13: 0x9164b0 [node] 14: 0xb8dc2f [node] 15: 0xb8e799 v8::internal::Builtin_HandleApiCall(int, v8::internal::Object*, v8::internal::Isolate) [node] 16: 0x1a32b45be1d Aborted

Last-Order commented 5 years ago

草 还真有点难调；我放弃用Promise.race了不知道现在会不会好一点。

LoveCany commented 5 years ago

Test Stream Link:https://www.youtube.com/watch?v=jicnqgYEwGI Error Report: [MINYAMI][INFO] Start fetching M3U8 file. [MINYAMI][INFO] M3U8 file fetched. [MINYAMI][INFO] Proccessing 7504 finished. (4120 / unknown | Avg Speed: 3.94 chunks/s or 7.88x) [MINYAMI][INFO] Proccessing 7505 finished. (4121 / unknown | Avg Speed: 3.94 chunks/s or 7.88x) [MINYAMI][INFO] Start fetching M3U8 file. [MINYAMI][INFO] M3U8 file fetched. [MINYAMI][INFO] Proccessing 7506 finished. (4122 / unknown | Avg Speed: 3.93 chunks/s or 7.87x) [MINYAMI][INFO] Start fetching M3U8 file. [MINYAMI][INFO] Proccessing 7507 finished. (4123 / unknown | Avg Speed: 3.93 chunks/s or 7.85x) [MINYAMI][INFO] M3U8 file fetched. [MINYAMI][INFO] Start fetching M3U8 file. [MINYAMI][INFO] M3U8 file fetched. [MINYAMI][INFO] Proccessing 7508 finished. (4124 / unknown | Avg Speed: 3.92 chunks/s or 7.83x) [MINYAMI][INFO] Start fetching M3U8 file.

<--- Last few GCs --->

[15501:0x3cc2340] 1055882 ms: Mark-sweep 1394.5 (1405.1) -> 1394.5 (1404.1) MB, 23.9 / 0.0 ms (average mu = 0.910, current mu = 0.104) last resort GC in old space requested [15501:0x3cc2340] 1055918 ms: Mark-sweep 1394.5 (1404.1) -> 1394.5 (1404.1) MB, 36.5 / 0.0 ms (average mu = 0.808, current mu = 0.001) last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

0: ExitFrame [pc: 0x2dcb684dbe1d]

Security context: 0x01d37ec777d9 1: stringSlice(aka stringSlice) [0xc8eb5c30b61] [buffer.js:594] [bytecode=0xc8eb5c697d9 offset=91](this=0x223b942826f1 ,buf=0x3b95040a91b1 ,encoding=0x303f3736be79 <String[4]: utf8>,start=0,end=3389374) 2: toString [0x2f68a81ed369] [buffer.js:667] [bytecode=0xc8eb5c692c9 offset=145](this=0x3b95040a91b1 <Uin...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory 1: 0x8fa060 node::Abort() [node] 2: 0x8fa0ac [node] 3: 0xaff7ee v8::Utils::ReportOOMFailure(v8::internal::Isolate, char const, bool) [node] 4: 0xaffa24 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate, char const, bool) [node] 5: 0xef3f32 [node] 6: 0xf0374f v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node] 7: 0xecb995 [node] 8: 0xed320a v8::internal::Factory::NewRawOneByteString(int, v8::internal::PretenureFlag) [node] 9: 0xed351b v8::internal::Factory::NewStringFromOneByte(v8::internal::Vector, v8::internal::PretenureFlag) [node] 10: 0xed3f6d v8::internal::Factory::NewStringFromUtf8(v8::internal::Vector, v8::internal::PretenureFlag) [node] 11: 0xb0d749 v8::String::NewFromUtf8(v8::Isolate, char const, v8::NewStringType, int) [node] 12: 0x9c16b8 node::StringBytes::Encode(v8::Isolate, char const, unsigned long, node::encoding, v8::Local*) [node] 13: 0x9164b0 [node] 14: 0xb8dc2f [node] 15: 0xb8e799 v8::internal::Builtin_HandleApiCall(int, v8::internal::Object*, v8::internal::Isolate) [node] 16: 0x2dcb684dbe1d Aborted

Last-Order commented 5 years ago

不要在dvr直播用--live，不过这个问题我会继续解决

下为发行说明

支持 YouTube HLS 支持范围仅为：直播[1]、开启DVR（实时回放）的直播[2]、直播后约一小时内，尚未转码为DASH格式的视频。 尤其是第三个，IDM不识别，youtube-dl等工具使用ffmpeg单线程下载，为了解决这个痛点，做了这次更新。

[1] 不开启DVR的直播理论上可以--live下载 [2] 开启DVR的直播使用--live下载时可能会有问题，因为每次都要下载一个特别庞大的播放列表

Last-Order commented 5 years ago

应该修正了