search

volumio / Volumio2

Volumio 2 - Audiophile Music Player
http://volumio.org
Other
1.37k stars 315 forks source link

Plugin manager: unziping during install over-pressures memory to death #1220

Closed macmpi closed 7 years ago

macmpi commented 7 years ago

It very much seems unzipping plugin archive with decompress-zip can easily generate critical low-memory conditions and also gets node process to take nearly all CPU. This can be seen on lower specked Pis (PiZero for instance) and big plugins like Youtube. Sometimes install will just crash after being blocked for a while right after Creating folder on diskmessage.

Maybe some other zip deflating node package may have better memory & CPU management (or deflating could be done natively)? BTW deflating directly into /data/plugins/... would also save RAM in /tmp, and more aggressive intermediate files cleanup strategies in /tmp might also help... (as pointed here)

sample log below:

Jun 11 13:11:14 volumio volumio[910]: info: Downloading plugin at http://volumio.github.io/volumio-plugins/plugins/volumio/armhf/music_service/youtube/youtube.zip
Jun 11 13:11:19 volumio volumio[910]: info: END DOWNLOAD: http://volumio.github.io/volumio-plugins/plugins/volumio/armhf/music_service/youtube/youtube.zip
Jun 11 13:12:38 volumio volumio[910]: info: [1497186758531] ControllerMpd::pushError
Jun 11 13:12:38 volumio volumio[910]: info:  Error: spawn ENOMEM
Jun 11 13:12:38 volumio volumio[910]: at exports._errnoException (util.js:1018:11)
Jun 11 13:12:38 volumio volumio[910]: at ChildProcess.spawn (internal/child_process.js:319:11)
Jun 11 13:12:38 volumio volumio[910]: at exports.spawn (child_process.js:378:9)
Jun 11 13:12:38 volumio volumio[910]: at Object.exports.execFile (child_process.js:143:15)
Jun 11 13:12:38 volumio volumio[910]: at Object.exports.exec (child_process.js:103:18)
Jun 11 13:12:38 volumio volumio[910]: at Object.status (/volumio/node_modules/wireless-tools/ifconfig.js:192:10)
Jun 11 13:12:38 volumio volumio[910]: at ControllerVolumioDiscovery.getDevices (/volumio/app/plugins/system_controller/volumiodiscovery/index.js:378:14)
Jun 11 13:12:38 volumio volumio[910]: at InterfaceWebUI.pushMultiroom (/volumio/app/plugins/user_interface/websocket/index.js:1829:34)
Jun 11 13:12:38 volumio volumio[910]: at InterfaceWebUI.pushState (/volumio/app/plugins/user_interface/websocket/index.js:1847:8)
Jun 11 13:12:38 volumio volumio[910]: at /volumio/app/index.js:293:26
Jun 11 13:12:38 volumio volumio[910]: info: [1497186758595] ------------------------------ 10101ms
Jun 11 13:12:43 volumio kernel: lowmemorykiller: Killing 'node' (910), adj 0,
                                   to free 171252kB on behalf of 'kswapd0' (23) because
                                   cache 6044kB is below limit 6144kB for oom_score_adj 0
                                   Free memory is 3108kB above reserved
volumio commented 7 years ago

We can use a simple native call to unzip.. We need to add it into recipes and then call it with execsync

macmpi commented 7 years ago

minizip (miniunzip) is usually quite efficient tiny package. Quite some changes to do in pluginmanager still, to wrap-it nicely, and optimize /tmp intermediate file cleanup...

volumio commented 7 years ago

Ok!

macmpi commented 7 years ago

@volumio While at it, I realize fs.move ( in renameFolder and moveToCategory for instance) duplicates files instead of doing filesystem mv. For big plugin folders, that makes a significant difference in terms of memory footprint & flash usage. Since portability is secondary (all linux) and we do only move data within same filesystem for plugin install, would it be acceptable to use more optimal mv command here which does not actually rewrite/duplicate data?

Thanks for feedback.