search

MycroftAI / mycroft-core

Mycroft Core, the Mycroft Artificial Intelligence platform.
https://mycroft.ai
Apache License 2.0
6.52k stars 1.27k forks source link

Memory leak in MyCroft #1246

Closed roadriverrail closed 6 years ago

roadriverrail commented 6 years ago

I last pulled the dev branch on Monday of this week and I have been experiencing some periodic memory leaks of a very serious nature. The python process running MyCroft has been caught twice consuming over 40 GB of memory on my workstation.

Memory leaks are tough to catch, so I'm filing this issue for y'all and also as a reminder to myself. I'll get a memory graph the next time this strikes me.

Worth noting this is on my Linux workstation, and I do actually take it in and out of suspend, so there could be some sort of strange issue related to that. We'll see as I get more info.

roadriverrail commented 6 years ago

I saw this come up this morning while taking my computer out of standby. A python process was consuminb 42 GB of memory. While I attempted to perform a heap analysis, it crashed. Looking through my currently running python processes, it's clear that the offending process was skills.py.

forslund commented 6 years ago

Thanks for the update. Do you think it has anything to do with hibernation/standby?

Also, can you provide a list of skills you have running? I've been experimenting with the fallback-aiml skill (inactivated by default), It consumes a lot of memory but I've never seen it spin out of control.

roadriverrail commented 6 years ago

I think hibernation/standby might play a role. There are some weird things that go on post-hibernation. For example, I have to unplug my USB microphone and put it back in before any part of my system will register audio input.

Is there an easy way to list my skills?

Best, Rhett

On Wed, Dec 6, 2017 at 3:21 PM Åke notifications@github.com wrote:

Thanks for the update. Do you think it has anything to do with hibernation/standby?

Also, can you provide a list of skills you have running? I've been experimenting with the fallback-aiml skill (inactivated by default), It consumes a lot of memory but I've never seen it spin out of control.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/MycroftAI/mycroft-core/issues/1246#issuecomment-349762715, or mute the thread https://github.com/notifications/unsubscribe-auth/AAlhYHM_X4aKJ14tMikGu0VLVJ4-jkKTks5s9vdfgaJpZM4Qnruh .

roadriverrail commented 6 years ago

Spotted this again after taking my workstation out of suspend. The skills process was consuming 13.5 GB. I attached gdb and generated a core file, but it seems like gdb-heap can't do much with it. I think it might be expecting the core file to come from python2.7-dbg so that there are more internal debugging symbols. I'd share the core file, but it's quite massive.

I tried to alter the mycroft-start.sh to use python2.7-dbg instead of python, but then I get import errors. It makes me think that I probably need to do some kind of pip install, but I don't know what the magic commands would be to have all these modules in the python2.7-dbg environment. Anyone else know?

BTW, here's the skills I have on hand; I presume all of them get loaded:

fallback-aiml              skill-alarm              skill-hello-world  skill-naptime           skill-singing   skill-weather
fallback-duckduckgo        skill-audio-record       skill-installer    skill-npr-news          skill-speak     skill-wiki
fallback-wolfram-alpha     skill-configuration      skill-ip           skill-pairing           skill-spelling  systemSkill-Mycroft-core-Skill-
GoogleCalendarSkill        skill-date-time          skill-joke         skill-personal          skill-stock
mycroft-homeassistant      skill-desktop-launcher   skill-mark1-demo   skill-playback-control  skill-stop
mycroft-skill-diagnostics  skill-google-play-music  skill-media        skill-reminder          skill-volume
MatthewScholefield commented 6 years ago

Try this to run Mycroft using the debugger (adding the correct MYCROFT_FOLDER value):

MYCROFT_FOLDER='/home/.../.../mycroft-core'
cd ~/.virtualenvs/mycroft
. bin/activate
mkdir bin/.debug
ln -s /usr/lib/debug/usr/bin/python2.7-gdb.py bin/.debug/python-gdb.py
ln -s /usr/lib/debug/usr/bin/python2.7 bin/.debug/

run_process() {
    gdb --args bin/python2.7 "$MYCROFT_FOLDER/$2" &> scripts/logs/$1.log &
}

run_process mycroft-bus mycroft/messagebus/service/main.py
run_process mycroft-skills mycroft/skills/main.py
run_process mycroft-audio mycroft/audio/main.py
run_process mycroft-voice mycroft/mycroft/client/speech/main.py
run_process mycroft-voice mycroft/mycroft/client/text/main.py
fg %5

I adapted this from here.

roadriverrail commented 6 years ago

Much obliged. I'll add this to my Mycroft autostart and hopefully we'll get a picture of what's going on.

On Thu, Dec 14, 2017 at 12:03 PM Matthew D. Scholefield < notifications@github.com> wrote:

Try this to run Mycroft using the debugger (adding the correct MYCROFT_FOLDER value):

MYCROFT_FOLDER='/home/.../.../mycroft-core'cd ~/.virtualenvs/mycroft. bin/activate mkdir bin/.debug ln -s /usr/lib/debug/usr/bin/python2.7-gdb.py bin/.debug/python-gdb.py ln -s /usr/lib/debug/usr/bin/python2.7 bin/.debug/ run_process() { gdb --args bin/python2.7 "$MYCROFT_FOLDER/$2" &> scripts/logs/$1.log & }

run_process mycroft-bus mycroft/messagebus/service/main.py run_process mycroft-skills mycroft/skills/main.py run_process mycroft-audio mycroft/audio/main.py run_process mycroft-voice mycroft/mycroft/client/speech/main.py run_process mycroft-voice mycroft/mycroft/client/text/main.pyfg %5

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/MycroftAI/mycroft-core/issues/1246#issuecomment-351773536, or mute the thread https://github.com/notifications/unsubscribe-auth/AAlhYEjeB5DvgbIvmbaohyXMyIDPyD7pks5tAVTTgaJpZM4Qnruh .

roadriverrail commented 6 years ago

As an update, I haven't seen this issue crop up again in a while. It might have been accidentally taken care of. If you'd like to keep the issue open in the event I do see it again, that's fine, but if you'd rather close it until I have more info, that's fine as well.

On Thu, Dec 14, 2017 at 12:44 PM Rhett Aultman roadriverrail@gmail.com wrote:

Much obliged. I'll add this to my Mycroft autostart and hopefully we'll get a picture of what's going on.

On Thu, Dec 14, 2017 at 12:03 PM Matthew D. Scholefield < notifications@github.com> wrote:

Try this to run Mycroft using the debugger (adding the correct MYCROFT_FOLDER value):

MYCROFT_FOLDER='/home/.../.../mycroft-core'cd ~/.virtualenvs/mycroft. bin/activate mkdir bin/.debug ln -s /usr/lib/debug/usr/bin/python2.7-gdb.py bin/.debug/python-gdb.py ln -s /usr/lib/debug/usr/bin/python2.7 bin/.debug/ run_process() { gdb --args bin/python2.7 "$MYCROFT_FOLDER/$2" &> scripts/logs/$1.log & }

run_process mycroft-bus mycroft/messagebus/service/main.py run_process mycroft-skills mycroft/skills/main.py run_process mycroft-audio mycroft/audio/main.py run_process mycroft-voice mycroft/mycroft/client/speech/main.py run_process mycroft-voice mycroft/mycroft/client/text/main.pyfg %5

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/MycroftAI/mycroft-core/issues/1246#issuecomment-351773536, or mute the thread https://github.com/notifications/unsubscribe-auth/AAlhYEjeB5DvgbIvmbaohyXMyIDPyD7pks5tAVTTgaJpZM4Qnruh .

penrods commented 6 years ago

I'll go ahead and close this. Thanks for the follow up!