search

RetroPie / RetroPie-Setup

Shell script to set up a Raspberry Pi/Odroid/PC with RetroArch emulator and various cores
Other
10.07k stars 1.39k forks source link

Update From Source Fails when Running Detailed Themes - Low Memory #1975

Closed TMNTturtleguy closed 7 years ago

TMNTturtleguy commented 7 years ago

Running RPI 3B with 2.5power supply with updated retropie 4.2 updated to most recent update on 5/6/2017.

When running a theme, like Comic Book Theme that uses a large amount of memory, updating emulationstation from source fails. I have run tests 6 times and I fail each time at:

81% building CXX objetct es-app/CMakeFiles/emulationstation.dir/src/gamelist.cpp.o

With this error code showing up sometimes, other times it stalls for several hours with no error, just frozen:

[1094.865873] Write-error on swap-device (179:0 :3925616)

Testing: To test this I changed my theme to Carbon. Ran an update from source on Emulationstation and it updated properly. I then changed back to comic book theme, updating failed at 81%. I then tried updating an es update script from @pjft to his ES build. This updated was from the experimental packages and update from source. This also failed at 81%. Changed back to carbon, updated @pjft build, updated properly. Again using carbon, i updated to the main branch es from source, updated properly.

I now switched to comic book theme, i updated form binary. SUCCESS.

Now I switched back to the comic book theme and updated from source. Failed at 81%. This time I exited ES to command line and entered the setup script manually, updated from source, this updated successfully.

Next step, I opened /home/pi/RetroPie-Setup/scriptmodules/supplementary/emulationstation.sh and I modified line 145 to rpSwap on to 750. Again tried to update the main branch of ES from source. Again it failed.

Last step, I modified rpSwap on to 1024 Again I tried to update the main branch of ES from source. This time it successfully updated. I attempted to update the script from @pjft and this was also successful.

joolswills commented 7 years ago

I dont want to increase the swapfile for es. The error doesn't look like an oom issue but possibly sdcard as oom should cause GCC to be killed. If you have too much in ram it will fail. Can't really help that. Even if the swap is increased.

Also you posted this to the forum. No need to duplicate it here - please read the info when opening a ticket.

pjft commented 7 years ago

Apologies for the duplication - I was the one suggesting that. I imagined that others with more elaborate themes would also be running into this.

I thought of SD card at the time, though there fact that it also stalled on my end recently in the same conditions made me question that. Also was intrigued by the fact that increasing the swap helped, so thought it'd be the right approach here.

Just for my own understanding, what is the main downside of increasing the swap file?

Thanks, have a great Sunday.

joolswills commented 7 years ago

Because it won't guarantee anything since more memory could be in use and in most cases would mean a much larger swap is created than is needed. If building from source you should exit es. The memory settings are based on not having too much stuff running. The swap error shouldn't happen though and that sounds like a hardware issue. GCC should be killed if the system runs out of memory.

TMNTturtleguy commented 7 years ago

Thank you, sorry, i just got this e-mail thread and once again made a mistake as I posted to the forum asking you a similar question, unintentionally duplicating the e-mail from @pjft. I can delete that post if you would like. Appreciate the help and the knowledge that we should be exiting ES when updating from source.

On Sat, May 6, 2017 at 10:41 PM, Jools Wills notifications@github.com wrote:

Because it won't guarantee anything since more memory could be in use and in most cases would mean a much larger swap is created than is needed. If building from source you should exit es. The memory settings are based on not having too much stuff running. The swap error shouldn't happen though and that sounds like a hardware issue. GCC should be killed if the system runs out of memory.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/RetroPie/RetroPie-Setup/issues/1975#issuecomment-299680363, or mute the thread https://github.com/notifications/unsubscribe-auth/AYzz46Bhh-KLfszG08pLaLOHUI_pd15Fks5r3T15gaJpZM4NS_Rv .

joolswills commented 7 years ago

If you have a lot of ram in use at least. Otherwise it will be slower anyway due to swapping.

pjft commented 7 years ago

Thanks for the details - didn't know that, it's good to have the context!

pjft commented 7 years ago

Hi.

Apologies for resurrecting this old thread, but since the last update I've now seen lots of reports in the forums on not being able to update from source, even after exiting ES.

I am, unfortunately, able to replicate it on my end: it gets stuck on 100%, memory stats for the compiler are pretty much maxed out for a pi 3 on the recommended 256 memory split:

PID Name VIRT RES 17095 root 20 0 416540 384216 0 D 3.2 51.0 0:40.32 cc1plus
17078 root 20 0 398060 300056 0 D 3.1 39.9 0:42.27 cc1plus

Zigurana was able to replicate, and by changing the memory split he was able to compile successfully.

In my case, I increased the swap to 950 (I tested small increases from 750) and it finally worked.

I know you're not keen on increasing the swap, so would appreciate some guidance.

Also, out of curiosity, why it's it that getting the code and compiling outside of RetroPie Setup works (I.e. traditional pull and make) but here doesn't? Could any of our compiler or swap options also be acting as a cap in case there are more available resources?

Let me know how I can help.

joolswills commented 7 years ago

Probably because it builds in parallel.

I will increase swap slightly - but on my rpi3 with memory split 256 I didn't have any issue when I last checked.

pjft commented 7 years ago

Thanks.

Actually, you are correct - I was just checking the processes and indeed when I make it on my own, it will only have a process whereby via RetroPie-Setup it will have two parallel processes.

It is up to you - once again, keen on having more people test and see if they run into issues - but if effectively we'd not be keen on increasing swap (which I understand), would making the ES compilation sequential a better option? Do we get a significant decrease in compilation time?

Best.

On Sat, Jul 8, 2017 at 7:47 PM, Jools Wills notifications@github.com wrote:

Probably because it builds in parallel.

I will increase swap slightly - but on my rpi3 with memory split 256 I didn't have any issue when I last checked.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/RetroPie/RetroPie-Setup/issues/1975#issuecomment-313874204, or mute the thread https://github.com/notifications/unsubscribe-auth/AVAV7b7uicTtBNNV8zEvatfeMzDPm46bks5sL87FgaJpZM4NS_Rv .

joolswills commented 7 years ago

I would think the compilation time would be worse yes. I've not timed it though.

joolswills commented 7 years ago

I may make some changes to the memory calculations to take into account currently used memory which it doesn't at the moment which may help also.

(Just to note - ES binaries are up to date - so on the RPI there is no need to build from source. I rebuild the bins every time I bump the version #).

pjft commented 7 years ago

Thanks.

I can time it and report back as well, both for the single process as well as for the RetroPie-Setup parallel version (in this case, though, with the bigger swap otherwise it'll start stalling).

For the record, my stats before running RetroPie-Setup via SSH are:

             total       used       free     shared    buffers     cached
Mem:        752852     123948     628904        932       2420      72556
-/+ buffers/cache:      48972     703880
Swap:       230392      20820     209572

EDIT:

The problem seems to take place when the two processes reach a high memory utilization each:

PID Name       VIRT   RES
17095 root 20 0 416540 384216 0 D 3.2 51.0 0:40.32 cc1plus
17078 root 20 0 398060 300056 0 D 3.1 39.9 0:42.27 cc1plus

EDIT 2:

I see. Thanks for the heads up. I always looked at the "Releases" section on GitHub and since I saw no binary there, I wasn't sure when they'd be generated and where they'd be stored to see what the latest version would be.

pjft commented 7 years ago

A non-scientific report on timing and tests:

Could we run the two stages (i.e. es-core and es-app) with different settings, meaning running es-core compilation with parallel processes, and es-app with a single one, as that's the one that seems to pretty much freeze when it gets to ViewController (stalling at 100% for over 1h, ended up killing it via Ctrl-C as everything else was unresponsive)?

Just a few thoughts.

Do let me know if/how I can help, namely with testing.

Not urgent in any way.

Thanks.

On Sat, Jul 8, 2017 at 7:58 PM, Jools Wills notifications@github.com wrote:

(Just to note - ES binaries are up to date - so on the RPI there is no need to build from source. I rebuild the bins every time I bump the version

).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/RetroPie/RetroPie-Setup/issues/1975#issuecomment-313874757, or mute the thread https://github.com/notifications/unsubscribe-auth/AVAV7fQks0mC_T57gqkIBYQ2gTryUlWKks5sL9FjgaJpZM4NS_Rv .

joolswills commented 7 years ago

Not really - but I have already increased swap - it should be sufficient now. BTW I never saw the freezes you did - perhaps it's also related to sdcard performance.

pjft commented 7 years ago

Might certainly be something like that, and I'm also not on the latest firmware and such (don't touch what's not broken) though I wouldn't fully rule out memory.

I believe this happens on both my development and "play" Pis, one with a 32gb Samsung EVO SD card, class 10, as well as on the 16gb class 10 one that came with the Pi, make unknown.

What stats does "free" return for you prior to starting compilation, if you have a chance, and do they differ from mine significantly?

I'll test the updated script version tomorrow - thanks for looking into this!

On Mon, 10 Jul 2017 at 18:23, Jools Wills notifications@github.com wrote:

Not really - but I have already increased swap - it should be sufficient now. BTW I never saw the freezes you did - perhaps it's also related to sdcard performance.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/RetroPie/RetroPie-Setup/issues/1975#issuecomment-314175486, or mute the thread https://github.com/notifications/unsubscribe-auth/AVAV7YWJQtFba2P40Tmh8voyU-LXyfOeks5sMl4RgaJpZM4NS_Rv .

pjft commented 7 years ago

Thanks - I just tested it and it ran as smooth as butter. Sorry for the trouble, and thanks for the quick fix.

On Mon, Jul 10, 2017 at 6:30 PM, Paulo Tavares paulo.j.tavares@gmail.com wrote:

Might certainly be something like that, and I'm also not on the latest firmware and such (don't touch what's not broken) though I wouldn't fully rule out memory.

I believe this happens on both my development and "play" Pis, one with a 32gb Samsung EVO SD card, class 10, as well as on the 16gb class 10 one that came with the Pi, make unknown.

What stats does "free" return for you prior to starting compilation, if you have a chance, and do they differ from mine significantly?

I'll test the updated script version tomorrow - thanks for looking into this!

On Mon, 10 Jul 2017 at 18:23, Jools Wills notifications@github.com wrote:

Not really - but I have already increased swap - it should be sufficient now. BTW I never saw the freezes you did - perhaps it's also related to sdcard performance.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/RetroPie/RetroPie-Setup/issues/1975#issuecomment-314175486, or mute the thread https://github.com/notifications/unsubscribe-auth/AVAV7YWJQtFba2P40Tmh8voyU-LXyfOeks5sMl4RgaJpZM4NS_Rv .