OutOfMemoryError: GC overhead limit exceeded after 30 min

[jorisbontje]

with bb657350a71dcb6b23f93de176fb4c8817ec89e1 on a 2GB DigitalOcean droplet

$ lein run -m examples.ircbot
05:31:58.323 [main] INFO  c.p.actors.ActorRegistry - Actor registry is co.paralleluniverse.actors.LocalActorRegistry@61375dff
Beliefs rules: 3151
Questions rules: 474
Goal rules: 3161
Quests rules: 474
"Elapsed time: 38474.858077 msecs"
16-07-11 09:32:53 ubuntu-2gb-ams2-01 INFO [narjure.core:105] - reset question filter
16-07-11 09:32:53 ubuntu-2gb-ams2-01 INFO [narjure.core:107] - Resetting concepts bagss:
16-07-11 09:32:53 ubuntu-2gb-ams2-01 INFO [narjure.core:109] - c-bag count:  0
16-07-11 09:32:53 ubuntu-2gb-ams2-01 INFO [narjure.core:113] - Reset system Parameters:
16-07-11 09:32:53 ubuntu-2gb-ams2-01 INFO [narjure.core:115] - nars-id:  -1
16-07-11 09:32:53 ubuntu-2gb-ams2-01 INFO [narjure.core:117] - nars-time:  0
16-07-11 09:32:53 ubuntu-2gb-ams2-01 INFO [narjure.core:120] - NARS initialising...
16-07-11 09:32:53 ubuntu-2gb-ams2-01 INFO [narjure.core:56] - Initialising system timers...
16-07-11 09:32:53 ubuntu-2gb-ams2-01 INFO [narjure.core:51] -   [OK] :inference-timer (110 ms)
16-07-11 09:32:53 ubuntu-2gb-ams2-01 INFO [narjure.core:51] -   [OK] :system-timer (150 ms)
16-07-11 09:32:53 ubuntu-2gb-ams2-01 INFO [narjure.core:68] - System timer initialisation complete.
16-07-11 09:32:54 ubuntu-2gb-ams2-01 INFO [narjure.core:128] - NARS initialised.
16-07-11 09:33:04 ubuntu-2gb-ams2-01 INFO [narjure.core:56] - Initialising system timers...
16-07-11 09:33:04 ubuntu-2gb-ams2-01 INFO [narjure.core:51] -   [OK] :inference-timer (25 ms)
16-07-11 09:33:04 ubuntu-2gb-ams2-01 INFO [narjure.core:51] -   [OK] :system-timer (50 ms)
16-07-11 09:33:04 ubuntu-2gb-ams2-01 INFO [narjure.core:68] - System timer initialisation complete.
Connecting... irc.freenode.org
Joining #narstest
NARS hears <{Tweety} --> [yellow]>.
NARS hears <canary --> [yellow]>.
NARS hears <canary --> bird>.
NARS hears <{Tweety} --> bird>?
NARS answer on <{Tweety} --> bird>? is <{Tweety} --> bird>.  %1.0;0.40276243093922653%
Exception in thread "SimpleQuartzScheduler:-1011881444_QuartzSchedulerThread" Exception in Fiber "fiber-10000002" Exception in thread "Thread-2" Exception in Fiber "fiber-10000005" java.lang.OutOfMemoryError: GC overhead limit exceeded
        at org.quartz.impl.JobExecutionContextImpl.<init>(JobExecutionContextImpl.java:76)
        at org.quartz.core.JobRunShell.initialize(JobRunShell.java:143)
        at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:375)

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-2"
Exception in thread "Timer-0" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "FiberTimedScheduler-default-fiber-pool" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "scheduling-worker-4" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in Fiber "fiber-10000171" Exception in Fiber "fiber-10000006" Exception in Fiber "fiber-10000064" Exception in Fiber "fiber-10000007" Exception in Fiber "fiber-10000024" Exception in Fiber "fiber-10000169" Exception in Fiber "fiber-10000067" Exception in Fiber "fiber-10000159" Exception in Fiber "fiber-10000013" Exception in Fiber "fiber-10000057" Exception in Fiber "fiber-10000079"

[TonyLo1]

Ok, I will disable memorization in the deriver completely to see if this resolves the issue.

From: Joris Bontje [mailto:notifications@github.com] Sent: 11 July 2016 11:15 To: opennars/opennars2 opennars2@noreply.github.com Subject: [opennars/opennars2] OutOfMemoryError: GC overhead limit exceeded after 30 min (#59)

with https://github.com/opennars/opennars2/commit/bb657350a71dcb6b23f93de176fb4c8817ec89e1 bb65735 on a 2GB DigitalOcean droplet

$ lein run -m examples.ircbot 05:31:58.323 [main] INFO c.p.actors.ActorRegistry - Actor registry is co.paralleluniverse.actors.LocalActorRegistry@61375dff Beliefs rules: 3151 Questions rules: 474 Goal rules: 3161 Quests rules: 474 "Elapsed time: 38474.858077 msecs" 16-07-11 09:32:53 ubuntu-2gb-ams2-01 INFO [narjure.core:105] - reset question filter 16-07-11 09:32:53 ubuntu-2gb-ams2-01 INFO [narjure.core:107] - Resetting concepts bagss: 16-07-11 09:32:53 ubuntu-2gb-ams2-01 INFO [narjure.core:109] - c-bag count: 0 16-07-11 09:32:53 ubuntu-2gb-ams2-01 INFO [narjure.core:113] - Reset system Parameters: 16-07-11 09:32:53 ubuntu-2gb-ams2-01 INFO [narjure.core:115] - nars-id: -1 16-07-11 09:32:53 ubuntu-2gb-ams2-01 INFO [narjure.core:117] - nars-time: 0 16-07-11 09:32:53 ubuntu-2gb-ams2-01 INFO [narjure.core:120] - NARS initialising... 16-07-11 09:32:53 ubuntu-2gb-ams2-01 INFO [narjure.core:56] - Initialising system timers... 16-07-11 09:32:53 ubuntu-2gb-ams2-01 INFO [narjure.core:51] - [OK] :inference-timer (110 ms) 16-07-11 09:32:53 ubuntu-2gb-ams2-01 INFO [narjure.core:51] - [OK] :system-timer (150 ms) 16-07-11 09:32:53 ubuntu-2gb-ams2-01 INFO [narjure.core:68] - System timer initialisation complete. 16-07-11 09:32:54 ubuntu-2gb-ams2-01 INFO [narjure.core:128] - NARS initialised. 16-07-11 09:33:04 ubuntu-2gb-ams2-01 INFO [narjure.core:56] - Initialising system timers... 16-07-11 09:33:04 ubuntu-2gb-ams2-01 INFO [narjure.core:51] - [OK] :inference-timer (25 ms) 16-07-11 09:33:04 ubuntu-2gb-ams2-01 INFO [narjure.core:51] - [OK] :system-timer (50 ms) 16-07-11 09:33:04 ubuntu-2gb-ams2-01 INFO [narjure.core:68] - System timer initialisation complete. Connecting... irc.freenode.org Joining #narstest NARS hears <{Tweety} --> [yellow]>. NARS hears <canary --> [yellow]>. NARS hears <canary --> bird>. NARS hears <{Tweety} --> bird>? NARS answer on <{Tweety} --> bird>? is <{Tweety} --> bird>. %1.0;0.40276243093922653% Exception in thread "SimpleQuartzScheduler:-1011881444_QuartzSchedulerThread" Exception in Fiber "fiber-10000002" Exception in thread "Thread-2" Exception in Fiber "fiber-10000005" java.lang.OutOfMemoryError: GC overhead limit exceeded at org.quartz.impl.JobExecutionContextImpl.(JobExecutionContextImpl.java:76) at org.quartz.core.JobRunShell.initialize(JobRunShell.java:143) at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:375)

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-2" Exception in thread "Timer-0" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "FiberTimedScheduler-default-fiber-pool" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "scheduling-worker-4" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in Fiber "fiber-10000171" Exception in Fiber "fiber-10000006" Exception in Fiber "fiber-10000064" Exception in Fiber "fiber-10000007" Exception in Fiber "fiber-10000024" Exception in Fiber "fiber-10000169" Exception in Fiber "fiber-10000067" Exception in Fiber "fiber-10000159" Exception in Fiber "fiber-10000013" Exception in Fiber "fiber-10000057" Exception in Fiber "fiber-10000079"

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/opennars/opennars2/issues/59 , or mute the thread https://github.com/notifications/unsubscribe/AKD1x0afD_-qhok6CNscJsu-4yB0FJEXks5qUhfAgaJpZM4JJOGx .

[TonyLo1]

it's already disabled. So cancel that

[TonyLo1]

I've been running a test for over 30 minutes now and the memory allocation is stable, actually reducing from its peak. Currently unable to reproduce this error. Will continue to investigate.

[patham9]

@jorisbontje Did you run with fast speed? If your machine is too slow for this mode, actor messages will begin to add up and memory will get full. I guess this is the reason. On my machine both modes seem to work fine.

[jorisbontje]

I'll rerun it in slow mode and report back how that goes.

[patham9]

closed since this one only occurrs if the system params are set too fast for the used hardware

[jorisbontje]

In "slow" mode it dies after ~28h:

Exception in thread "SimpleQuartzScheduler:-1011881444_QuartzSchedulerThread" *** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message can't create byte arrau at JPLISAgent.c line: 813
*** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message can't create byte arrau at JPLISAgent.c line: 813
java.lang.OutOfMemoryError: Java heap space

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "SimpleQuartzScheduler:-1011881444_QuartzSchedulerThread"
Exception in Fiber "fiber-10000002" Exception in Fiber "fiber-10000016" *** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message can't create byte arrau at JPLISAgent.c line: 813
*** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message can't create byte arrau at JPLISAgent.c line: 813

[patham9]

What are your system resources? (RAM, CPU and its frequency) My first guess is that slow mode is still slightly too fast for your system, causing actor messages to stack up. But 28h is acceptable, as the entire memory state can be saved with the save button, and a fresh start can continue based on this memory.

Best regards, Patrick

2016-07-13 22:46 GMT+02:00 Joris Bontje notifications@github.com:

In "slow" mode it dies after ~28h:

Exception in thread "SimpleQuartzScheduler:-1011881444QuartzSchedulerThread" *\ java.lang.instrument ASSERTION FAILED *: "!errorOutstanding" with message can't create byte arrau at JPLISAgent.c line: 813 _* java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message can't create byte arrau at JPLISAgent.c line: 813 java.lang.OutOfMemoryError: Java heap space

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "SimpleQuartzScheduler:-1011881444QuartzSchedulerThread" Exception in Fiber "fiber-10000002" Exception in Fiber "fiber-10000016" *\ java.lang.instrument ASSERTION FAILED *: "!errorOutstanding" with message can't create byte arrau at JPLISAgent.c line: 813 _* java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message can't create byte arrau at JPLISAgent.c line: 813

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/opennars/opennars2/issues/59#issuecomment-232481352, or mute the thread https://github.com/notifications/unsubscribe/AH5qBWDthBrs4uRqA5SbwobK9Yl8wQIPks5qVU6agaJpZM4JJOGx .

[jorisbontje]

It is a 2GB digital ocean droplet; https://www.digitalocean.com/pricing/ /proc/cpuinfo reports 2x Intel(R) Xeon(R) CPU E5-2630L 0 @ 2.00GHz

Under AIKR, shouldn't NARS reduce its resources rather than go belly up?

[patham9]

I agree in principle. However it seems to be a actor model thingy. If the system parameters are set slow enough for the machine it will run on, the issue doesn't seem to appear.

Best regards, Patrick

2016-07-13 22:59 GMT+02:00 Joris Bontje notifications@github.com:

It is a 2GB digital ocean droplet; https://www.digitalocean.com/pricing/ /proc/cpuinfo reports 2x Intel(R) Xeon(R) CPU E5-2630L 0 @ 2.00GHz

Under AIKR, shouldn't NARS reduce its resources rather than go belly up?

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/opennars/opennars2/issues/59#issuecomment-232484914, or mute the thread https://github.com/notifications/unsubscribe/AH5qBWbxdgY9Huw_BlzfOsFZsoaNP2fEks5qVVG7gaJpZM4JJOGx .

[patham9]

For me it never went above 2GB RAM, it started with 1.4GB and with full memory it was stable below 2GB over hours. I just saw "2GB digital ocean droplet;" I guess for this the preconfigured max. memory size is too big, in this case its not the system timers speed thats the issue (processing speed), but the max. memory size that is allowed (storage size).

Best regards, Patrick

2016-07-13 23:04 GMT+02:00 Patrick Hammer patham9@gmail.com:

I agree in principle. However it seems to be a actor model thingy. If the system parameters are set slow enough for the machine it will run on, the issue doesn't seem to appear.

Best regards, Patrick

2016-07-13 22:59 GMT+02:00 Joris Bontje notifications@github.com:

It is a 2GB digital ocean droplet; https://www.digitalocean.com/pricing/ /proc/cpuinfo reports 2x Intel(R) Xeon(R) CPU E5-2630L 0 @ 2.00GHz

Under AIKR, shouldn't NARS reduce its resources rather than go belly up?

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/opennars/opennars2/issues/59#issuecomment-232484914, or mute the thread https://github.com/notifications/unsubscribe/AH5qBWbxdgY9Huw_BlzfOsFZsoaNP2fEks5qVVG7gaJpZM4JJOGx .

[PtrMan]

In all respect but,

But 28h is acceptable, as the entire memory state can be saved with the save button, and a fresh start can continue based on this memory.

What de fuck dude, thats not acceptable...

"Oh look, just restart the software ever 5 minutes, should be fine"... not ... acceptable

2016-07-13 23:07 GMT+02:00 patham9 notifications@github.com:

For me it never went above 2GB RAM, it started with 1.4GB and with full memory it was stable below 2GB over hours. I just saw "2GB digital ocean droplet;" I guess for this the preconfigured max. memory size is too big, in this case its not the system timers speed thats the issue (processing speed), but the max. memory size that is allowed (storage size).

Best regards, Patrick

2016-07-13 23:04 GMT+02:00 Patrick Hammer patham9@gmail.com:

I agree in principle. However it seems to be a actor model thingy. If the system parameters are set slow enough for the machine it will run on, the issue doesn't seem to appear.

Best regards, Patrick

2016-07-13 22:59 GMT+02:00 Joris Bontje notifications@github.com:

It is a 2GB digital ocean droplet; https://www.digitalocean.com/pricing/ /proc/cpuinfo reports 2x Intel(R) Xeon(R) CPU E5-2630L 0 @ 2.00GHz

Under AIKR, shouldn't NARS reduce its resources rather than go belly up?

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <https://github.com/opennars/opennars2/issues/59#issuecomment-232484914 , or mute the thread < https://github.com/notifications/unsubscribe/AH5qBWbxdgY9Huw_BlzfOsFZsoaNP2fEks5qVVG7gaJpZM4JJOGx

.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/opennars/opennars2/issues/59#issuecomment-232487046, or mute the thread https://github.com/notifications/unsubscribe/ABBLkJ_oiaggjbvQgWnX1Gyftl-Drgfoks5qVVOSgaJpZM4JJOGx .

[patham9]

As I said its due to a wrong configuration which is not set in respect to the resources of the machine. We could find a way to open an error message if the expected max. machine resources the system uses in this configuration are bigger than what the host platform can give. The system needs a given set of resources the host machine supports in order to work and will stick with it in its entire lifetime. Simple example but which makes the point clear: For example there can't be more concepts than bytes in your memory, setting that max. amount of concepts will lead to the same issue, the system crashing once the resource promise the host machine gave NARS does fall apart. ^^

[patham9]

If it also happens when resource settings are set low enough for the machine, we have to re-open the issue, but it doesn't seem to be the case for me, also not for you, TonyLo1, right?

[TonyLo1]

A specific response and then some general comments:

This situation is entirely caused by running the system with insufficient resources for the configuration specified. Some specific advice for anyone encountering this issue: firstly, run the system in slow mode, selected in the UI HUD, or alternatively, for the more adventurous, reduce the inference-tick-interval and/or reduce the max-concept-selections in defaults.clj.

In general, the system is CPU constrained and not memory constrained. In theory, RAM capacity is not a limit when you take swapping into account but the swapping overhead will further reduce the CPU bandwidth.

We need to do a better job of setting expectations on the minimum system requirements. Also we probably can allow the system to auto-configure its config based on available memory, CPU speed and number of threads available.

Finally, a point to bear in mind, OpenNARS is an AGI, you will only be able to run simple examples on a laptop :)

[jorisbontje]

After switching from OpenJDK to OracleJDK I am seeing a 8+ days uptime on the same environment.

[TonyLo1]

Jorisbontje, good to hear, we thought may be a contributing factor :)

[patham9]

so it was a memory leak in OpenJDK? Interesting, I didn't expect this, and we might report it to them. I am personally also running OracleJDK.

[jorisbontje]

FYI I am now running the ircbot application with:

lein with-profile production trampoline run -m examples.ircbot

and (set-fast-speed) disabled.

[PtrMan]

Haha, Java crap bugz always hit you hard in your face, patham9...

this fucking software is too complicated, there is no way in hell that this is the right solution

2016-07-22 16:09 GMT+02:00 Joris Bontje notifications@github.com:

FYI I am now running the ircbot application with:

lein with-profile production trampoline run -m examples.ircbot

and (set-fast-speed) disabled.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/opennars/opennars2/issues/59#issuecomment-234553981, or mute the thread https://github.com/notifications/unsubscribe-auth/ABBLkLoz9kt0vVhIk_GlZyHe6Yl0vSW8ks5qYM8xgaJpZM4JJOGx .

[PtrMan]

As I said its due to a wrong configuration which is not set in respect to the resources of the machine. No it wasn't

2016-07-22 22:38 GMT+02:00 robertw89@googlemail.com:

Haha, Java crap bugz always hit you hard in your face, patham9...

this fucking software is too complicated, there is no way in hell that this is the right solution

2016-07-22 16:09 GMT+02:00 Joris Bontje notifications@github.com:

FYI I am now running the ircbot application with:

lein with-profile production trampoline run -m examples.ircbot

and (set-fast-speed) disabled.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/opennars/opennars2/issues/59#issuecomment-234553981, or mute the thread https://github.com/notifications/unsubscribe-auth/ABBLkLoz9kt0vVhIk_GlZyHe6Yl0vSW8ks5qYM8xgaJpZM4JJOGx .

[PtrMan]

This situation is entirely caused by running the system with insufficient resources for the configuration specified. WTF?

2016-07-22 22:38 GMT+02:00 robertw89@googlemail.com:

As I said its due to a wrong configuration which is not set in respect to the resources of the machine. No it wasn't

2016-07-22 22:38 GMT+02:00 robertw89@googlemail.com:

Haha, Java crap bugz always hit you hard in your face, patham9...

this fucking software is too complicated, there is no way in hell that this is the right solution

2016-07-22 16:09 GMT+02:00 Joris Bontje notifications@github.com:

FYI I am now running the ircbot application with:

lein with-profile production trampoline run -m examples.ircbot

and (set-fast-speed) disabled.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/opennars/opennars2/issues/59#issuecomment-234553981, or mute the thread https://github.com/notifications/unsubscribe-auth/ABBLkLoz9kt0vVhIk_GlZyHe6Yl0vSW8ks5qYM8xgaJpZM4JJOGx .

[PtrMan]

Just stop using untested big buggy software you didn't write and can't control and which changes all the time, thank you.

This is not acceptable for an "AGI", absolutly not.

What is AIKR worth if you need infinite memory because of a mem leak in the retarded VM?

2016-07-22 22:39 GMT+02:00 robertw89@googlemail.com:

This situation is entirely caused by running the system with insufficient resources for the configuration specified. WTF?

2016-07-22 22:38 GMT+02:00 robertw89@googlemail.com:

As I said its due to a wrong configuration which is not set in respect to the resources of the machine. No it wasn't

2016-07-22 22:38 GMT+02:00 robertw89@googlemail.com:

Haha, Java crap bugz always hit you hard in your face, patham9...

this fucking software is too complicated, there is no way in hell that this is the right solution

2016-07-22 16:09 GMT+02:00 Joris Bontje notifications@github.com:

FYI I am now running the ircbot application with:

lein with-profile production trampoline run -m examples.ircbot

and (set-fast-speed) disabled.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/opennars/opennars2/issues/59#issuecomment-234553981, or mute the thread https://github.com/notifications/unsubscribe-auth/ABBLkLoz9kt0vVhIk_GlZyHe6Yl0vSW8ks5qYM8xgaJpZM4JJOGx .

[0xc1c4da]

I will again put forth my suggestion to this issue, perhaps we should set an X amount of ram and cpu cores (as CLI params?) to use by the system and let the system adjust configuration to maximize that usage, with a maximum parameters to the amount available to the operating system.

This would eliminate human errors and system failure due to misconfiguration.

As for JVM - OpenNARS has been written on JVM since at least 2008. And since then I haven't seen much traction on the C/C++ version. So lets work with what we have.

[pei2nars]

NARS was in C++ in the period of 1994-96 as my PhD project. Since 1997 the implementation has been in Java (except a version of the rules in Prolog). 2008 is the year when I put the code online. The C++ code hasn't been touched for 20 years.

Regards,

Pei

On Fri, Jul 22, 2016 at 5:32 PM, Jarrad notifications@github.com wrote:

I will again put forth my suggestion to this issue, perhaps we should set an X amount of ram and cpu cores to use by the system and let the system adjust configuration to maximize that usage, with a maximum parameters to the amount available to the operating system.

This would eliminate human errors and system failure due to misconfiguration.

As for JVM - OpenNARS has been written on JVM since at least 2008 https://github.com/opennars/opennars_googlecode/commit/ae115e743161a1858d6e1ccc8ece0a4e2c1f73b6. And since then I haven't seen much traction on the C/C++ version. So lets work with what we have.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/opennars/opennars2/issues/59#issuecomment-234662226, or mute the thread https://github.com/notifications/unsubscribe-auth/AHxfsSHrQaLyumqoHoHkYiVbm0N0j1XEks5qYTb3gaJpZM4JJOGx .

[patham9]

This out of-memory-bug is a non-issue since it only happens on OpenJDK. And even if it would also happen on OracleJDK, which it does not: It is the direction to go and it would surely lie in the interest of Oracle to fix this bug (whether this also holds for the OpenJDK we will see but I also think so). Also there is a general agreement to use a actor-framework due to future project-demands (ability to run on supercomputers, distributed on the network etc.) by these team-members who contributed code to OpenNARS-v2.0.0.

No worry, Patrick