"this" corrupted

[tsieger]

"this" gets corrupted in user's methods overriding Qt widgets methods (like the paintEvent method below) running the 'qt4' branch and Qt4:

qsetClass("Label", Qt$QLabel, function(parent = NULL) {
  super(parent)
})

qsetMethod("paintEvent", Label, function(event) {
  print(this)
  super("paintEvent",event)
},"protected")

window <- Qt$QWidget()
queryLabel1 <- Label("label 1")
queryLabel2 <- Label("label 2")
queryLabel3 <- Label("label 3")

layout <- Qt$QHBoxLayout()
layout$addWidget(queryLabel1)
layout$addWidget(queryLabel2)
layout$addWidget(queryLabel3)

window$setLayout(layout)
window$show()

When resizing the window for a while (few seconds at most), or simply focusing and defocusing the window, R crashes saying:

 *** caught segfault ***
address 0x10, cause 'memory not mapped'

Traceback:
 1: print(this)
 2: (function (event) {    print(this)    super("paintEvent", event)})(<environment>)

The problem is neither related to creating several instances of the same class, nor to QtLabel.

Reproducible on Ubuntu 14.04 on Intel Core 2 Duo, both 64-bit and 32-bit, but also on Mac OS X 10.8.4, Mac Pro Pro 5.1, 6-Core Intel Xeon 2.4GHz, but not on Fedora Core Linux 3.6.11-1.fc17.i686.PAE i686.

sessionInfo:
 R version 3.1.0 (2014-04-10)
Platform: x86_64-pc-linux-gnu (64-bit)

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base  qtbase_1.0.6

Not sure whether the same problem appears using the main qtbase branch and Qt5.2+ (can't build qtbase for some reason - working on it).

Any ideas are welcome.

[lawremi]

Looked at it for a few hours this morning; no solution yet. When R evaluates "this" it allocates an environment that uses the same object table as the environment enclosing the paintEvent handler. My working theory is that if R finalizes "this", somehow the object table also gets finalized, so the enclosing environment of the handler is no longer valid. Not sure if the R implementation expects two environments to share an object table. That would not ordinarily happen, but it should work in theory. We can't just use the same environment, because the parent environment is different.

On Fri, Jun 6, 2014 at 4:45 AM, Tomas Sieger notifications@github.com wrote:

"this" gets corrupted in user's methods overriding Qt widgets methods (like the paintEvent method below) running the 'qt4' branch and Qt4:

qsetClass("Label", Qt$QLabel, function(parent = NULL) { super(parent) })

qsetMethod("paintEvent", Label, function(event) { print(this) super("paintEvent",event) },"protected")

window <- Qt$QWidget() queryLabel1 <- Label("label 1") queryLabel2 <- Label("label 2") queryLabel3 <- Label("label 3")

layout <- Qt$QHBoxLayout() layout$addWidget(queryLabel1) layout$addWidget(queryLabel2) layout$addWidget(queryLabel3)

window$setLayout(layout) window$show()

When resizing the window for a while (few seconds at most), or simply focusing and defocusing the window, R crashes saying:

* caught segfault * address 0x10, cause 'memory not mapped'

Traceback: 1: print(this) 2: (function (event) { print(this) super("paintEvent", event)})()

The problem is neither related to creating several instances of the same class, nor to QtLabel.

Reproducible on Ubuntu 14.04 on Intel Core 2 Duo, both 64-bit and 32-bit, but also on Mac OS X 10.8.4, Mac Pro Pro 5.1, 6-Core Intel Xeon 2.4GHz, but not on Fedora Core Linux 3.6.11-1.fc17.i686.PAE i686.

sessionInfo: R version 3.1.0 (2014-04-10) Platform: x86_64-pc-linux-gnu (64-bit)

attached base packages: [1] stats graphics grDevices utils datasets methods base qtbase_1.0.6

Not sure whether the same problem appears using the main qtbase branch and Qt5.2+ (can't build qtbase for some reason - working on it).

Any ideas are welcome.

— Reply to this email directly or view it on GitHub https://github.com/ggobi/qtbase/issues/21.

[tsieger]

Thanks for sharing your theory. Indeed, the problem is not only with "this", but perhaps with the (parent) environment of the handler - even if I do not touch "this" in the handler, but only print something, after a while the handler crashes, e.g. when flushing stdout (!):

 *** caught segfault ***
address 0x10, cause 'memory not mapped'

Traceback:
 1: flush(stdout())
 2: (function (event) {    print("paintEvent3")    flush(stdout())    print("leaving paintEvent3")    flush(stdout())})(<environment>)

At the same time, the bug seems to be (quite logically) related to global memory conditions (memory allocation, or possibly garbage collection?). When I inspect "this" using .Internal(inspect(this)) in the handler, I can see the address of "this" is different on each call to the handler - it keeps gradually changing from say <0x25cb908> thru <0x2610de0> up to <0x32f6620> and then, once this gets (re)allocated "lower" in the address space, e.g. <0x1c1a520>, R crashes. Interestingly, it crashes not in the handler in which "this" is being inspected, but another one - so the problem is more global. (Perhaps a trivial observation.)

[lawremi]

Thanks for looking into this more deeply. Yes, a new environment gets allocated each time, but it shares the same symbol table as the parent environment of the handler frame. The allocation to a lower address is probably after the GC has run (and invalidated things inappropriately). I don't get why R finalizes the symbol table (an externalptr), despite it being referenced by the handler frame itself. I have noticed that if I invoke gc() in the handler, the crashes go away, and the garbage collector behaves correctly. So it's something about references being left around over multiple invocations of the handler... but it's tough to pin-down. I even tried running gc() randomly in the handler, e.g., ~ every 50 invocations, and everything kept working. I'm guessing that if gc() were invoked much less frequently, R would end up invoking it, perhaps from a different context and maybe that makes a difference? Starting to smell like a bug/limitation in R...

On Wed, Jun 11, 2014 at 12:28 PM, Tomas Sieger notifications@github.com wrote:

Thanks for sharing your theory. Indeed, the problem is not only with "this", but perhaps with the (parent) environment of the handler - even if I do not touch "this" in the handler, but only print something, after a while the handler crashes, e.g. when flushing stdout (!):

* caught segfault * address 0x10, cause 'memory not mapped'

Traceback: 1: flush(stdout()) 2: (function (event) { print("paintEvent3") flush(stdout()) print("leaving paintEvent3") flush(stdout())})()

At the same time, the bug seems to be (quite logically) related to global memory conditions (memory allocation, or possibly garbage collection?). When I inspect "this" using .Internal(inspect(this)) in the handler, I can see the address of "this" is different on each call to the handler - it keeps gradually changing from say <0x25cb908> thru <0x2610de0> up to

<0x32f6620> and then, once this gets (re)allocated "lower" in the address space, e.g. <0x1c1a520>, R crashes. Interestingly, it crashes not in the handler in which "this" is being inspected, but another one - so the problem is more global. (Perhaps a trivial observation.) — Reply to this email directly or view it on GitHub https://github.com/ggobi/qtbase/issues/21#issuecomment-45788120.

[tsieger]

Thanks for the further investigation and explanation. The trick with the explicit calls to GC seems to work, however, I'm experiencing that the GC must be called much more often than on each 50th call to the handler, which makes my app terribly slow :-(. Also, the exact GC calling frequency is probably system-dependent and hard to guess in advance.

In the meantime, I've tried to workaround the problem by bypassing the usage of the "bad" environments in the handler by running the handler in my own environment holding only the "event" parameter (and having the global env as the parent), and replacing "this" with a globally visible singleton (see below), but that still does not work, perhaps because the "bad" environments get still constructed and used during the call to the handler (even though the handler does not touch them).

myenv<-function(nm,val){e<-new.env(globalenv());assign(nm,val,e);e}

f1<-function(event) {
evalq({
  qtbase:::qinvokeSuper(label1,'paintEvent',event)
},myenv('event',event),globalenv())
}
f2<-function(event) {
evalq({
  qtbase:::qinvokeSuper(label2,'paintEvent',event)
},myenv('event',event),globalenv())
}

qsetClass("Label1", Qt$QLabel, function(parent = NULL) {
  super(parent)
})
qsetMethod("paintEvent", Label1, f1,"protected")

qsetClass("Label2", Qt$QLabel, function(parent = NULL) {
  super(parent)
})
qsetMethod("paintEvent", Label2, f2,"protected")

window <- Qt$QWidget()
label1 <- Label1("label 1")
label2 <- Label2("label 2")

layout <- Qt$QHBoxLayout()
layout$addWidget(label1)
layout$addWidget(label2)

window$setLayout(layout)
window$show()

BTW provided your theory is true (and pardon my limited knowledge) - would not be it possible to solve the problem by not sharing the object table between "this" and the environment enclosing the paintEvent handler, but copying the table (plus implementing some write-thru mechanism, if necessary)?

[tsieger]

Update: the issue demonstrates only in R3.1.0+ (not in previous R versions, even R 3.0.3). It still appears under patched (3.1.0 Patched (2014-06-12 r65926)) and development (R Under development (unstable) (2014-06-12 r65926)) R versions.

BTW I've noticed that NEWS to R 3.1.0 read: "Also, a number of other changes to reduce copying of objects; all contributed by or based on suggestions by Michael Lawrence." :-)

[lawremi]

This is an interesting finding, thanks. My changes were to the implementation of copy-on-write; pretty much unrelated to the garbage collector. But maybe we should bring this to the attention of Luke?

On Fri, Jun 13, 2014 at 7:46 AM, Tomas Sieger notifications@github.com wrote:

Update: the issue demonstrates only in R3.1.0+ (not in previous R versions, even R 3.0.3). It still appears under patched (3.1.0 Patched (2014-06-12 r65926)) and development (R Under development (unstable) (2014-06-12 r65926)) R versions.

BTW I've noticed that NEWS to R 3.1.0 read: "Also, a number of other changes to reduce copying of objects; all contributed by or based on suggestions by Michael Lawrence." :-)

— Reply to this email directly or view it on GitHub https://github.com/ggobi/qtbase/issues/21#issuecomment-46019820.

[tsieger]

That's a good idea. Could you do it, please?

[lawremi]

Luke says that it is very unlikely to be an issue with the GC. He says that it must just be a slight change in memory allocation behavior. That may be the case; however, I've scoured the code for any missing PROTECT()s and came up empty. I just don't understand how the parent of the current frame of evaluation does not have a protected object table.

[tsieger]

I investigated it more deeply and found out the problem is that the object table gets finalized as the environment it is associated to is considered unreachable in finalization (even though the object table is in use at that time in another environment).

I've injected R and qtbase with some debugs and mention some output here for reference:

key:
a0x45977a8 - R_ObjectTable instance
a0x66765e8 - SEXP R external pointer wrapper for a0x45977a8
a0x5d728b8 - WeakRef for a0x66765e8
a0x66765b0 - SEXP environmnent (perhaps of the "paintEvent" handler)
a0x55de298 - SEXP environmnent (perhaps of another invocation of the "paintEvent" handler)
a0x415e370 - parent environment of the two environments above

graphically:
                          +- a0x66765e8 -+   
a0x66765b0 -(hashtab)-->  |   wrapped:   |  <-(WeakRef)- a0x5d728b8
a0x55de298 -a0x5d728b8->  |  a0x45977a8  |
                          +--------------+

log (the "story" follows):
01 [qtbase:SmokeObject::enclose() this a0xfcd6fc0:] calling internalSexp
02 [qtbase:SmokeObject::internalSexp() this a0xfcd6fc0:] calling createSexp for env a0x415e370
03 [qtbase:SmokeObject::createSexp() this a0xfcd6fc0:]
04 [qtbase:SmokeObject::createSexp() this a0xfcd6fc0:] setting enclosing env a0x415e370 for env a0x66765b0
05 [qtbase:SmokeObject::internalSexp() this a0xfcd6fc0:] calling internalTable
06 [qtbase:SmokeObject::internalTable() this a0xfcd6fc0:] called
07 [qtbase:SmokeObject::internalTable() this a0xfcd6fc0:] calling _klass->createObjectTable(this)
08 [qtbase:SmokeObject::internalTable() this a0xfcd6fc0:] creating sexp for hashtab a0xfcd7730
09 [qtbase:ObjectTable::createSexp this a0xfcd7730:] wrapPointer(tb a0x45977a8)
10 [qtbase:wrapPointer:] wrapping a0x45977a8 into sexp a0x66765e8
11   [WEAK REF CHECK:] sexp a0x5d728b8 not ready, not marked
12 [R_MakeWeakRefC:] making weakRef a0x5d728b8 for key a0x66765e8
13 [qtbase:SmokeObject::internalTable() this a0xfcd6fc0:] internal table sexp a0x66765e8 created for hashtab a0xfcd7730
14 [qtbase:SmokeObject::internalSexp() this a0xfcd6fc0:] setting hashtab a0x66765e8 to sexp a0x66765b0
15 [envir.c: findFun] checkEnv: rho a0x66765b0, table a0x45977a8
16 findVarInFrame3 rho a0x66765b0, symbol {, table a0x45977a8
17 findVarInFrame3 rho a0x66765b0, symbol {, table a0x45977a8: a0x40a156c

18 [CheckFinalizers:] setting a0x5d728b8 as ready to finalize (was not ready, not marked)
19 [FORWARD_NODE:] marking node a0x5d728b8
20 [FORWARD_NODE:] marking node a0x66765e8
21  [WEAK REF CHECK:]sexp a0x5d728b8 ready, marked

22 [qtbase:SmokeObject::enclose() this a0xfcd6fc0:] calling internalSexp
23 [qtbase:SmokeObject::internalSexp() this a0xfcd6fc0:] calling createSexp for env a0x415e370
24 [qtbase:SmokeObject::createSexp() this a0xfcd6fc0:]
25 [qtbase:SmokeObject::createSexp() this a0xfcd6fc0:] setting enclosing env a0x415e370 for env a0x55de298
26 [qtbase:SmokeObject::internalSexp() this a0xfcd6fc0:] calling internalTable
27 [qtbase:SmokeObject::internalTable() this a0xfcd6fc0:] called
28 [qtbase:SmokeObject::internalSexp() this a0xfcd6fc0:] setting hashtab a0x66765e8 to sexp a0x55de298
29 [envir.c: findFun] checkEnv: rho a0x55de298, table a0x45977a8
30 findVarInFrame3 rho a0x55de298, symbol {, table a0x45977a8
31 findVarInFrame3 rho a0x55de298, symbol {, table a0x45977a8: a0x40a156c
32 [envir.c: findFun] checkEnv: rho a0x55de298, table a0x45977a8
33 findVarInFrame3 rho a0x55de298, symbol if, table a0x45977a8
34 findVarInFrame3 rho a0x55de298, symbol if, table a0x45977a8: a0x40a156c
35 [envir.c: findFun] checkEnv: rho a0x55de298, table a0x45977a8
36 findVarInFrame3 rho a0x55de298, symbol {, table a0x45977a8
37 findVarInFrame3 rho a0x55de298, symbol {, table a0x45977a8: a0x40a156c
38 [envir.c: findFun] checkEnv: rho a0x55de298, table a0x45977a8
39 findVarInFrame3 rho a0x55de298, symbol cat, table a0x45977a8
40 findVarInFrame3 rho a0x55de298, symbol cat, table a0x45977a8: a0x40a156c
41 findVarInFrame3 rho a0x55de298, symbol print.ls_str, table a0x45977a8
42 findVarInFrame3 rho a0x55de298, symbol print.ls_str, table a0x45977a8: a0x40a156c
43  [WEAK REF CHECK:] sexp a0x5d728b8 ready, marked
44 [R_RunWeakRefFinalizer(ts_rho2)] checkEnv: rho a0x55de298, table a0x45977a8
45 [qtbase:ObjectTable::~ObjectTable() this=a0xfe99fd0] extptr a0x6d5961c, tb a0xfe9a010
46 [R_RunWeakRefFinalizer(ts_rho2)-post1] checkEnv: rho a0x55de298, table a0x45977a8
47 about to run R_RunWeakRefFinalizer a0x5d728b8 (saved rho a0x6722bcc)
48 [R_RunWeakRefFinalizer(ts_rho2)] checkEnv: rho a0x55de298, table a0x45977a8
49 [qtbaseLInstanceObjectTable::~InstanceObjectTable() this a0xfcd7730] sexp a0x66765e8
50 [qtbase:ObjectTable::~ObjectTable() this=a0xfcd7730] extptr a0x66765e8, tb a0x45977a8
51 [R_RunWeakRefFinalizer(ts_rho2)-post1] checkEnv: rho a0x55de298, table a(nil)
52 [R_RunWeakRefFinalizer(ts_rho2)] checkEnv: rho a0x55de298, table a(nil)
53 [qtbase:ObjectTable::~ObjectTable() this=a0x45133e0] extptr a0x52f2348, tb a0x4513420
54 [R_RunWeakRefFinalizer(ts_rho2)-post1] checkEnv: rho a0x55de298, table a(nil)
55 [R_RunWeakRefFinalizer(ts_rho2)] checkEnv: rho a0x55de298, table a(nil)
56 [R_RunWeakRefFinalizer(ts_rho2)-post1] checkEnv: rho a0x55de298, table a(nil)
57 [envir.c: findFun] checkEnv: rho a0x55de298, table a(nil)
58 findVarInFrame3 rho a0x55de298, symbol cat, table a(nil)
59 [R_RunWeakRefFinalizer(ts_rho2)-post1] checkEnv: rho a0x55de298, table a(nil)

The problem is that the handler environment a0x66765b0, which is associated with the object table a0x45977a8 (lines 09-14) seems to get lost somehow (?), i.e. not seen anymore (for the last time at line 17). As a finalizer has been associated with the object table a0x45977a8 thru the "lost" environment a0x66765b0 (line 12), and the environment can't be reached in CheckFinalizers() (just guessing, no direct evidence), the weak ref is set as "ready to be finalized" (line 18). Then, the object table gets reused in another environment a0x55de298 (lines 22-28) and used heavily (lines 29-42). However, even though the object table a0x45977a8 is in use, it is finalized (lines 49-50) along with its wrapping external pointer a0x66765e8.

I do not know the reason why the environment a0x66765b0 disappears (or simply can't be reached). I'd happilly go deeper in it, but now I don't see where and how to look. I can also provide my hacked R/qtbase sources, if you like.

As a quick, dirty (and incorrect!) workaround, I tried to instantiate a new object table on each request (see https://github.com/tsieger/qtbase/tree/object_table_copy), which seems to work (no more crashing). However, it destroys the sharing of the object table, so it is not a proper solution.

Would it be possible to share the object table and enable proper finalization at the same time? I thought of some reference counting mechanism, but I don't know how to make callers release the refcounts. On the other hand, instantiating a new table on each request and delegating operations to some shared table would also be problematic - how to housekeep that shared table? (I don't understand why smoke object can't housekeep the object table, and just orphans it? Maybe, the answer would invalidate the shared object table idea.)

Would you prefer to investigate the cause of the problem (i.e. in R, how?), or adapt qtbase?

[lawremi]

Thanks for this in-depth analysis. I really wish I understood the problem. Perhaps it is that R decides it will free an object prior to actually calling the finalizer. This would cause qtbase to reuse the table without knowing that it was a "dead man walking".

If that is indeed the case, then it means that qtbase has made invalid assumptions all over the place, and it would be tricky to adapt the design. But I'm pretty sure that's not the case.

On Sun, Jun 22, 2014 at 12:16 PM, Tomas Sieger notifications@github.com wrote:

I investigated it more deeply and found out the problem is that the object table gets finalized as the environment it is associated to is considered unreachable in finalization (even though the object table is in use at that time in another environment).

I've injected R and qtbase with some debugs and mention some output here for reference:

key: a0x45977a8 - R_ObjectTable instance a0x66765e8 - SEXP R external pointer wrapper for a0x45977a8 a0x5d728b8 - WeakRef for a0x66765e8 a0x66765b0 - SEXP environmnent (perhaps of the "paintEvent" handler) a0x55de298 - SEXP environmnent (perhaps of another invocation of the "paintEvent" handler) a0x415e370 - parent environment of the two environments above

graphically: +- a0x66765e8 -+ a0x66765b0 -(hashtab)--> | wrapped: | <-(WeakRef)- a0x5d728b8 a0x55de298 -a0x5d728b8-> | a0x45977a8 | +--------------+

log (the "story" follows): 01 [qtbase:SmokeObject::enclose() this a0xfcd6fc0:] calling internalSexp 02 [qtbase:SmokeObject::internalSexp() this a0xfcd6fc0:] calling createSexp for env a0x415e370 03 [qtbase:SmokeObject::createSexp() this a0xfcd6fc0:] 04 [qtbase:SmokeObject::createSexp() this a0xfcd6fc0:] setting enclosing env a0x415e370 for env a0x66765b0 05 [qtbase:SmokeObject::internalSexp() this a0xfcd6fc0:] calling internalTable 06 [qtbase:SmokeObject::internalTable() this a0xfcd6fc0:] called 07 [qtbase:SmokeObject::internalTable() this a0xfcd6fc0:] calling _klass->createObjectTable(this) 08 [qtbase:SmokeObject::internalTable() this a0xfcd6fc0:] creating sexp for hashtab a0xfcd7730 09 [qtbase:ObjectTable::createSexp this a0xfcd7730:] wrapPointer(tb a0x45977a8) 10 [qtbase:wrapPointer:] wrapping a0x45977a8 into sexp a0x66765e8 11 sexp a0x5d728b8 not ready, not marked 12 [R_MakeWeakRefC:] making weakRef a0x5d728b8 for key a0x66765e8 13 [qtbase:SmokeObject::internalTable() this a0xfcd6fc0:] internal table sexp a0x66765e8 created for hashtab a0xfcd7730 14 [qtbase:SmokeObject::internalSexp() this a0xfcd6fc0:] setting hashtab a0x66765e8 to sexp a0x66765b0 15 [envir.c: findFun] checkEnv: rho a0x66765b0, table a0x45977a8 16 findVarInFrame3 rho a0x66765b0, symbol {, table a0x45977a8 17 findVarInFrame3 rho a0x66765b0, symbol {, table a0x45977a8: a0x40a156c

18 [CheckFinalizers:] setting a0x5d728b8 as ready to finalize (was not ready, not marked) 19 [FORWARD_NODE:] marking node a0x5d728b8 20 [FORWARD_NODE:] marking node a0x66765e8 21 sexp a0x5d728b8 ready, marked

22 [qtbase:SmokeObject::enclose() this a0xfcd6fc0:] calling internalSexp 23 [qtbase:SmokeObject::internalSexp() this a0xfcd6fc0:] calling createSexp for env a0x415e370 24 [qtbase:SmokeObject::createSexp() this a0xfcd6fc0:] 25 [qtbase:SmokeObject::createSexp() this a0xfcd6fc0:] setting enclosing env a0x415e370 for env a0x55de298 26 [qtbase:SmokeObject::internalSexp() this a0xfcd6fc0:] calling internalTable 27 [qtbase:SmokeObject::internalTable() this a0xfcd6fc0:] called 28 [qtbase:SmokeObject::internalSexp() this a0xfcd6fc0:] setting hashtab a0x66765e8 to sexp a0x55de298 29 [envir.c: findFun] checkEnv: rho a0x55de298, table a0x45977a8 30 findVarInFrame3 rho a0x55de298, symbol {, table a0x45977a8 31 findVarInFrame3 rho a0x55de298, symbol {, table a0x45977a8: a0x40a156c 32 [envir.c: findFun] checkEnv: rho a0x55de298, table a0x45977a8 33 findVarInFrame3 rho a0x55de298, symbol if, table a0x45977a8 34 findVarInFrame3 rho a0x55de298, symbol if, table a0x45977a8: a0x40a156c 35 [envir.c: findFun] checkEnv: rho a0x55de298, table a0x45977a8 36 findVarInFrame3 rho a0x55de298, symbol {, table a0x45977a8 37 findVarInFrame3 rho a0x55de298, symbol {, table a0x45977a8: a0x40a156c 38 [envir.c: findFun] checkEnv: rho a0x55de298, table a0x45977a8 39 findVarInFrame3 rho a0x55de298, symbol cat, table a0x45977a8 40 findVarInFrame3 rho a0x55de298, symbol cat, table a0x45977a8: a0x40a156c 41 findVarInFrame3 rho a0x55de298, symbol print.ls_str, table a0x45977a8 42 findVarInFrame3 rho a0x55de298, symbol print.ls_str, table a0x45977a8: a0x40a156c 43 sexp a0x5d728b8 ready, marked 44 [R_RunWeakRefFinalizer(ts_rho2)] checkEnv: rho a0x55de298, table a0x45977a8 45 [qtbase:ObjectTable::~ObjectTable() this=a0xfe99fd0] extptr a0x6d5961c, tb a0xfe9a010 46 [R_RunWeakRefFinalizer(ts_rho2)-post1] checkEnv: rho a0x55de298, table a0x45977a8 47 about to run R_RunWeakRefFinalizer a0x5d728b8 (saved rho a0x6722bcc) 48 [R_RunWeakRefFinalizer(ts_rho2)] checkEnv: rho a0x55de298, table a0x45977a8 49 [qtbaseLInstanceObjectTable::~InstanceObjectTable() this a0xfcd7730] sexp a0x66765e8 50 [qtbase:ObjectTable::~ObjectTable() this=a0xfcd7730] extptr a0x66765e8, tb a0x45977a8 51 [R_RunWeakRefFinalizer(ts_rho2)-post1] checkEnv: rho a0x55de298, table a(nil) 52 [R_RunWeakRefFinalizer(ts_rho2)] checkEnv: rho a0x55de298, table a(nil) 53 [qtbase:ObjectTable::~ObjectTable() this=a0x45133e0] extptr a0x52f2348, tb a0x4513420 54 [R_RunWeakRefFinalizer(ts_rho2)-post1] checkEnv: rho a0x55de298, table a(nil) 55 [R_RunWeakRefFinalizer(ts_rho2)] checkEnv: rho a0x55de298, table a(nil) 56 [R_RunWeakRefFinalizer(ts_rho2)-post1] checkEnv: rho a0x55de298, table a(nil) 57 [envir.c: findFun] checkEnv: rho a0x55de298, table a(nil) 58 findVarInFrame3 rho a0x55de298, symbol cat, table a(nil) 59 [R_RunWeakRefFinalizer(ts_rho2)-post1] checkEnv: rho a0x55de298, table a(nil)

The problem is that the handler environment a0x66765b0, which is associated with the object table a0x45977a8 (lines 09-14) seems to get lost somehow (?), i.e. not seen anymore (for the last time at line 17). As a finalizer has been associated with the object table a0x45977a8 thru the "lost" environment a0x66765b0 (line 12), and the environment can't be reached in CheckFinalizers() (just guessing, no direct evidence), the weak ref is set as "ready to be finalized" (line 18). Then, the object table gets reused in another environment a0x55de298 (lines 22-28) and used heavily (lines 29-42). However, even though the object table a0x45977a8 is in use, it is finalized (lines 49-50) along with its wrapping external pointer a0x66765e8.

I do not know the reason why the environment a0x66765b0 disappears (or simply can't be reached). I'd happilly go deeper in it, but now I don't see where and how to look. I can also provide my hacked R/qtbase sources, if you like.

As a quick, dirty (and incorrect!) workaround, I tried to instantiate a new object table on each request (see https://github.com/tsieger/qtbase/tree/object_table_copy), which seems to work (no more crashing). However, it destroys the sharing of the object table, so it is not a proper solution.

Would it be possible to share the object table and enable proper finalization at the same time? I thought of some reference counting mechanism, but I don't know how to make callers release the refcounts. On the other hand, instantiating a new table on each request and delegating operations to some shared table would also be problematic - how to housekeep that shared table? (I don't understand why smoke object can't housekeep the object table, and just orphans it? Maybe, the answer would invalidate the shared object table idea.)

Would you prefer to investigate the cause of the problem (i.e. in R, how?), or adapt qtbase?

— Reply to this email directly or view it on GitHub https://github.com/ggobi/qtbase/issues/21#issuecomment-46789926.

[tsieger]

qtbase resuses the table because it assumes R still references it, which is definitely not true in R 3.1.0, in which the table gets finalized for some unknown reason.

It seems that my hack (https://github.com/tsieger/qtbase/tree/object_table_copy) is not incorrect - even though I create a new table on each request, all the tables act through the shared smoke object. This solution does not seem to leak, while the memory overhead is relatively small. Would you please consider to accept this hack as a (temporary) solution (until a better one appears)? I understand that it would not be nice from the design perspective, but I don't see another option to make qtbase working on R 3.1.0, which is, unfortunately, the most commonly used version nowadays.

[lawremi]

Your fix should work and we very much appreciate your help. The main concern is that we do not understand why this is happening, so there could be situations where the external "_sexp" falls victim, as well, and that would require broader changes.

Please go ahead and submit a pull request. You might consider changing the QList of references to a QSet, just because it seems more semantically appropriate.

On Fri, Jun 27, 2014 at 12:08 AM, Tomas Sieger notifications@github.com wrote:

qtbase resuses the table because it assumes R still references it, which is definitely not true in R 3.1.0, in which the table gets finalized for some unknown reason.

It seems that my hack ( https://github.com/tsieger/qtbase/tree/object_table_copy) is not incorrect - even though I create a new table on each request, all the tables act through the shared smoke object. This solution does not seem to leak, while the memory overhead is relatively small. Would you please consider to accept this hack as a (temporary) solution (until a better one appears)? I understand that it would not be nice from the design perspective, but I don't see another option to make qtbase working on R 3.1.0, which is, unfortunately, the most commonly used version nowadays.

— Reply to this email directly or view it on GitHub https://github.com/ggobi/qtbase/issues/21#issuecomment-47314007.

[tsieger]

Thank you. I replaced QList with QSet (thanks for the notice; QList was just a relic) and have submitted two pull requests (one for the master, one for the qt4 branch).

Hopefully, more light will be shed on this issue some day and this rather ad-hoc hack will become unnecessary.

[lawremi]

Pull requests have been merged. At this point, we should probably make a release of the 5.x branch to CRAN. Hopefully we get to the bottom of this some day.