Garbage Collection - causing problems.

[markmpx]

David,

Is there any easy way to disable garbage collection globally other then having to manually x: POINTER [1] everywhere thru the source code ?

I have converted the Latest Oberon07 compiler to voc and it produces strange results intermittently on exactly the same code, so it seems to be that its either a) unintialised variable or b) memory corruption.

I recloned the repo just incase and rebuilt voc. Am on Ubuntu 18.04 LTS (latest patches) running in a Xeon workstation that has ECC memory and i know ECC is working, so it cant be memory corruption on this machine. I have multiple identical workstations and it happens on all of them, so i dont think its the hardware in my machine.

Am running gcc 7.3.0 on x86_64 machine. These workstations have 32 Gigs of memory, so am not concerned about garbage collection on a short running program currently.

I have a number of tests that reliably reproduce random SEGFAULTS thru to incorrect behaviour.

If i can prove that this is the problem happy to help track them down and get them fixed.

Am currently converting the source code to the latest Oxford Oberon compiler, to track down any unintialised variables. (This compiler has fabulous error messages). Unfortunately the code the Oxford Oberon produces runs more than 100 times slower on some of my code, so i really want to stay with voc in the long run.

all my voc usage is voc -V -OC blah.mod

Any ideas ?

[dcwbrown]

Not exactly but it is easy to rebuild voc with src/runtime/Heap.mod PROCEDURE GC edited to do nothing (or maybe show a message and do nothing). I've just tried this by hacking the first line after BEGIN from

IF (lockdepth = 0) OR (lockdepth = 1) & ~markStack THEN

to

`IF FALSE & ((lockdepth = 0) OR (lockdepth = 1) & ~markStack) THEN

and voc builds and all tests pass fine.

-- Dave.

[markmpx]

Dave,

Thanks for your quick response. I shall try it and let you know the results.

[markmpx]

Dave,

I tried your quick hack to Heap.Mod and it solved the SEGFAULTS. I have now reverted this change, and use POINTER [1] which also solves the issue.

In the process i read thru every issue (both opened and closed).

Should i close the issue, or do you ? (My apologies, i am still very much a github newbie).

and finally, Thanks very much to yourself and all the others who produced such an excellent compiler.

[dcwbrown]

You're welcome to close it, but first could I ask you to confirm whether ypou were able to pinpoint a real garbage collection issue? Or could it have been the Oberon compiler itself manipulating pointers directly?

[markmpx]

Dave,

I believe what voc is doing is correct, so i dont believe that it is a bug. When the Oberon07 based compiler i am using (which i compile and run with voc) uses NEW() to allocate memory, the C code generated by voc calls NEWREC() in Heap.mod. NEWREC will attempt to allocate a spare block of memory from its free lists and if none, it will call GC (garbage collector) and then if still no memory, allocate more.

So GC (garbage collection) in Heap.mod is triggered by calls to NEW() to allocate more memory. I modified the root pointers for all the data structures allocated by my compiler with POINTER [1] to inhibit the GC trying to collect and free this memory and it resolved my SEGFAULTS.

My memory of writing and dealing with garbage collection over the years is that there can be times where the GC may attempt to free memory that you are still using. I strongly suspect that this occurred with voc and caused the SEGFAULTS.

This is not a problem with voc but a problem that can exist with most all garbage collectors under the right pathological circumstances, is my understanding. My Oberon07 compiler does NOT do any pointer arithmetic. I suspect the voc GC is doing a fine job with memory allocation used internally (call by value records, arrays and return values (in expressions)). Please correct me if i am wrong.

I have now had time to look at Heap.mod in more detail and upon a quick inspection of the code it appears to be a typical Mark&Sweep garbage collector. The code seems reasonable and well written. Having written garbage collectors myself (tho not in detail for some years) my impression is that its reasonable.

Over the years experience has taught that i am almost always better off writing my own routines to allocate and free memory, and to deal with all the problems that can exist directly. I know that this is not considered normal these days, but i have usually had to write code that runs where very tight control of the machine was required (or in an embedded environment, where memory was minimal). My dislike for garbage collectors is due to a) the non deterministic nature of when the GC will run and the number of cpu cycles wasted b) GC schemes require extra memory for bookkeeping and this is wasteful in a memory constrained environment. c) GC schemes may not always get it right and may reclaim memory in use, or fail to reclaim memory that it should have. d) GC's can easily cause memory fragmentation. e) Proving the correctness of the GC can be difficult and/or complex. f) It can be difficult to add debugging to your GC to allow you to diagnose your problems. I assume a thru f account for the popularity of tools like valgrind etc these days.

Dave, i found your info on how Oberon files work very useful initially. Perhaps it may be worth adding more info in the doc directory, dealing with memory and garbage collection.

A few final comments on Heap.mod:

1. heapSize0 - controls the initial size of the heap and the minimum size of each heap expansion. Given that my compiler does a vast number of calls to NEW(), increasing this substantially, drastically lowers the number of attempts to call the GC. I notice there are heuristics in Heap.mod to attempt to prevent heap thrashing. There is no one size fits all for heapSize0 so i believe it should be left as it is.

2. I note that norayr in previously closed issues, explained about the POINTER [1] and also that its possible to replace the garbage collector with your own solution. If i have further issues i expect to do this. There are numerous simple effective and efficient schemes that can be used in most cases. Sadly i suspect that none of them are taught these days, resulting in programmers being so reluctant to deal with these issues directly.

If i have further issues, i will write a replacement for Heap.mod, keeping it a simple as possible and hopefully serving as a documented example to others, of how to do this.

As i continue to use voc, i am more convinced that its close to what i would consider an industrial strength compiler. (which is saying a lot for me)

Finally, are you familiar with the nim? (google nim lang) It in many ways inherits the ideas of wirth/oberon.

Sorry for the wall of text.

[dcwbrown]

Thanks for the detailed comments Mark.

I believe the Vishap GC is intended to free only memory that is no longer accessible in any way via global pointers and pointers on the stack. If it is freeing any such memory then it's really a bug.

Is it possible that the Oberon07 code that you are working on casts pointers to integers, hangs on to them as integers for a while, then casts them back to pointers?

It seems unlikely, but that is one way that a GC could fail to realise a memory block is in use. (The Oberon compiler does reuse RECORD fields for multiple purposes, but I don't remember it repurposing an integer feld as a pointer.)

I've done some work on parts of the Oberon 2 compiler, but little on the garbage collector and I certainly don't claim to understand it very well. I'm not up to documenting the GC process.

A well documented, straightforward heap.mod would be great.

I'm glad my notes on files were helpful - I found it most confusing at first, e.g. the function 'Close' seems particularly misleadingly named.

I've liked what I've seen of Nim though I haven't looked in detail or built anything with it. Then again if you're looking to avoid GC, Rust seems to be the language best able to manage pointers safely.

Cheers -- Dave.

[markmpx]

Dave,

Thanks for your (as always) insightful comments.

I really need to get my primary job done currently, but i suspect that its an 80% chance thats its a bug in voc (solved by POINTER [1]), and a 20% chance its the Oberon07 compiler doing naughty stuff.

While working on my main project, i will keep this in mind and hopefully revisit this at a later time, and finally track down the cause. It would be great to finally understand in full what was happening. I suspect i will have to fully understand and document Heap.Mod to do this.

I have looked at Rust but i think the cognitive overload is just too high to make it practical. (I find it a MOST unpleasant experience using rust, but i must agree they solved an age old problem).

I am hopeful that Nim may eventually solve the pointer safety problem that rust has solved in far more unobtrusive way.

regards -- Mark