tclSE / sebres tcl-fork (still private repo)

[sebres]

This should be a collection of all kinds of changes, fixes, refactorings in relation of the differences between both editions, and should basically describe my aim resp. the inspiration for writing of own tcl-fork (tclSE).

Not part of this repository (no public access ATM)

[Q] Why:

What I always had been missing in original tcl edition, was the inconsistent handling round about oo-objects of all kinds (no matter which engine - xotcl, itcl, etc.), the absence of many primitives and goodies that are already long time a part of many other modern languages.

Nevertheless I love tcl (and it has always been and remains my favourite language), but...

I mean:

• bad integration of oo-subsystems with core (become better since v.8.7, but IMHO still far from the ideal)
• very bad implementation of references: oo-objects are not references in sense of tcl, in opposite to all other "objects" in tcl, whether primitives or complexes, (including many internal handles of many back-end implementation) that are Tcl_Obj (with internal representation that corresponds to the real object).
  But all the tcl oo-objects are commands with namespaces (yes, I mean the objects not the classes). And so have many disadvantages, namely:
  • that's why they are very slow - object construction and destruction especially by many objects (because of command/namespace creating or removing per object), raises thereby epoch change (on several namespace, that are static normally) with recompiling or invalidation of many internal things in core.
  • each extension round about of namespace resp. command handling, makes oo-objects slower and slower
  • they are quasi "eternal" resp. the developer have to worry about and should always explicit destroy they (no really ref-counting in sense of tcl-core, because does not work as with other above mentioned pure tcl-objects).
  • as result of this, there is no last reference to oo-object, with this the object would be deleted also - the reference is a quasi pointer to Tcl_Command object, so leaving the scope (in tcl call-frame) does not free all oo-objects created there but not referenced somewhere else.
  • tcl uses system and namespace related literals by compiling of source code, so by nesting classes resp. in methods applied to many different derivations or in inheritance hierarchy, it causes often re-invalidation of internal command representation, so each time "find command" and co (because of the strange algorithm of command resolver, which maps it from one namespace to another and of course because of the same Tcl_Obj, well is a single literal). [Important] Especially it concerns very often used objects resp. literals like {}. So this object may switch its internal representation multiple times during of execution of small code pieces. Recently it was tclDictType now it is a tclListType, and below it will be unicode object. Bad is that thereby previous representations are lost. You can increment refCount of such objects, but it does not prevent the switching of the internal representation. So if some code currently works with dict/list innards and the object switches its type in-between during the iteration (e. g. any of the TclEvals executed in-between, something like foreach), then segfaults are the best what will happen here (worse if it occurs much later, so no chance to find the actual error). The only way to prevent such situations is to duplicate objects right before (and original tcl does it also very often). But, IMHO it is a very wrong resp. questionable way. Thereby in the tclSE the only reasons to duplicate objects would be:
  • creating a copy of object (e. g. to change it but let the origin unchanged);
  • (conditionally) sending an object to another thread (only objects with not thread-safe representation);
  • (conditionally) sending an object to another interpreter (only objects with not interp-safe representation, e. g. contains some interp references, etc.);
  • made more difficult to implement a reasonable debugging, profiling, and coverage tool.
  • does not has many goodies of other languages like mixins, treats, promises, closures, lambdas (I mean real normal lambdas and not tcl-own coroutines, that are scopeless), properties, decorators, but even very simple things like a generators resp. iterators have been missing also. Yes, I know tcl yield (again coroutines) and no I mean something like class or type "iterable";
  • IMHO, very bad support of C-integration within oo-objects subsystem (if building "mixins" of C-code with tcl-code inside the same class or its inheritances).
  • etc, etc, etc.
• sporadically loss of internal representation by suddenly in-between converting to something other (e. g. somewhat needs unicode representation and converts object to string, thereby destroys a real object referenced with this Tcl_Obj via internal representation). I mean, some objects should get normally something like main representation, that would be not removable (only with object self by refCount=0). Therefore only one way to safe representation - to reference it somewhere else (lists or hashes inside modules, even commands or namespaces), what makes the essentially good ref-counting mechanism useless again (BTW. I know all the cons and pros of ref-counting, please don't disturb me with this theme).
• deficiency of few small primitives, that can good facilitate the work - singletons, lazy loaders and constructors, etc. But also other type of binary integration aka libfi and co (I know about ffidl, that is good but... what I mean here is oo-integration again).
• absence of preprocessor possibility (one could encapsulate some code-scope or procedure within something like if $GLOBAL_VAR {...}, but it is not the same as real preprocessor).

This list is not completed and will grow, once I'll remember about something again...

[Q] What is tclSE:

Currently it is my private tcl engine, that resolves good almost all abovementioned problems. It is fast, robust and comfortable. But thereby not 100% compatible.

Here is a small list of important changes (only):

• own much better command, namespace and variable resolvers; Newer command resolver has made possible that the tclSE oo-objects are Tcl_Obj's (only classes, structs, mixins etc are namespaces).
• better interpreter tracing, so debugger, profiler, code coverage and variable watching also for and inside the byte-code execution (mentioned in https://github.com/flightaware/Tcl-bounties/issues/19).
• many goodies (above mentioned sugars) are implemented and works out of the box from kernel (in-core implementation); another are modules with lazy auto-loading.
• Tcl_ObjType is much more powerful: more as one internal representation, possibility for the cross-thread operable objects (without duplication, simple GIL with atomic ref-counting, other tcl-obj pool, shared-mem (planned), etc.
  It describes now also which kind of references or pointing features this type supports (ref, smart, shared or weak object pointers).
• Tcl-code (or essentially its byte code) can be shared between threads and interpreters now - once compiled it'll be minimal invalidated by little portion attaching to another thread/interp (some references in thread- resp. interpreter epoch-related data). Cons - can not be unloaded (eternal), but replaceable. Pros are obvious. Additionally it allows to save byte-code as binary, organize auto-loading e.g. in/from script.tclc (and auto-recompile it by checking of modification time of original), or even encrypting or signing of the byte code additionally etc;
• Except many primitives enhancements, implemented also many new complexes types, like struct, hash (dict with structure definition), record (ordered hash), ... but also iterable, promise (event based or not), properties, call-, access- singletons and anchors, etc;
• For example cacheable anchors allow to execute something once by first access/call and to bind its result under object internal representation, linked to the anchored object (without if something not exists ..., searching somewhere in hashes, global and namespace references etc.)
  For example it's a way to build dynamically calculated constants or the constants dependent on some value, object even class.
• you can add your own preprocessor, that will be executed by loading with source (if shared - only once);
• scope (callFrame) bound (closure like) code-pieces, lambdas and sub-proc's.
• new auto index handling with lazy loading + loading by module / namespace access:

  ## by exact command call (command, constructor of object or type creation):
  set auto_index(::Cmd::Or::Class::Or::Namespace) [list source ... | <loader-cmd>]
  ## by proc of namespace or class (you do not need mention each single function):
  set auto_index(::Class::Or::Namespace::*) [list source ... | <loader-cmd>]
  ## by access of member of namespace or class (or namespace unknown within) :
  set ns_auto_index(::Class::Or::Namespace) [list source ... | <loader-cmd>]

• several loose and strict modes settings (per type, namespace/class or object, or even scope) - e. g. by variable resolving, so manages accessibility of member variable of class object with and without this, e. g. lets better differentiate between local and member variable in strict mode but also allow easier implement tcl-code, that would be inside backwards compatible to several other more loose oo-subsystems like itcl (that allows accessing or even rewriting member variables without this).
• many special predicates for proc, command or method declarations: e. g. virtual for methods may be redefined in inheritances, or override that allow however to replace a non-virtual method in current inheritance (also if that will be called in inherited class without this). Or for example predicate silent, that allows to disable call stack expansion inside by throwing errors from this scope (if the routine is "safe", but throws many exceptions, and the stack of calls inside it should be definitely not included). This handling may be disabled if running under debugger.

To be continued...

[sebres]

About sub procs. Here is a small comparison between original tcl and tclSE:

Original tcl-code (subroutine sub faked):

proc test {a b} {
  set c $a; incr c $b
  set sub {
    set prev_c $c
    incr c 100
    puts "c != a + b: $c != $a + $b"
    set c $prev_c
  }
  puts "== 1st time: =="
  puts "c == a + b: $c == $a + $b"
  set i 3; while {[incr i -1]} $sub
  puts "c == a + b: $c == $a + $b"

  puts "== 2nd time: =="
  puts "c == a + b: $c == $a + $b"
  set i 3; while {[incr i -1]} $sub
  puts "c == a + b: $c == $a + $b"
}

% test 10 20
== 1st time: ==
c == a + b: 30 == 10 + 20
c != a + b: 130 != 10 + 20
c != a + b: 130 != 10 + 20
c == a + b: 30 == 10 + 20
== 2nd time: ==
c == a + b: 30 == 10 + 20
c != a + b: 130 != 10 + 20
c != a + b: 130 != 10 + 20
c == a + b: 30 == 10 + 20

tclSE-code (subroutine sub is real sub proc), same result here:

proc test {a b} {
  set c $a; incr c $b
  sub-proc sub {} {
    ## c is local variable now after declaration bellow ("set" does it not):
    var c $c
    incr c 100
    puts "c != a + b: $c != $a + $b"
  }
  puts "== 1st time: =="
  puts "c == a + b: $c == $a + $b"
  set i 3; while {[incr i -1]} { sub }
  puts "c == a + b: $c == $a + $b"

  puts "== 2nd time: =="
  puts "c == a + b: $c == $a + $b"
  set i 3; while {[incr i -1]} { sub }
  puts "c == a + b: $c == $a + $b"
}

The pros are obvious:

• is callable as proc (not as body or code piece)
• can take arguments
• can contains local variables, so developer have not to worry about saving uplevel scope vars, it can be used e. g. recursively without organizing of the queue for the call stack, etc.
• can access and change variables of the uplevel scope until not overriden with local or var (also upvar can be used)