Safe Derivable C++ classes for Clasp

[drmeister]

13:46 <drmeister> flash-  Do you use derivable classes in clbind?
13:46 <drmeister> Where you can define a Common Lisp class that can derive from a C++ class?   It's the darkest of dark magic in clbind.
13:47 <drmeister> It's required by the static analyzer and it's giving me grief in both MPS and the new fast dynamic cast code that I added to the Boehm GC version of Clasp
13:48 <flash-> I don't think I currently do. IIRC I tried to do that, but didn't manage to.
13:49 <flash-> I think you convinced me to stay out of that rabbit hole unless absolutely necessary.
13:49 <drmeister> Hmm, it's something I may need to rethink.   Many C++ libraries provide the programmer with classes that they are supposed to subclass - it's something I very, very much want to do in Common Lisp and I currently can do.  But it pushes the edge of what is possible and what is sane.
13:50 <drmeister> I kind of remember that.
13:51 <flash-> I think stassats might be interested in that. Qt relies heavily on the user subclassing Qt classes.
13:52 <stassats> i'm not using clasp for interfacing qt
13:52 <stassats> just for parsing c++
13:52 <drmeister> What stassats said.
13:53 <drmeister> But - I or someone else would use it for Qt in Clasp
13:53 <flash-> Ah, OK, I misunderstood or misremembered that.
13:53 <drmeister> Nope - no problem.
13:53 <stassats> or i'm not using clasp at all
13:53 <stassats> to be precise
13:53 <flash-> I thought the ultimate goal was directly interfacing with Qt.
13:54 <drmeister> It would be for Clasp - for other CL's it would be to create a better FFI for Qt
13:55 <drmeister> But it's outside my bailiwick 
13:55 <drmeister> I'm just happy to have company :-)
13:56 <flash-> Well, the ultimate goal is of course the binding generator that automatically builds the Qt binding. ;-)
13:57 <drmeister> Anyway if stassats or anyone did go down that road it would still be important to them - because the clang AST tooling library relies on this capability as well.
13:57 → guest925 joined (~username@2600:100e:b02a:6f29:0:d:b7e2:db01)
13:58 <jackdaniel> if library is compiled with gcc it can't be linked with a program build with llvm? (sorry for potential ignorance on this matter)
13:58 <drmeister> jackdaniel: I think it can be.  
13:59 <drmeister> Although the very latest release of gcc threw a wrench into that.
13:59 <jackdaniel> uhm
14:00 <drmeister> Hmm, there might be a very, very simple fix to this.
14:00 <drmeister> Please humor me for a moment while I think aloud.
14:01 <jackdaniel> you mean - shall I tell you a joke? or humor has a different meaning in this context?
14:01 <drmeister> In clasp/src/clbind/include/derivable.h   I define the template <class Alien> class Derivable : public core::Instance_O, public Alien {...}
14:03 <drmeister> It merges the Clasp exposed class Instance_O (that contains a vector of slots and a CLASS and all CLOS instances are instances of this class) with an arbitrary class "Alien"
14:03 ⇐ guest925 quit (~username@2600:100e:b02a:6f29:0:d:b7e2:db01) Ping timeout: 246 seconds
14:04 <drmeister> One thing I can't do is allow instances of Derivable<Alien> to move in memory - I can't guarantee that whatever C++ library defined Alien doesn't have raw pointers to instances of Alien.
14:04 <drmeister> I can fix that because Clasp defines another template class that defines the GC properties of each class managed by the GC.
14:06 <drmeister> Here is an example for the Cons_O class:
14:06 <drmeister> https://www.irccloud.com/pastebin/tZjjiTRz/
 Plain Text • 7 lines raw | line numbers 

1
2
3
4
5
6
7
template <>                                                     

struct gctools::GCInfo<core::Cons_O> {                          

  static bool constexpr NeedsInitialization = false;            

  static bool constexpr NeedsFinalization = false;              

  static bool constexpr Moveable = true;                        

  static bool constexpr Atomic = false;                         

};                                                              

14:06 <drmeister> gctools::GCInfo defines what the GC can do with the class core::Cons_O
14:07 <drmeister> This one says it doesn't need to be initialized, it doesn't need to be finalized, it can be put in moveable pool and it is not atomic (it contains pointers to other GC'd objects).
14:07 <drmeister> I just set Moveable = false   for every Derivable<Foo> class
14:08 <drmeister> Ah, there's another problem.
14:08 <drmeister> There's liveness.
14:09 <drmeister> If there are no CL pointers to a Derivable<Foo> class - then it will be collected, even if the alien C++ library has pointers to the Alien object.   That's not a trivial problem.
14:09 <drmeister> That's your typical "I have to merge two memory models - someone please shoot me".
14:10 <jackdaniel> (make-instance 'gun)
14:10 <jackdaniel> o, I told a joke :)
14:11 <drmeister> The other idea would be to put it in the root-object pool, where it has to be explicitly deleted.
14:11 <drmeister> I don't have a GCInfo field for that - but I could.
14:12 <drmeister> It's better to risk having objects accumulate than having them disappear when they are still being used.
14:12 <flash-> I don't think there is a general solution to this problem.
14:13 <drmeister> Also (ROOM) could be used to report on how many Derivable objects are alive.
14:13 <drmeister> flash-: Yeah
14:13 <flash-> Because C++ libraries themselves handle things differently.
14:14 <drmeister> Yup.  C++ is all about    g = new Gun();  g.shoot_foot();
14:14 <flash-> There might be C++ libraries that create classes and tells the user to delete them, there might be C++ libraries that creates classes and wants to delete them themselves, and there are libraries that have the user create classes, but deletes the classes get deleted by the library (Qt does this).
14:15 <flash-> (- deletes)
14:16 — jackdaniel wonders if drmeister used "." instead "->" in purpose (in any case - funny ;D)
14:16 <drmeister> jackdaniel: You are correct - that code would not compile :-)
14:17 <flash-> So ultimately the best compromise would IMO be to have a template flag for the person who writes the binding, with which he can decide whether an instantiation should be collected or put into the root-object pool.
14:17 — drmeister relies heavily on the C++ compiler
14:18 <drmeister> Yeah and a   Collectable = true/false field would do the job --   or do you think   RootObject = false/true would be more evocative?
14:19 <flash-> Collectable sounds fine to me.
14:20 <drmeister> Collectable=false   would override Moveable = true
14:20 <drmeister> ExplicitDelete = true    might be even more evocative
14:20 → guest925 joined (~username@173-14-25-94-Colorado.hfc.comcastbusiness.net)
14:20 <flash-> Yes.
14:22 <drmeister> Whats useful about using C++ template functions for this purpose is that I can define the default policy and then programmers can define more specialized policies for their own classes.
14:24 <drmeister> The next problem is inheritance.    Every Derivable<Foo> template class inherits from Instance_O and currently that needs to be compiled into the executable.
14:26 <drmeister> But I need to do something else with Instance_O.   Every class has a "Kind" integer value associated with it and it's written into the header of every object managed by the GC
14:26 <drmeister> The Kind value is assigned by the static analyzer.
14:27 <drmeister> Instance_O needs to be special, with a very large Kind value and CLOS classes will get their own unique Kind value that will never be reused.   This will allow Clasp to implement beach's generic function dispatch method.
14:28 <drmeister> Derivable<...> classes will also be given a Kind value larger than Kind(Instance_O)
14:29 <drmeister> Then it will be trivial to determine if a instance of Derivable<...> can be cast to a Instance_O
14:30 <drmeister> I may need to add a new template class to denote this - something like template <class C> InheritsFromInstance<C> { ... };
14:31 <drmeister> Then the allocators will know to assign a Kind value that from the next available Instance_O derived class Kind.
14:31 <drmeister> I think I see the shape of this.
14:32 <drmeister> It will take a little rearranging.
14:33 <flash-> Well, I only understood about half of it, but if it leads to something, it sounds good to me.
14:33 <drmeister> No problem - it would take longer to explain it all.  It's involving internal Clasp structures that evolved over years.
14:35 <drmeister> The key is what I have is not to far away from what I want  (1) Safe CL classes that derive from C++ classes (2) Faster generic function dispatch.
14:36 <drmeister> I have all the different memory pools (collectable/moveable, collectable/non-moveable, non-collectable/non-moveable  [non-collectable/moveable is a nonsense combination])
14:39 <drmeister> The allocators (in clasp/src/gctools/include/gcalloc.h) use the GCInfo template class to allocate in the appropriate pools...
14:39 <drmeister> Except they don't utilize the non-collectable/non-moveable pool - I have to allocate within that pool explicitly
14:40 <drmeister> And currently the Kind value they write into the header of each object is hard-wired and assigned by the static analyzer.
14:40 <drmeister> So the static analyzer will need to be altered slightly
14:41 <drmeister> Then the question is:  Do it now or after I release.  
14:42 <drmeister> Currently only the asttooling facility uses it.
14:42 <drmeister> And I only need that to run the static analyzer.

[drmeister]

I changed the GCInfo struct and removed the Moveable and Atomic properties and consolidated them into a Policy property that can have one of the values:

• atomic == Object contains no internal tagged pointers, is collectable
• normal == Object contains internal tagged pointers, is collectable
• collectable_immobile == Object cannot be moved but is collectable
• unmanaged == Object cannot be moved and cannot be automatically collected

collectable_immobile may be a useless category - how/why would an object be collectable but not moveable?

The new category is "unmanaged" - this tells the garbage collector to allocate instances of this class in a pool whose contents are not moved or collected by the GC. To remove instances of these classes use (gctools:deallocate-unmanaged-instance obj). DO NOT reference obj after this function is called. If this function is called on any instance of a class that is not unmanaged - nothing happens, it's a nop.

I need to add finalizers so that containers that contain unmanaged instances can deallocate them when the container is finalized - this will help prevent memory leaks.