Lifetime checking on user-defined managed types / smart pointers

[BryantLam]

General musings on whether and how lifetime-checking should occur on user-defined managed types / smart pointers.

What I assume to be true:

• owned and borrow require compiler assistance to perform static lifetime checking
• shared does not have static lifetime checking and--other than borrow support--doesn't fundamentally need compiler assistance; shared could just be a library type.

Lifetime checking on user-defined managed types could be achieved with a proposal like:

• Some kind of marker/annotation to denote this type as a user-defined managed type.
  • Are constrained generics something that could enable this? E.g., a marker/annotation on a constrained generic that user-defined managed types have to implement.
  • Is borrow support also implemented as a marker?
• This type should have runtime lifetime checking similar to shared. As a sanity check, shared could (should?) be reimplemented as a library managed type.
• This type still needs borrow support from the compiler.
• This type may need different argument intents than the default for records. However, I do not want to imply that this type should have unique argument intents, rather it may be sufficient to reuse whatever the argument intents are for owned and/or shared.
• This type needs option for nullability, with runtime-checked nullable/non-nullable support. E.g., OwnedNullable could (should?) be a library managed type.

Ideas for library-based managed types:

• Weak and SharedWeak (or simply Shared) that Weak has to be converted to before it can obtain ownership.
• Unmanaged
• Autocopy/Copy -- Make a class behave like a record. A constrained-generic argument is probably required, such as Cloneable.
• OwnedNullable

[mppf]

General musings on whether and how lifetime-checking should occur on user-defined managed types / smart pointers.

What I assume to be true:

• owned and borrow require compiler assistance to perform static lifetime checking

Not really more than any other type. The do currently use pragma "owned" but a user-defined pointer management type (e.g. Copy) could just use owned itself to get that part.

• shared does not have static lifetime checking and--other than borrow support--doesn't fundamentally need compiler assistance; shared could just be a library type.

I view owned and shared as library types now. There is some compiler support to do the following:

• parse things like owned MyType (but maybe we can support that for user "type modifiers" in the future)
• coercions to borrow (which are activated by pragma "managed pointer" today). The compiler will rely on there being a type field t (which is the borrow type) and a method .borrow() and in some cases _cast overloads for this coercion.
  • subtyping (e.g. owned ChildClass subtype of owned ParentClass) is also activated by pragma "managed pointer" and relies on type field t and some _cast overloads.
  • different argument intents from records (also activated by pragma "managed pointer" )
  • tell the borrow checker that a pointer is "owned" even though it might e.g. be marked as unmanaged or as a borrow (pragma "owned" today).

See e.g.:

https://github.com/chapel-lang/chapel/blob/a9cd48b638e5ae9a4448f3a2bf71c294280dac0f/modules/internal/SharedObject.chpl#L94-L107

Lifetime checking on user-defined managed types could be achieved with a proposal like:

I don't see that lifetime checking has that much to do with it. True, we have a pragma "owned" right now but we alternatively could have the compiler set that based on pragma "managed pointer". And then, at that point, the question is not "How can the lifetime checker support user-defined managed types" but rather "How can the compiler support user-defined managed types?". In particular getting the coercions to borrow to work seems to me to be the harder part.

• Some kind of marker/annotation to denote this type as a user-defined managed type.

Depending on your goals, the current pragma might be sufficient for this. Certainly pragmas aren't the best long-term solution.

• Are constrained generics something that could enable this? E.g., a marker/annotation on a constrained generic that user-defined managed types have to implement.

Sure, a type could "implement" a special interface instead of using the pragma.

• Is borrow support also implemented as a marker?

yes, coercions to borrow rely on a pragma "managed pointer" right now. Long term, if we don't want undocumented features involved, we'll need to complete user-defined coercions and casts (#5054).

• This type should have runtime lifetime checking similar to shared. As a sanity check, shared could (should?) be reimplemented as a library managed type.

I don't know what runtime lifetime checking is.

• This type still needs borrow support from the compiler.

Do you just mean coercions to borrow?

• This type may need different argument intents than the default for records. However, I do not want to imply that this type should have unique argument intents, rather it may be sufficient to reuse whatever the argument intents are for owned and/or shared.

Right, you can have that today with the pragma.

• This type needs option for nullability, with runtime-checked nullable/non-nullable support. E.g., OwnedNullable could (should?) be a library managed type.

Right now there isn't any null-checking at compile time. It seems to me we need (at least an outline) of nil-checking is approached by the language, including some compile-time component, before we design how managed types interact with compile-time nil checking. I view resolving issue #9500 as the starting point there.

Ideas for library-based managed types:

• Weak and SharedWeak (or simply Shared) that Weak has to be converted to before it can obtain ownership.

How is Weak different from a borrow?

• Unmanaged

How is this different from unmanaged ?

• Autocopy/Copy -- Make a class behave like a record. A constrained-generic argument is probably required, such as Cloneable.

These ones seem interesting to me & reasonable to have in the library. Seems that there's only one type here though, right? The variants are just different potential names?

• OwnedNullable

This one makes sense only after there is compile-time checking for nil for owned, so it doesn't make sense to me to design until we resolve issue #9500.

[BryantLam]

I view owned and shared as library types now. There is some compiler support to do the following:

It sounds like pragma "managed pointer" does most of what I proposed it should do, which is great! The only missing item is the compile-time null-checking after #9500.

tell the borrow checker that a pointer is "owned" even though it might e.g. be marked as unmanaged or as a borrow (pragma "owned" today).

What does pragma "owned" do? Is it strictly for the borrow checker?

the question is not "How can the lifetime checker support user-defined managed types" but rather "How can the compiler support user-defined managed types?".

👍

I don't know what runtime lifetime checking is.

I consider lifetime checking under two "scopes":

• compile-time checks
  • use after free / out of scope #10365
  • use after move #10835
  • null checks where appropriate #9500 (uncommon case)
• runtime checks
  • use after free/move (e.g., trying to dereference a freed shared; it's not possible to compile-time check shared in the general case)
  • null checks where appropriate (common case)

This type still needs borrow support from the compiler.

Do you just mean coercions to borrow?

Yes. Sorry about that. It sounds like either pragma "managed pointer" or pragma "owned" does what needs to be done if a user-defined managed type was decorated appropriately (subject to usability improvements).

Ideas for library-based managed types:

This list was not meant as something Chapel should provide (while not the intent, Chapel could provide/all some of them). The list was meant as examples that users should be able to write themselves with compiler support. It sounds like Chapel is already headed that way, so this list is less important now.

That said ...

How is Weak different from a borrow?

An analogy from C++ and Rust, a user is not allowed to dereference a Weak pointer without promoting it to shared or some Shared-like type. The shared object is allowed to delete the object despite Weak pointers to it, so Weak needs to check to see if the underlying object is deleted before it can be upgraded to a shared (or equivalent). It's typically used to break circular ownership in a graph or tree.

In this case, I don't think any compiler support is relevant for Weak, because you can't really do anything with it unless it is upgraded to shared. You should not be allowed to coerce to borrow with Weak, so it's a bad example of a user-defined managed type that needs compiler support.

How is this different from unmanaged ?

A "user-defined Unmanaged' would require coercion to borrows, but you already covered that there are pragmas for that.

This one makes sense only after there is compile-time checking for nil for owned, so it doesn't make sense to me to design until we resolve issue #9500.

Fair enough. This issue was meant more to make the point that compile-time+runtime null checking is another thing that would be needed from the compiler on a user-defined managed type.

[bradcray]

Just to make sure we're all on the same page (I think Michael's notes implied this, but subtly), while pragmas are our mechanism for enabling some of this special smart-pointer behavior today, I think we'll want a more first-class way of doing so within the language that doesn't rely on pragmas once end-users are writing their own smart pointer types (that's not to say that end-users couldn't use pragmas in the meantime, just that I think they do so at their own risk until they're replaced by more of a language-level feature).