Add support for explicit backing fields

[lunakoly]

Please, see explicit-backing-fields.md for the full text in PR #289.

Summary

Sometimes, Kotlin programmers need to declare two properties which are conceptually the same, but one is part of a public API and another is an implementation detail. This is known as backing properties:

class C {
    private val _elementList = mutableListOf<Element>()

    val elementList: List<Element>
        get() = _elementList
}

With the proposed syntax in mind, the above code snippet could be rewritten as follows:

class C {
    val elementList: List<Element>
        field = mutableListOf()
}

[BreimerR]

Interesting read. Question: What if the backing field is a var and the actual field is a val.

private class SynchronizedLazyImpl<out T>(initializer: () -> T, lock: Any? = null) : Lazy<T>, Serializable {
    private var initializer: (() -> T)? = initializer
    @Volatile private var _value: Any? = UNINITIALIZED_VALUE
    // final field is required to enable safe publication of constructed instance
    private val lock = lock ?: this

    override val value: T
    /// Other code is here Referenced from kotlin.SynchronizedLazyImpl
}

How would this be catered for?

[LouisCAD]

I have one concern: It looks like it'll not be possible to use asStateFlow(), and other read-only type conversion methods, unless you want to pay the price of that operation (and memory allocation) being run on each access.

Has the possibility of allowing it been considered, despite the fact that it'd require more than one actual backing field?

[lunakoly]

@BreimerR,

As of now, we assume val properties have immutable backing fields, and var properties have mutable ones.

Thanks for your example, I've added it to the enhancements section.

[lunakoly]

@LouisCAD,

If you call asStateFlow() inside a getter, this call will happen upon each access, true. If you do something like this:

val flow: StateFlow<T>
    field = createSomeMutableStateFlow()
    get() = field.asStateFlow()

Then the smart type narrowing won't work, since it only knows how to deal with the default getters.

[gildor]

It looks like it'll not be possible to use asStateFlow(), and other read-only type conversion methods, unless you want to pay the price of that operation (and memory allocation) being run on each access.

I think it's not a big deal in terms of price, it's just a small object without any actual state, only with 1 reference

[LouisCAD]

How doable would it be to address this use case in the design of this feature, or at least, make some room for it for future evolution?

[pdvrieze]

@BreimerR,

As of now, we assume val properties have immutable backing fields, and var properties have mutable ones.

Thanks for your example, I've added it to the enhancements section.

At least on the jvm that is not valid. "final" and "immutable" are different concepts although they are mostly equivalent. More important is that from a consumer perspective (rather than class implementor/compiler) the assumption cannot be made as the backing field (or lack thereof) cannot be assumed - changing it is API and ABI compatible.

[mcpiroman]

I'm much more after the original proposal, because (from my understanding):

• From a private (/more restricted, like internal) scope a property is essentially accessed directly through a field. This breaks the current encapsulation of backing fields being local to properties and disables the use of custom getter from the private scope.
• In future, the custom getter could be analyzed to work with the Smart Type Narrowing (STN), but it would still need to return the same instance, which would not work for the case like the one mentioned above:

  val flow: StateFlow<T>
  field = createSomeMutableStateFlow()
  get() = field.asStateFlow()

• This proposal does not work with delegates, the original does.
• The original would also work very well with setter overloading as mentioned here

You write there are complications with the original proposal - well, my answer to both linked questions are yes and yes, but this could be considered incrementally, just as here you start only with private and internal fields. You also mention there were problems with overloading, but here in rules you say that the "property must be final", so it's essentially the same in both cases?

So I'd like to iterate on why the former proposal cannot be implemented, I don't see the why now.

[lunakoly]

@LouisCAD,

How doable would it be to address this use case in the design of this feature, or at least, make some room for it for future evolution?

Hypothetically, it's possible to introduce a syntax like flow#field if we really need the direct access. Wouldn't it solve the problem?

[lunakoly]

@pdvrieze,

At least on the jvm that is not valid. "final" and "immutable" are different concepts although they are mostly equivalent. More important is that from a consumer perspective (rather than class implementor/compiler) the assumption cannot be made as the backing field (or lack thereof) cannot be assumed - changing it is API and ABI compatible.

Thanks for your correction. I simply meant that assignment to field is forbidden inside a val property getter:

val test: Int = 1
    get() {
        field = 2
        return field
    }

And right now it works the same for properties with an explicit backing field as well

[lunakoly]

@pdvrieze,

It seems to me, sometimes consumers need more control over the stored value, which brings in backing properties. Feels like introducing a field declaration which guarantees the presence of a backing field is what we need here, isn't it?

[lunakoly]

@mcpiroman,

This breaks the current encapsulation of backing fields being local to properties and disables the use of custom getter from the private scope.

I suppose, analyzing getter return value could still make them usable without loosing the STN.

It's true that the STN may not be able to solve all the problems, but it still solves the base problem that gave birth to this discussion.

Please, keep in mind that "Delegated property cannot have accessors with non-default implementations". The original proposal doesn't mention anything about delegated properties.

From what I see, there's nothing preventing the support for setters overloading in the current proposal. I think it's a bit different topic, not "a direct enhancement".

[lunakoly]

@mcpiroman,

The biggest reason why personally I prefer the new approach is the fact that it's really easy to understand. It builds around the already known idea - a backing field - which is just an implementation detail, and this information may be further used on the implementation side to narrow down the type.

The original proposal doesn't mention how exactly the new getters (the ones with a more permissive visibility) would work. Would they work via the same mechanism as the STN? Then the explicit backing fields syntax seems less cryptic to me. Would they not get called when accessing from the scope relevant to the visibility of the property itself so that we have direct access to the backing field? Then we break the rule "getters are always called". How would they interact with usual getters? Prioritization rules of mixed getters seem unintuitive.

The concept of new "exposing" getters introduces such questions as "can we both override the property and its exposing getter" (while implementing the original proposal, the ability to perform such overrides was added). The use with delegates was also unclear.

[mcpiroman]

@lunakoly

It's true that the STN may not be able to solve all the problems, but it still solves the base problem that gave birth to this discussion.

Mostly yes, but it's better to solve more in one step than less, if only because it should end up being less complicated.

The original proposal doesn't mention anything about delegated properties.

Yes, it also doesn't mention their not supported :). And, it looks to me like they should easily be possible. Right now you can do:

private val _foo by lazy { mutableListOf("cOmPLeX") }
val foo: List<String> get() = _foo

So it could become:

val foo by lazy { mutableListOf("cOmPLeX") }
   public get(): List<String>

Which would not be possible with explicit fields.

From what I see, there's nothing preventing the support for setters overloading in the current proposal. I think it's a bit different topic, not "a direct enhancement".

Yes, I mean that it would be nicely symmetric to have multiple getters and multiple setters.

The biggest reason why personally I prefer the new approach is the fact that it's really easy to understand

So for me at least, this is the other way around. I knew at a glimpse what the original syntax was about (maybe because I already knew of the proposal), and I actually pondered quite a bit on what's going on with the new one:

• Why do I have to mess with a backing field to change the (private) type of a property - two different concepts? What if there's no backing field (actually wait, what if there's no backing field? no delegates, ok, but an accessor?),
• Does accessing the property from private scope now go though the getter or directly though the field? After all the filed has the private type, not accessor.
• Default private for field? Public was always the default.

[The rest of your comment]

So to clarify: my view is that there is ever only one actual getter with body, the other ones are just declarations that say more about it's type (and therefore probably should not have parentheses as proposed), the same way as private get does. I also assume the more private type has to be more narrow. With that I frankly can't imagine any of your concerns.

However, I do see an unmentioned one:

val foo: List<String>
   private get: MutableList<String> // <- Property is visible outside under type List<MutableList>

val bar: List<String>
   private get  // <- Property is not visible outside

These too very closely looking syntaxes have quite a different meaning and so are not intuitive. This is not much of a problem right now because Getter visibility must be the same as property visibility so there is no point in using the latter construct, but may be once this restriction is dropped - most probably to support more visible setters as proposed somewhere.

It's not to say it is perfect either, you may e.g. want to change the order of visibility modifiers so it becomes:

private val foo: MutableList<String>
   public get: List<String>

Which would read as "There is a private property foo of type MutableList<String>, which is also exposed publicly under the type List<String>".

[mglukhikh]

I'd like to add my 5 cents.

... why the former proposal cannot be implemented

IMO it's not a question of "cannot", of course it can be implemented. It's a question of how clear the result is. Let's take even the basic example

private val foo: MutableList<String> = ...
    public get(): List<String>

Let's try to answer: MutableList<String> is a type of what here? Is it a type of the property? Well, this sounds strange because property is no more than its getter (and setter if it's mutable), and getter type is List<String> here. But in new syntax we have

val foo: List<String>
    (private) field: MutableList<String> = ...

and now it's quite clear that MutableList<String> is the backing field type and List<String> is the corresponding property type. Moreover it's clear that the backing field is private and the corresponding property is public.

With possible overrides, situation is even more interesting:

interface Base {
    val foo: MutableList<String>
}
class Derived : Base {
    // I'd say that 'override private val' is nightmarish for itself
    override private val foo: MutableList<String> = ...
        public get(): List<String>
}

Note that code above isn't valid despite of the fact it's Ok for first glance (type of Base.foo and Derived.foo is the same). However, in fact Base.foo should change its type to List<String> for this code to be valid (or public get should be removed for Derived.foo). Why so? Because in this syntax Derived.foo is in fact not a property but only its backing field. With the field-based syntax, things look much clearer here:

interface Base {
    val foo: List<String>
}
class Derived : Base {
    (final) override val foo: List<String> = 
        (private) field: MutableList<String> = ...
}

[mglukhikh]

About delegates, I'd not say that the code like

val foo by lazy { mutableListOf("cOmPLeX") }
   public get(): List<String>

cannot be written in new syntax at all. Well, it's true that it's not supported currently but in future we could have something like (just a hypothetical syntax)

val foo: List<String>
    (private) field by lazy { mutableListOf("cOmPLeX") }

or, may be (longer but a bit clearer)

val foo: List<String> by field
    (private) field = lazy { mutableListOf("cOmPLeX") }

[mcpiroman]

private val foo: MutableList<String> = ...
public get(): List<String>

Let's try to answer: MutableList is a type of what here?

The type is List from public scope and MutableList from private scope. Note the above is kind of a shorthand for:

val foo = ...
   public get: List<String>
   private get: MutableList<String>

This may look cleaner (also I use just get, it is a better option imo). What can be a real problem though is that I think the team wanted to avoid having context-sensitive/multiple types - which is the case here - while for me this is unfortunately the way to go. Especially that it is probably also required for the mentioned setter overloading.

val foo: List<String>
    (private) field: MutableList<String> = ...

and now it's quite clear that MutableList<String> is the backing field type and List<String> is the corresponding property type. Moreover it's clear that the backing field is private and the corresponding property is public.

This sounds clear - you have a MutableList property and a private List field. So when you do foo.add(1) from within a class, you access the field, the one with a type of MutableList, right? No, you access the property though the getter having a type of List, but just smart narrowing the type. That's what's confusing - you declare a field with some access, as though exposing it privately, but you don't use the field anymore, you use the property (the getter) but with modifying it's type. But maybe that's just me or maybe this can be corrected within the proposal.

Now to the inheritance:

Your first example of course wouldn't work - it wouldn't even now, you can't narrow down the type of overloaded property. So yes, Base.foo should change its type to List<String>, otherwise there's no sense declaring it this way. So now it becomes a fair comparation and my take looks like:

interface Base {
    val foo: List<String>
}
class Derived : Base {
    override private val foo: MutableList<String> = ...
        public get: List<String>

   // But it could be also written as:
    override public val foo: List<String> = ...
        private get: MutableList<String>

  // Or maybe like that? It should be decided which ways are allowed and which are not to make it the most clean.
    override val foo = ...
        public get: List<String> 
        private get: MutableList<String>
}

The rule is simple, the interface only declares a public type and the public type of overridee cannot be narrower than it - exactly like presently. Then the private type cannot be narrower than the public one; this then expands to other visibilities.

Because in this syntax Derived.foo is in fact not a property but only its backing field.

No, it's still a property. At least at the backend there is a getter method override ArrayList<String> foo() { ... }. Or you're talking more abstractly, then sorry, I don't get it.

About delegates:

val foo: List<String>
    (private) field by lazy { mutableListOf("cOmPLeX") }

Probably won't work - field cannot be implemented by delegate, only property can.

val foo: List<String> by field
    (private) field = lazy { mutableListOf("cOmPLeX") }

The type of field then becomes Lazy<MutableList<String>>, it would need to be accessed as foo.getValue().add(1).

But even if one of these, or some new syntax would work, the delegate instance would still need to be in the backing field, probably yes, this is often the case, but not always. And then it still doesn't support properties without a backing field.

[elizarov]

UPDATE: There has been a massive update to the proposal text in PR #289 that addresses many of the issues raised here. Please, see explicit-backing-fields.md for the full text:

• Use-cases are orthogonalized. Similar use-cases are grouped together.
• Additional use-cases are explained to cover all the major pieces of the proposed design, including the access to the property delegate.
• Design is worked out in more detail, with grammar and various restrictions more explicitly spelled out.
• This list of future enhancements is narrowed down to the ones that are feasible in the near future.

[mcpiroman]

I happened to stumble across this branch which seems to implement a syntax for Explicit Backing Filed Access in a form of:

var number: String
    field: Int = 0
    get() = field.toString()
    set(value) { field = value.toInt() }

fun updateNumber() {
    number#field += 100
}

This reminded me I was about to suggest mine, which is to, instead of adding language syntax, add a field property to KProperty:

fun updateNumber() {
    ::number.field += 100
}

• KProperty also needs to have a type parameter for this property, representing type of the field
  • It is Nothing if the property does not have backing field
  • It is Nothing if KProperty is obtained outside of the visibility scope of backing filed, e.g. foo::number
• For this, it would be helpful to have default type arguments to help with working with KProperty
  • Alternatively, there could be new KBackedProperty subtypes
• Properties' delegates could be made accessible in similar way, i.e. delegate as a typesafe replacement for getDelegate()

[lunakoly]

Hello, @mcpiroman!

The direct field access syntax is still TBD, and this branch is a sandbox where I'm looking for possible problems we may face if we decide to implement the feature in the form of property#field. The branch isn't currently ready for review, and alternative design ideas are, of course, welcome.

The idea of having an additional type parameter for KProperty sounds interesting, but as I understand, Reflection is a mechanism to "introspect the structure of the program at runtime", and the direct backing field access is not.

[mcpiroman]

Yeah, I just used the syntax from the branch to compare it to KProperty approach.

I see the KProperty as a low key reflection, more of a helper. It already has get, set, getDelegate methods which do not inspect the structure of program and I think field property, which is in itself kind of ad-hoc/low-level rather fits there.

The problem though would be in such a case:

class Foo {
   val number: String
    field: Int = 0
    get() = field.toString()

  fun exposeNumberProp() = ::number
}

fun main() {
   val n = Foo().exposeNumberProp().field
|

the backing field is meant to be class-private in ABI (at least on JVM), so it wouldn't work.

So either the field would need to be public (but inaccessible for java) or some additional tracking rules would need to be applied - quite possibly a simple one, but still.

[shaun-wild]

val names: MutableList<String>= mutableListOf()
  get(): List<String>

I like this syntax, it makes more sense to me at least, developers need only set a return type on the getters using an already familiar syntax.

[elizarov]

I like this syntax, it makes more sense to me at least, developers need only set a return type on the getters using an already familiar syntax.

val names: MutableList<String>= mutableListOf()
  get(): List<String>

That was the syntax in the original proposal. We've prototyped it and ran into multiple problems with open properties, with overrides, with the overall mental model on "what is the type of the property", and with related readability. That's why we changed it to the more explicit and more explicit syntax that we have in the proposal now:

val names: List<String>                            // clearly tells the type of the property upfront
   field: MutableList<String> = mutableListOf()    // the field type is just an implementation detail

[Zhuinden]

According to the issue https://youtrack.jetbrains.com/issue/KT-14663 this feature has a target version of 1.7.0 and is marked as fixed, but based on the list on https://kotlinlang.org/docs/whatsnew17.html#top it's not available in 1.7.0 yet?

Is there any public information available on what version of Kotlin this might hopefully be featured in?

[Zhuinden]

I've been told that

1.) it requires explicit enabling of K2 compiler

2.) it requires explicit opt-in as experimental feature

However, it can be made to work

https://gist.github.com/dellisd/a1e2ae1a7e6b61590bef4b2542a555a0

[elizarov]

Prototype of this proposal has been delivered in Kotlin 1.7.0 as a part of K2 compiler preview. In order to try out his new feature you need to enable the K2 compiler with -Xuse-k2 command line option and enable this language feature with -XXLanguage:+ExplicitBackingFields. The implementation is not stable yet and will generate pre-release binary. There is no IDE support yet.

[gildor]

@thumannw It's not only shorter, but the biggest advantage is naming (_no _more _this) and improved code locality. It's not groundbreaking but it was highly requested because it's one of things which now looks awkward and widely used

And one can argue that setter only property workaround is to use method instead of property

[MartinHaeusler]

Hello from kotlin conf 2024! This matter was just discussed in a talk by Michail.

I was wondering if this proposal also involves setter overloading:

private var user: String
    set(user: String) { field = user)
    set(user: User) { field = user.username }

This would be very useful for DSL building. Also, another use case would be a private field which needs to be nullable, but the setter should not accept null as an input. In Java, setters are just normal methods and therefore overloading works out of the box. It would be really nice to have the same experience with kotlin properties.

[Luke830]

https://youtu.be/tAGJ5zJXJ7w?t=1860 (KotlinConf'24) In this video, we introduce it as a feature coming in kotlin 2.2.

[merfemor]

I was wondering if this proposal also involves setter overloading

Hi @MartinHaeusler. No, this is actually a different feature. We're aware of the use-case with DSL, but there are currently no plans in this regard. Here is an issue to follow for further updates: KT-4075.