proposal: reflect: add DeepCopy

[changkun]

Previous proposal updates

As a duality complement of `DeepEqual`, I propose to add a new API for deep copy: ```go package reflect // DeepCopy copies src to dst recursively. // // Two values of identical type are deeply copied if one of the following cases applies. // // Array and slices values are deeply copied, including its elements. // // Struct values are deeply copied for all fields, including exported and unexported. // // Interface values are deeply copied if the underlying type can be deeply copied. // // Map values are deeply copied for all of its key and corresponding values. // // Pointer values are deeply copied for its pointed value. // // One exception is Func value. It is not copiable, and still points to the same function. // // Other values - numbers, bools, strings, and channels - are deeply copied and // have different underlying memory address. func DeepCopy[T any](dst, src T) // or // // DeepCopy panics if src and dst have different types. func DeepCopy(dst, src any) // or // // DeepCopy returns an error if src and dst have different types. func DeepCopy(dst, src any) error ``` DeepCopy may likely be very commonly used. Implement it in reflect package may receive better runtime performance. The frist version seems more preferrable because type parameters permits compile time type checking, but the others might also be optimal. The proposed document of the API is preliminary. --- Update: Based on the discussions until https://github.com/golang/go/issues/51520#issuecomment-1061506887, the documented behavior could be: ```go // DeepCopy copies src to dst recursively. // // Two values of identical type are deeply copied if one of the following // cases apply. // // Array and slices values are deeply copied, including its elements. // // Struct values are deeply copied for all fields, including exported // and unexported. // // Map values are deeply copied for all of its key and corresponding // values. // // Pointer values are deeply copied for their pointed value, and the // pointer points to the deeply copied value. // // Numbers, bools, strings are deeply copied and have different underlying // memory address. // // Interface values are deeply copied if the underlying type can be // deeply copied. // // There are a few exceptions that may result in a deeply copied value not // deeply equal (asserted by DeepEqual(dst, src)) to the source value: // // 1) Func values are still refer to the same function // 2) Chan values are replaced by newly created channels // 3) One-way Chan values (receive or read-only) values are still refer // to the same channel // // Note that for stateful copied values, such as the lock status in // sync.Mutex, or underlying file descriptors in os.File and net.Conn, // are retained but may result in unexpected consequences in follow-up // usage, the caller should clear these values depending on usage context. // // The function panics/returns with an error if // // 1) source and destination values have different types // 2) destination value is reachable from source value ``` Depending on the design choice, the content included and after exceptions could decide on different behaviors, see https://github.com/golang/go/issues/51520#issuecomment-1061453900, and https://github.com/golang/go/issues/51520#issuecomment-1061506887 as examples. The API signature could select one of the following possible options. The main differences between them include two aspects: 1) panic (break the code flow, and enter defer recover) or return an error (user handled directly) 2) destination value in either parameter (more GC friendly) or return value (always allocates) ```go func DeepCopy[T any](dst, src T) error func DeepCopy[T any](dst, src T) func DeepCopy(dst, src any) error func DeepCopy(dst, src any) func DeepCopy[T any](src T) (T, error) func DeepCopy[T any](src T) T func DeepCopy(src any) (any, error) func DeepCopy(src any) error ``` Either version is optimal, and purely depending on the final choice.

Update (until https://github.com/golang/go/issues/51520#issuecomment-1080602090):

I propose to add a new API for deep copy:

// DeepCopy copies src to dst recursively.
//
// Two values of identical type are deeply copied if one of the following
// cases apply.
//
// Numbers, bools, strings are deeply copied and have different underlying
// memory address.
//
// Slice and Array values are deeply copied, including its elements.
//
// Map values are deeply copied for all of its key and corresponding
// values.
//
// Pointer values are deeply copied for their pointed value, and the
// pointer points to the deeply copied value.
//
// Struct values are deeply copied for all fields, including exported
// and unexported.
//
// Interface values are deeply copied if the underlying type can be
// deeply copied.
//
// There are a few exceptions that may result in a deeply copied value not
// deeply equal (asserted by DeepEqual(dst, src)) to the source value:
//
// 1) Func values are still refer to the same function
// 2) Chan values are replaced by newly created channels
// 3) One-way Chan values (receive or read-only) values are still refer
//    to the same channel
//
// Note that while correct uses of DeepCopy do exist, they are not rare.
// The use of DeepCopy often indicates the copying object does not contain
// a singleton or is never meant to be copied, such as sync.Mutex, os.File,
// net.Conn, js.Value, etc. In these cases, the copied value retains the
// memory representations of the source value but may result in unexpected
// consequences in follow-up usage, the caller should clear these values
// depending on their usage context.
func DeepCopy[T any](src T) (dst T)

Here is an implementation for trying out: https://pkg.go.dev/golang.design/x/reflect

[atdiar]

Ok, thanks. So I guess that in general, one should not expect deepcopied values to be deepEqual. It's not just functions, but any struct, map, slice that holds a function for instance.

Do you have examples where it would be useful? Trying to think about it, maybe copying a request object or some kind of copy-on-write... what else?

What are the potential issues when using such a function? Hidden side-effectfulness when for example, copying a mutex or a FD?

Is deep-copying always semantically relevant? (for instance, an object that relies on shared mutable state... deep copying such state might not make sense)

For example, I was wondering what it would mean to copy a js.Value that was refering to a DOM Element. How to signal that another reference has been created? What's with the *gcPtr ? Leading to the question of what if an object creation was side-effectful. (for instance, a count of created objects is kept somewhere). Then deep-copying would be an error. One might argue that this is a programmer oversight to deep-copy.

So back to the use-cases, when is it fitting to use deep-copy? Can some kind of rule be established to make it safe?

[ianlancetaylor]

If this is the argument then there are too many examples in the language against this argumentation. The recent generics is a big one of them. All of us have no experience and do not fully understand if a late decision was the right choice.

I think I've said what want to say on this issue and I don't think further repetition will get us anywhere. I'll just reply to this one comment.

Yes, generics is an unusual case. It is a language change, so we did not have the option of trying a third party package. Instead, we spent several years giving talks about it and explaining what it would look like. We built an alternate toolchain (https://go.dev/blog/generics-next-step) and encouraged people to use that to see if generics would work as they expected and to see whether it met their needs. We gathered a lot of input from a lot of people (thanks everyone!) and used that to make decisions about the design. The result is undoubtedly imperfect, but I think we did everything we could to try out possibilities before we committed to a specific design.

[rittneje]

So back to the use-cases, when is it fitting to use deep-copy? Can some kind of rule be established to make it safe?

This I think strikes at the heart of the issue with this proposal. In order for me to safely use DeepCopy on an object, I have to examine all of its fields (recursively!) to determine whether the operation will be safe. For example, as previously mentioned, an *os.File probably cannot be (naively) copied. This issue is exacerbated by the desire to also deeply copy all unexported fields. Worse, this means that adding fields to a struct can become a breaking change, if anyone was using DeepCopy on it (directly or indirectly), if those new fields cannot be naively copied.

I still very much believe that adding a method to the type in question is a much safer and clearer way of supporting a deep copy operation in general.

[changkun]

Ok, thanks. So I guess that in general, one should not expect deepcopied values to be deepEqual. It's not just functions, but any struct, map, slice that holds a function for instance.

I am not sure why it was concluded "in general". In general, regular struct, maps, slices don't hold functions.

Do you have examples where it would be useful? Trying to think about it, maybe copying a request object or some kind of copy-on-write... what else?

There are discussed example in this thread. If onr need more examples why people need it, check this https://pkg.go.dev/search?limit=25&m=symbol&q=deepcopy.

A summarize of use cases (to prevent being asked once again) from the top 25 listed results from https://pkg.go.dev/search?limit=25&m=symbol&q=deepcopy:

1. Deep copy struct struct messages, such as protobuf, json, config etc.
   • https://github.com/tendermint/go-amino
   • https://github.com/yunionio/jsonutils/blob/f6733e38783e9dab6a7edbc42aeafb0fb07c467e/clone.go
   • https://github.com/containerd/cri/blob/aa2d5a97cdc4ef93919fb7d243213ce33b089aa2/pkg/util/deep_copy.go
   • https://github.com/tendermint/go-amino-x/blob/4e1c2066fdb2fa5a4e20a4d7d6d22dfa9aee3759/deep_copy.go
   • https://github.com/infinitbyte/framework/blob/678c32f7526fc5e343b6302ebb8f4a9591fb724f/core/util/bytes.go
   • https://github.com/Azure/aks-engine/blob/47c439d1e8891a1cc63ced39332418f1506e6a41/pkg/armhelpers/converter.go
   • https://github.com/andersfylling/disgord/blob/b19378bcf043df6d0105c2412eebed840b00ac9d/disgord.go
   • https://github.com/ghetzel/go-stockutil/blob/7dc4179186c4f3015b02a5b4ab1de58d041e2736/maputil/maputil.go
   • https://github.com/TIBCOSoftware/labs-lightcrane-contrib/blob/main/common/util/helper.go
   • https://github.com/yunionio/pkg/blob/28690b33ce38/gotypes/deepcopy.go#L34
   • https://github.com/restream/reindexer/blob/v4.2.1/reindexer.go#L112
   • https://github.com/medcl/gopa-abandoned/blob/6b52c20532a7/core/util/bytes.go#L56
   • https://github.com/PerformLine/go-stockutil/blob/v1.9.3/maputil/maputil.go#L997
   • https://github.com/johnsiilver/boutique/blob/v0.1.1-beta.1/boutique.go#L738
2. Deep copy values can be gob.NewEncoder gob.Decoder.
   • https://github.com/openconfig/ygot/blob/b05ad39f60db4043c948463601c7e29304ceadb0/ygot/struct_validation_map.go#L600https://github.com/lxc/lxd/blob/1aeed330441dd83a879a999f215f79540967b5a2/shared/util.go
   • https://github.com/evergreen-ci/evergreen/blob/e29a626d90903b9f65f7e87fd8a7db7526f986f8/util/copy.go
   • https://github.com/xitongsys/parquet-go/blob/6cb95b788db8bb9a5110e28fbefe251afd7454c0/common/common.go
   • https://github.com/haiyiyun/utils/blob/3f7aeeafa0fe/help/reflect.go#L28
   • https://github.com/godaddy-x/jorm/blob/v1.0.65/util/common.go#L420
   • https://github.com/Peakchen/xgameCommon/blob/v0.0.5/utls/cvr.go#L93
3. Deep copy structures such as linked list, trees, graphs, tensor.
   • https://github.com/goki/ki/blob/685b18f4225ea4965ff53497f57b199dc4c6b0ff/ki/ki.go
   • https://github.com/polyred/polyred/blob/5323a964a6504b22b49305d69ae05c2d9987ba96/geometry/primitive/vertex.go#L104
   • https://github.com/tensorflow/tensorflow/blob/master/tensorflow/go/op/wrappers.go

What are the potential issues when using such a function? Hidden side-effectfulness when for example, copying a mutex or a FD? Is deep-copying always semantically relevant? (for instance, an object that relies on shared mutable state... deep copying such state might not make sense)

This was discussed previously.

For example, I was wondering what it would mean to copy a js.Value that was refering to a DOM Element. How to signal that another reference has been created? What's with the *gcPtr ? Leading to the question of what if an object creation was side-effectful. (for instance, a count of created objects is kept somewhere). Then deep-copying would be an error. One might argue that this is a programmer oversight to deep-copy.

Yes, js.Value is not a use case for deep copy. But why would one try to copy a js.Value?

So back to the use-cases, when is it fitting to use deep-copy?

I hope the document explains when it is fitting. If not, we could improve the language.

Can some kind of rule be established to make it safe?

It's unclear what is the definition of safe here. Could the current reflect always prevent panicking from calling a nil?

[changkun]

In order for me to safely use DeepCopy on an object, I have to examine all of its fields (recursively!) to determine whether the operation will be safe. For example, as previously mentioned, an *os.File probably cannot be (naively) copied. This issue is exacerbated by the desire to also deeply copy all unexported fields. Worse, this means that adding fields to a struct can become a breaking change, if anyone was using DeepCopy on it (directly or indirectly), if those new fields cannot be naively copied.

I doubt this argument. When you desired to copy a value, how uncertain regarding the details of the value? Say, we created a file, and a pointer points to it. Then for some crazy reason, we want to copy this file pointer, what would we expect out of this deep copy? Another pointer to read the file? A duplicated file? Can't we simply assign the pointer to a different value? Why would we have an idea to deep copy a file pointer?

[rittneje]

@changkun I don't understand your question. If an *os.File gets deep copied naively (meaning that all struct fields are deep copied recursively), you will end up with two logical files that are using the exact same file descriptor. This is extremely dangerous, because now the file descriptor can be closed through one logical file, while it is still being used through the copy.

Imagine I have a library that exposes a struct like this:

type Foo struct {
    privateField string
}

Your code might then leverage your proposed reflect.DeepCopy to make deep copies of Foos. And at first it will work. But now in a newer version of my library, I want to add an *os.File to it for one reason or another.

type Foo struct {
    privateField string
    someFile *os.File
}

Now your code is dangerously broken for the aforementioned reason.

If however I had a Copy (or Clone or whatever) method, then there is an actual contract, and the burden is on me as the library author to maintain it.

func (f *Foo) Copy() *Foo {
    return &Foo{
        privateField: f.privateField,
        // this method doesn't exist today, but you get the idea
        someFile: f.someFile.Copy(),
    }
}

Please note that *os.File is only meant to illustrate a specific example of where your proposed DeepCopy is a problem. The general issue is that you are trying to violate the principle of abstraction.

[changkun]

Sure in this case it is not safe to use after deep copy. As warned in the doc:

// Note that for stateful copied values, such as the lock status in
// sync.Mutex, or underlying file descriptors in os.File and net.Conn,
// are retained but may result in unexpected consequences in follow-up
// usage, the caller should clear these values depending on usage context.

But let's take one step back: I don't fully understand the motivation of your described use case too. How frequent for a struct will hold an unexported file? Or Is this really a good practice? When the file will be closed? Automatically using a finalizer? Ask the user to remember to call an exported Close method for closing the file? Either way seems to have a danger of not closing the file. If a package offers a customized copy method, should not using DeepCopy be considered as misuse in this package?

[rittneje]

Your doc comment shifts the burden to the wrong place. The whole point under discussion here is that some types cannot be naively copied, and a DeepCopy implementation should not copy them like that in the first place. Asking the client to do it isn't really a solution, especially because it may be some private field they don't have access to.

How frequent for a struct will hold an unexported file?

I obviously cannot give a concrete answer here, but it is entirely within a library author's jurisdiction to introduce such changes. And keep in mind that just because the struct has a reference to the file, does not mean that it is what created it. As a contrived example:

func NewFooWithFile(f *os.File) *Foo {
    return &Foo{someFile: f}
}

And as I mentioned, *os.File was just one example where this doesn't work. Mutexes are also a problem, and they are going to come up for any struct that requires thread-safety. But then you will have to hold the lock while doing the deep copy, which means that the copied struct will have its mutex locked. Then you have to somehow unlock the copy, which may not be feasible depending on the API of the library.

If a package offers a customized copy method, should not using DeepCopy be considered as misuse in this package?

Again, I don't see any purpose for DeepCopy to exist. It really just seems like a way to avoid writing a proper Copy method, but only for specific cases where the struct in question is a DTO. However, there is no way for a library author to actually agree to this contract. And without that, you are definitely in the realm of relying on implementation details of your library.

[changkun]

The whole point under discussion here is that some types cannot be naively copied

Of course, the discussion was about singleton that should not be naively copied. The counter argumentation was if this is the case, why should not DeepCopy be avoided in this case, and considered as a misuse of this API? I think your argument is about "as long as an API can't design in a way that its user can't blindly throw arguments in and get an intuitive response out, we shouldn't have it." There are enough examples against this, and we have discussed them before.

Again, I don't see any purpose for DeepCopy to exist. It really just seems like a way to avoid writing a proper Copy method, but only for specific cases where the struct in question is a DTO. However, there is no way for a library author to actually agree to this contract. And without that, you are definitely in the realm of relying on implementation details of your library.

Furthermore, even we have the concerning danger. One can still use a DeepCopy to save a lot of copying work without introducing any danger when offering customized Copy:

func (f *Foo) Copy() *Foo {
    var nf *Foo
    reflect.DeepCopy(nf, f)         // save a lot of work on dealing with field assignments
    nf.someFile = f.someFile.Copy() // clear this danger for the API user
    return nf
}

This is particularly helpful when Foo is a linked list, tree, graph, etc.

[rittneje]

What happens if Foo is itself part of another struct, say Bar?

type Bar struct {
  f *Foo
  ...
}

What does reflect.DeepCopy(*Bar) do here? Something needs to call f.Copy(). But earlier you felt that DeepCopy should not support custom behavior. So Foo has "tainted" Bar, meaning we now need something like:

func (b *Bar) Copy() *Bar {
    b2 := reflect.DeepCopy(b)
    b2.f = b2.f.Copy()
}

It very much seems like if reflect.DeepCopy is going to exist, it needs to respect the Copy method (or whatever we call it) somehow. Otherwise, you are going to have breaking changes. For instance, clients previously would have done reflect.DeepCopy(f). But now they'd have to switch to f.Copy, or their code is broken. However, if DeepCopy called the Copy method then it would just work.

I also still believe that if reflect.DeepEqual is going to exist it should ignore private fields by default, like json.Marshal does. If they should be copied along, you should implement a proper Copy method. But, as a convenience for simple cases, perhaps we can use struct tags to opt them in?

type Foo struct {
    someField string `deepcopy:"true"`
    someFile *os.File
}

func (f *Foo) Copy() *Foo {
   type foo Foo
   f2 := reflect.DeepCopy((*foo)(f))
   f2.someFile = f.someFile.Copy()
   return (*Foo)(f2)
}

[changkun]

[...] meaning we now need something like:

func (b *Bar) Copy() *Bar {
    b2 := reflect.DeepCopy(b)
    b2.f = b2.f.Copy()
}

This is not true because a DeepCopy already copied f. Respect a customized Copy or equivalent, is similar to what was rejected in https://github.com/golang/go/issues/20444

I also still believe that if reflect.DeepEqual is going to exist it should ignore private fields by default, like json.Marshal does.

Then we don't need deep copy anyway, one can simply marshal and unmarshal.

If they should be copied along, you should implement a proper Copy method. But, as a convenience for simple cases, perhaps we can use struct tags to opt them in?

I personally object to this idea. No special reason, sorry. (Perhaps you could fill a separate proposal?)

[changkun]

Yes, generics is an unusual case. [...] The result is undoubtedly imperfect, but I think we did everything we could to try out possibilities before we committed to a specific design.

My apologies if the argument was annoying. In fact, I like the current generics design; not really criticizing it or so, but thanks for much effort in making it happen. I use generics as an argument against "I think that we need experience with an implementation to understand what the right choices maybe." Especially when we have evidence covering the past years to understand why and what the user needs as a bare minimum in terms of deep copy. Look back to the meeting proposal, Mutex.TryLock is an example that being accepted, but its document really doesn't say anything about when should we use them and only creates the impression "don't use it if you don't know what you are doing":

// TryLock tries to lock m and reports whether it succeeded.
//
// Note that while correct uses of TryLock do exist, they are rare,
// and use of TryLock is often a sign of a deeper problem
// in a particular use of mutexes. 

If this makes people feel better about introducing TryLock, DeepEqual seems could do it similarly:

// Note that while correct uses of DeepCopy do exist, they are not rare,
// but the use of DeepCopy is often indicating the copying object does not
// contain a singleton that should not be copied.

[rittneje]

This is not true because a DeepCopy already copied f.

No, DeepCopy naively copied f, meaning that it naively copied f.someFile, which, as mentioned several times, is unacceptable. One way or another, something has to call f.Copy(). If DeepCopy doesn't, then it is useless, because in general a Copy method could be added at any time in the future, and so we are back to square one.

Respect a customized Copy or equivalent, is similar to what was rejected in https://github.com/golang/go/issues/20444

That proposal isn't relevant. DeepCopy is more like json.Marshal/json.Unmarshal (which does support customization) than reflect.DeepEqual, especially given that you have already mentioned a few times that a deep copy won't be deep equal to the original value. In fact, it was argued against specifically because of the implication with unexported fields, but your original proposal violates this anyway.

I will also mention that personally, I've never had any use for reflect.DeepEqual, not just because of its lack of customization, but also because it lacks any mechanism to tell you what the discrepancy was. Honestly, I think adding it to the standard library was a mistake. I don't consider it a good yardstick to measure any new API against.

Then we don't need deep copy anyway, one can simply marshal and unmarshal.

You can, except that it will be significantly less efficient and cannot handle circular data structures.

Anyway, I feel like we're just cycling through the same points again and again, so I will leave it with this:

1. I am opposed to this API being in the standard library in its present form. I feel it will only encourage fragile code that is reliant upon implementation details that are subject to change. I am also opposed any reliance on the unsafe package (to copy unexported fields) that doesn't require the client to explicitly import unsafe.
2. To me, methods are always better than reflect magic, because there is an explicit contract that the library author has agreed to uphold.
3. There are several situations in which naively deep copying is wrong or ill-defined: *os.File, *sync.Mutex, *sync.Once, <-chan X, etc. Without the ability to customize DeepCopy, introducing any such fields can become a breaking change, even if the library author never intended the struct in question to be deep copied.

[changkun]

Anyway, I feel like we're just cycling through the same points again and again, so I will leave it with this:

I agree. Until now, all of us are simply repeating the same topic repeatedly. All arguments have been repeated too many times, and yet no new insights. I'll stop arguing if there are no more new interesting argumentations:

1. I am opposed to this API being in the standard library in its present form. I feel it will only encourage fragile code that is reliant upon implementation details that are subject to change. I am also opposed any reliance on the unsafe package (to copy unexported fields) that doesn't require the client to explicitly import unsafe.

You are requesting feature extensions that had been constantly rejected from the DeepEqual perspective. Referring to the third point of https://github.com/golang/go/issues/51520#issuecomment-1079645855

2. To me, methods are always better than reflect magic, because there is an explicit contract that the library author has agreed to uphold.

This is a purely subjective opinion. If that's the case, the reflect package should not exist.

3. There are several situations in which naively deep copying is wrong or ill-defined: *os.File, *sync.Mutex, *sync.Once, <-chan X, etc. Without the ability to customize DeepCopy, introducing any such fields can become a breaking change, even if the library author never intended the struct in question to be deep copied.

This is once again the first point. Referring to the updated proposal a few weeks ago:

// There are a few exceptions that may result in a deeply copied value not
// deeply equal (asserted by DeepEqual(dst, src)) to the source value:
//
// 1) Func values are still refer to the same function
// 2) Chan values are replaced by newly created channels
// 3) One-way Chan values (receive or read-only) values are still refer
//    to the same channel
//
// Note that for stateful copied values, such as the lock status in
// sync.Mutex, or underlying file descriptors in os.File and net.Conn,
// are retained but may result in unexpected consequences in follow-up
// usage, the caller should clear these values depending on usage context.
//
// The function panics/returns with an error if
//
// 1) source and destination values have different types
// 2) destination value is reachable from source value

If not enough, this could be:

// Note that while correct uses of DeepCopy do exist, they are not rare,
// but the use of DeepCopy is often indicating the copying object does not
// contain a singleton that should not be copied.

which holds similar functionality to what was documented in Mutex.TryLock.

[changkun]

For anyone who would like to try out this proposal, or argue against no actual implementation yet, I quickly prototyped an implementation here: https://pkg.go.dev/golang.design/x/reflect

After the implementation, I think this is the latest proposal:

// DeepCopy copies src to dst recursively.
//
// Two values of identical type are deeply copied if one of the following
// cases apply.
//
// Numbers, bools, strings are deeply copied and have different underlying
// memory address.
//
// Slice and Array values are deeply copied, including its elements.
//
// Map values are deeply copied for all of its key and corresponding
// values.
//
// Pointer values are deeply copied for their pointed value, and the
// pointer points to the deeply copied value.
//
// Struct values are deeply copied for all fields, including exported
// and unexported.
//
// Interface values are deeply copied if the underlying type can be
// deeply copied.
//
// There are a few exceptions that may result in a deeply copied value not
// deeply equal (asserted by DeepEqual(dst, src)) to the source value:
//
// 1) Func values are still refer to the same function
// 2) Chan values are replaced by newly created channels
// 3) One-way Chan values (receive or read-only) values are still refer
//    to the same channel
//
// Note that while correct uses of DeepCopy do exist, they are not rare.
// The use of DeepCopy often indicates the copying object does not contain
// a singleton or is never meant to be copied, such as sync.Mutex, os.File,
// net.Conn, js.Value, etc. In these cases, the copied value retains the
// memory representations of the source value but may result in unexpected
// consequences in follow-up usage, the caller should clear these values
// depending on their usage context.
func DeepCopy[T any](src T) (dst T)

Thanks to everyone for previous clarifying discussions.

---

PS. There was a large discussion on customization. A branched prototype implements a possible proposal (although this would be a different proposal):

// DeepCopyOption represents an option to customize deep copied results.
type DeepCopyOption func(opt *copyConfig)

// DisallowCopyUnexported returns a DeepCopyOption that disables the behavior
// of copying unexported fields.
func DisallowCopyUnexported() DeepCopyOption

// DisallowCopyCircular returns a DeepCopyOption that disables the behavior
// of copying circular structures.
func DisallowCopyCircular() DeepCopyOption

// DisallowCopyBidirectionalChan returns a DeepCopyOption that disables
// the behavior of producing new channel when a bidirectional channel is copied.
func DisallowCopyBidirectionalChan() DeepCopyOption

// DisallowTypes returns a DeepCopyOption that disallows copying any types
// that are in given values.
func DisallowTypes(val ...any) DeepCopyOption

// DeepCopy copies src to dst recursively.
//
// (...)
//
// To change these predefined behaviors, use provided DeepCopyOption.
func DeepCopy[T any](src T, opts ...DeepCopyOption) (dst T)

[rsc]

It doesn't sound like there is any consensus about adding this, which means we should probably decline this proposal.

[rsc]

Based on the discussion above, this proposal seems like a likely decline. — rsc for the proposal review group

[changkun]

It doesn't sound like there is any consensus about adding this

Frankly, it is not convincing and constructive enough to reject a proposal because there is no consensus in a discussion. The argument would be much stronger for technical reasons, such as language spec violation, implementation difficulty, etc. If there is a decision difficulty, it might be a reasonably well-balanced decision to put this on hold for a few years rather than close it right away. In this way, the issue can collect more opinions regarding the proposal.

[mvdan]

@changkun respectfully, I disagree that keeping this proposal open will help. If you want to continue the flow of ideas, I would once again suggest that you start a third-party package and use the issue tracker or discussion forum there. It's worked very well for go-cmp for example, which culminated into https://github.com/golang/go/issues/45200 after years of effort and validation from real users.

I would also like to give my two cents from the point of view of someone who has been quietly reading this thread: I'm getting the impression that you are defending this proposal rather than trying to see what the best outcome for Go would be, even if reaching that best outcome takes more experimentation outside of this repository. I understand that spending time on a proposal and having it rejected can be underwhelming, but this is part of what sets Go apart from other languages: it has kept the language and standard library small by rejecting many ideas.

[changkun]

I would once again suggest that you start a third-party package and use the issue tracker or discussion forum there. It's worked very well for go-cmp for example, which culminated into #45200 after years of effort and validation from real users.

I am not really objecting to this idea and have already placed the prototyped implementation in a third location. However, on behalf of the purpose of this proposal, I remained a defending position in this thread.

The "go-cmp" is another special case where the home organization account is google, which is more attractive to the public hence (I think) nicely yielding a successful example. The situation becomes different when such a location is fully a third party, which creates less impactful attraction and is considered much more untrustable. Based on my (naive) observation, I think the levels of trust and reliability decrease, from the standard package to golang.org/x, top-grade organizations, and then individual sources. Therefore it might be more helpful to gain more exposure when the association maintains.

As (the majority, I think) developers love to gain their individual impact, and sometimes might even be more frustrating for them when their idea is rejected from somewhere, achieved success in a third place, then being "stolen" (emphasize: this is not the optimal wording, but sometimes being abused under this context) to the place where originally said no. This may be considered harmful for both parties at extreme cases.

I would also like to give my two cents from the point of view of someone who has been quietly reading this thread: I'm getting the impression that you are defending this proposal rather than trying to see what the best outcome for Go would be, even if reaching that best outcome takes more experimentation outside of this repository. I understand that spending time on a proposal and having it rejected can be underwhelming, but this is part of what sets Go apart from other languages: it has kept the language and standard library small by rejecting many ideas.

Now, this is even more off-topic. Your encouraging words are quite appreciated in a community discussion. I think I am simply taking this chance to explore and exploit the current decision tendency when it comes to a borderline proposal. Specifically, when we (potentially) reach the Pareto optimality of a proposal, it is interesting to see the factors that determine a decision choice between all found Pareto frontiers. Compared to a controversial arena proposal, the latest comment was to add it under an experiment flag other than a similar decision of "likely decline." While I agree they might not be a completely fair comparison, this still reveals more observed potentially contradicting behaviors. In terms of this proposal, I'd say there are no actual hard feelings but simply observing a choice after an explorative discussion.

Subsequently, this made me an idea of having an official field experiment location might be interesting, where every controversial or suboptimal proposal could encourage people to contribute to the implementation. That place has no guarantee of any kind. Nevertheless, I'll stop here as it is completely not the topic of this proposal discussion.

[mvdan]

The "go-cmp" is another special case where the home organization account is google

Okay, so perhaps more examples would help. Take https://github.com/frankban/quicktest, which led to accepted proposals like https://github.com/golang/go/issues/41260 and https://github.com/golang/go/issues/32111.

While I agree they might not be a completely fair comparison

It really is comparing apples and oranges. It is literally impossible to implement a runtime change like arenas outside of the Go source tree.

It is of course within your right to stick to your arguments - my argument was that you would have a better chance of success if you emulate success stories like go-cmp or quicktest. It takes a lot more patience and effort to do it that way, but that's what it takes to get APIs right and write good proposals with detail and user experience.

[changkun]

Okay, so perhaps more examples would help. Take https://github.com/frankban/quicktest, which led to accepted proposals like #41260 and #32111.

Agree. On behalf of the proposal, I think we could continue arguing this, such as those examples exist, but are still rare compared to the declined ones, etc. :)

It really is comparing apples and oranges. It is literally impossible to implement a runtime change like arenas outside of the Go source tree.

This part I am not sure about. It might be more suitable to say comparing ants and elephants. From the discussion there, it seems it remains possible to implement the arena in a third package.

It is of course within your right to stick to your arguments - my argument was that you would have a better chance of success if you emulate success stories like go-cmp or quicktest. It takes a lot more patience and effort to do it that way, but that's what it takes to get APIs right and write good proposals with detail and user experience.

I do not object to this argument at all. That's why the proposal summarizes many existing practices from pkg.go.dev.

[atdiar]

I don't think I will be able to convince you, because you seem resolute that it is a good idea to have eventually such a function in the standard library but I still see problems. Any type whose constructor function is side-effectful cannot be safely deep-copied because it would break the semantics/invariants.

But you don't know at which depth such types may appear in a datastructure. Especially if these are unexported types that appear in unexported fields.

So even if deep-copying is possible it might not be a good idea in the general case to do it indiscriminately. Now in the few cases where it is desirable, it shouldn't be difficult to write the copying function by hand a priori.

Note that comparing values does not suffer from this because a comparison is not effectful in general. Which is why having deepEqual is less a problem.

I think that warrants caution and while I appreciate that you mentioned a few edge cases in the documentation, I think the issue can be more pervasive and seem underestimated.

[EbenezerJesuraj]

Hi, Having a function such as Deep - Copy which is again shared by many popular languages such as Python,JAVA in itself is a very good valid argument to have the feature.

In my opinion i don't see any drawbacks in language having such a feature in a standard library.

The issue as of now in my opinion seems to be more of a developer shortage one to implement such a feature in the library. But again the Benefits far outweigh the cons is what i am able to infer from here.

[EbenezerJesuraj]

Okay, so perhaps more examples would help. Take https://github.com/frankban/quicktest, which led to accepted proposals like #41260 and #32111.

Agree. On behalf of the proposal, I think we could continue arguing this, such as those examples exist, but are still rare compared to the declined ones, etc. :)

It really is comparing apples and oranges. It is literally impossible to implement a runtime change like arenas outside of the Go source tree.

This part I am not sure about. It might be more suitable to say comparing ants and elephants. From the discussion there, it seems it remains possible to implement the arena in a third package.

It is of course within your right to stick to your arguments - my argument was that you would have a better chance of success if you emulate success stories like go-cmp or quicktest. It takes a lot more patience and effort to do it that way, but that's what it takes to get APIs right and write good proposals with detail and user experience.

I do not object to this argument at all. That's why the proposal summarizes many existing practices from pkg.go.dev.

I do like the Apples and Oranges references here..

[rsc]

No change in consensus, so declined. — rsc for the proposal review group