Promote DeferredNativeArray from exploratory first Native Collection to useful, safe and consistent data structure

[jkeon]

The DeferredNativeArray is used very heavily throughout the Task system in ways that UnsafeList doesn't quite support.

• Different allocators for the collection and the underlying memory
• Not needing all the Add/Remove functionality.

However we now do need the ability to reset what the length is which makes it more like a list with Capacity and Length vs an Array which is always the one size.

The use case here is we know the max size an array can be so we allocate it. Then we loop through some candidates that may be assigned. Once done we reset the length of the array to only the amount of candidates that were actually written so that future jobs don't process garbage data at the end of the array.

This breaks the mental model of "Array" so think about renaming this class maybe? And maybe adding more capacity functionality?

[mbaker3]

FixedCapacityList is maybe an appropriate name?

Just using List doesn't quite work because these can't have their capacity expanded...or could they? What efficiencies do we realize by restricting the ability for the grow the capacity vs a List that you just choose not to grow.

Another thought, could we still use DeferredNativeArray and then just return a slice to represent the valid range?

[jkeon]

With the current work done in this class and how it's being used, it's definitely a List now.

The differences/advantages over using the built in UnsafeList are:

• Ours takes the Length/Capacity along with it in the pointer so you don't lose that information.
• The ability to have a different allocator for the list memory.

This task should now be about:

• Changing the name and updating the API to have all the list like functionality needed.
• Or investigating if we can accomplish what we need to using the UnsafeList and some internal Extension methods.

[mbaker3]

Sounds good.

The ability to have a different allocator for the list memory.

What does this mean? Do you mean the element data vs the capacity/length meta data unsafe storage?

[jkeon]

What does this mean? Do you mean the element data vs the capacity/length meta data unsafe storage?

Yep, so with a regular UnsafeList, if you want it to be TempJob allocated, you would create a new one, then dispose it and then create a new one etc etc.

The issue with that is that if you need to share that UnsafeList across different locations in the app, they all need to know whenever you create a new one or maintain a reference to somewhere to be able to get it. Which leads to a lot of coupling.

By having the meta data separate from the array data, you can allocate the meta data as Persistent, and everyone who needs to have access can get it once on construction and it never changes. The array data can be TempJob and changed every frame.

[mbaker3]

At that point what's the benefit of TempJob vs the data just being allocated persistent and cleared/resized by the owner anyway?

[jkeon]

Honestly without profiling I'm not sure.

But theoretically, it can be used to reduce memory pressure for data that doesn't need to be kept persistently all the time.

https://docs.unity3d.com/Manual/JobSystemNativeContainer.html

[jkeon]

Based on this comment:

https://github.com/decline-cookies/anvil-unity-dots/pull/142#discussion_r1081732702

We should also do a pass on when/how the safety is used and make it sure it is consistent with what Unity is doing.

Probably makes sense to have an Unsafe version and a Native version that wraps the Unsafe version like NativeList/UnsafeList.

[jkeon]

Make sure we check this comment https://github.com/decline-cookies/anvil-unity-dots/pull/142#discussion_r1081735766 and ensure that existing use cases are either supported or supported by a different structure and we communicate clearly when to use each data structure