Add 64 bits support to Array underlying storage

[GPSnoopy]

While System.Array API supports LongLength and operator this[long i], the CLR does not allow arrays to be allocated with more than 2^31-1 elements (int.MaxValue).

This limitation has become a daily annoyance when working with HPC or big data. We frequently hit this limit.

Why this matters

• .NET arrays is the de facto storage unit of many APIs & collections.
• Currently one has to allocate native memory instead, which does not work with managed code (Span could have helped, but it's been limited to int32 as well).
• HPC frameworks expects data to be contiguous in memory (i.e. the data cannot be split into separate arrays). E.g. BLAS libraries.
• Requires applications to be coded and designed differently if they handle <2G elements or >2G elements. I.e. It does not scale.
• It's an arbitrary limitation with little value to the user and application:
  • With a desktop with 64GB, why can one only allocate 3.1% of its memory in one go?
  • With a data centre machine with 1,536GB, why can one only allocate 0.1% of its memory in one go?

In C++ this is solved with std::size_t (whose typedef changes depending on the target platform). Ideally, .NET would have taken the same route when designing System.Array. Why they haven't is a mystery, given that AMD64 and .NET Framework appeared around the same time.

Proposal I suggest that when the CLR/JIT runs the .NET application in x64, it allows the array long constructor to allocate more than int.MaxValue items:

• Indexing the array with operator this[long i] should work as expected and give access to the entire array.
• Indexing the array with operator this[int i] should work as expected but implicitly limit the access to only the first int.MaxValue elements.
• The LongLength property should return the total number of elements in the array.
• The Length property should return the total number of elements in the array, or throw OverflowException of there are more than int.MaxValue elements (this matches the current behaviour on multi-dimensional arrays with more than int.MaxValue elements).

I naively believe that the above should not break any existing application.

Bonus points for extending 64-bit support to Span and ReadOnlySpan.

[tannergooding]

I naively believe that the above should not break any existing application.

One of the simplest breaks becomes any program that is doing the following:

for (int i = 0; i < array.Length; i++)
{
}

This works fine for any existing programs, since CoreCLR actually defines a limit that is just under int.MaxValue. Allowing values that are greater than int.MaxValue would cause an overflow to -2147483648 and either cause unexpected behavior or cause an IndexOutOfRangeException to be thrown.

[giuliojiang]

I frequently run into this array when processing large chunks of data from network streams into byte arrays, and always need to implement chunking logic in order to be able to process the data. It would be great to be able to use long indexes on arrays

[GPSnoopy]

@tannergooding That's why I proposed above to keep the existing behaviour of throwing OverflowException on Length when there are more than int.MaxValue elements. Nothing changes there.

I'm suggesting that changing the CLR implementation as proposed above would allow applications and people who want to to use large arrays without breaking existing applications. You are right that simply passing a large array into a library that does not support it will break, but at least this will give us choice. We need to start somewhere, and .NET cannot keep ignoring this problem.

[philjdf]

Yesterday I happily created an array in Python which contained more than two billion elements. When can I do the same in .NET?

Currently we get an exception if we try to construct an array with more than 2B elements. What's wrong with deferring that exception until something calls the Length property which can no longer return a valid int? @tannergooding's example wouldn't cause problems. Are there other examples which break?

[GrabYourPitchforks]

This is an interesting idea, but I wonder if it would be better to have a LargeArray<T> or similar class that has these semantics rather than try to shoehorn it into the existing Array class. The reason I suggest this course of action is that the GC currently has a hard dependency on the element count of any variable-length managed object fitting into a 32-bit signed integer. Changing the normal Array class to have a 64-bit length property would at minimum also affect the String type, and it may have other unintended consequences throughout the GC that would hinder its efficiency, even when collecting normal fixed-size objects. Additionally, accessing the existing Array.Length property would no longer be a simple one-instruction dereference; it'd now be an overflow check with associated branching.

If we had a theoretical LargeArray<T> class, it could be created from the beginning with a "correct" API surface, including even using nuint instead of long for the indexer. If it allowed T to be a reference type, we could also eliminate the weird pseudo-covariance / contravariance behavior that existing Array instances have, which would make writing to a LargeArray<T> potentially cheaper than writing to a normal Array.

[GPSnoopy]

@GrabYourPitchforks interesting facts about the GC. I wasn't aware of such limitations.

A LargeArray<T> class could be a temporary solution. My main concern is that it would stay a very niche class with no interoperability with the rest of the .NET ecosystem, and ultimately would be an evolutionary dead end. I do like the idea of nuint/nint though.

My gut feeling is that Array, String and the GC are ripe for a 64 bits overhaul. We should bite the bullet and do it. So far I've been quite impressed by the .NET Core team willingness to revisit old decisions and areas that Microsoft had kept shut in the past (e.g. SIMD/platform specific instructions, float.ToString() roundtrip fixes, bug fixes that change backward compatibility, etc).

[juliusfriedman]

I guess I could palette a LargeArray but if that's going to be implemented than I hope it doesn't create a new type which is not actually an Array, IMHO it would have been much easier to address this if we would have created another sub type of Array internally instead of inventing Span however Span also solves other problems....

[GSPP]

These days 2GB arrays are barely enough for many applications to run reliably. RAM prices have stagnated for a few years now. Surely, the industry will resolve this problem sooner or later. As RAM amounts resume increasing at Moores law rate this 2GB array issue will become very commonplace sooner or later.

A LargeArray<T> type might be a good medium term solution. But will 2GB arrays not be very commonplace 10 years from now? Do we then want to litter application code and API surfaces with LargeArray<T>? It would often be a hard choice whether to go for LargeArray<T> or T[].

Thinking in the very long term it seems far better to find a way to fix T[].

[GSPP]

If 64 bit support is implemented there could be a tool that analyzes your code for legacy patterns (e.g. usage of int Length or the typical for loop for (int i = 0; i < array.Length; i++)). The tool should then be able to mass upgrade the source code. This could be a Roslyn analyzer.

[GrabYourPitchforks]

Since this would be such a massive ecosystem breaking change, one other thing you'd probably have to do is analyze the entire graph of all dependencies your application consumes. If the change were made to T[] directly (rather than LargeArray<T> or similar), assemblies would need to mark themselves with something indicating "I support this concept!", and the loader would probably want to block / warn when such assemblies are loaded. Otherwise you could end up in a scenario where two different assemblies loaded into the same application have different views of what an array is, which would result a never-ending bug farm.

[juliusfriedman]

Not if large array was an array... i.e. derived from it if only internally perhaps; like I suggested back in the span threads.

[GrabYourPitchforks]

If large array is a glorified array, then you could pass a large array into an existing API that accepts a normal array, and you'd end up right back with the problems as originally described in this thread.

[GrabYourPitchforks]

Furthermore, I'm not sure I buy the argument that adding large array would bifurcate the ecosystem. The scenario for large array is that you're operating with enormous data sets (potentially over 2bn elements). By definition you wouldn't be passing this data to legacy APIs anyway since those APIs wouldn't know what to do with that amount of data. Since this scenario is so specialized it almost assumes that you've already accepted that you're limited to calling APIs which have been enlightened.

[juliusfriedman]

You have LongLength on the Array

The only fundamental diff is that one is on the LOH and one is not.

By virtue of the same fact span wouldn't be able to hold more than such either so large span must be needed also...

[GPSnoopy]

From what I can gather and summarise from the above, there are two pieces of work.

1. Update the CLR so that Array can works with 64-bit length and indices. This includes changes to the Array implementation itself, but as comments have pointed above, also to System.String and the Garbage Collector. It is likely to be relatively easy to come up with a fork of coreclr that can achieve this, as a proof of concept with no regard for backward compatibility.

2. Find a realistic way to achieve backward compatibility. This is the hard part. I think this is unlikely to succeed without compromising some aspect of the CLR. Whether it is Length throwing on overflow, or awkwardly introducing new specific classes like LargeArray.

But the more I think about it, the more I think this issue is missing the point and ultimately the real problem with .NET as it stands. Even if the initial proposal was to be implemented, it would only fix the immediate 64-bit issue with Array but still leave collections and Span with the same indexing and length limitations.

I've started reading The Rust Programming Language (kind of felt overdue) and it struck me that Rust also mimics C++ size_t and ssize_t with usize and isize. C# on the other hand somehow decided not to expose this CPU architectural detail and forces everyone to the lowest common denominator for most of it's API: a 32-bit CPU with 32-bit addressing.

I'd like to emphasis that the 32-bit limitation is purely arbitrary from a user point of view. There is no such thing as a small array and a big array; an image application should not have to be implemented differently whether it works with 2,147,483,647 pixels or 2,147,483,648 pixels. Especially when it's data driven and the application has little control on what the user is up to. Even more frustrating if the hardware has long been capable of it. If you do not believe me or think I'm talking nonsense, I invite you to learn how to program for MS-DOS 16-bit with NEAR and FAR pointers (hint: there is a reason why Doom required a 386 32-bit CPU).

Instead of tinkering around the edges, what is the general appetite for a more ambitious approach to fix this limitation?

Here is a controversial suggestion, bit of a kick in the nest:

• Take on @GrabYourPitchforks idea, introduce nint and nuint (I also like size and ssize, but I can imagine a lot of clashes with existing code).
• Allow implicit conversion from int to nint (and uint to nuint) but not the reverse,
• Change all Length properties on Array, String, Span and collections to return nint (leaving aside the debate of signed vs unsigned and keeping the .NET convention of signed lengths).
• Change all indexing operators on Array, String, Span and collection to only take nint.
• Remove LongLength properties and old indexing operators.
• Take a page from Nullable Reference book and allow this to be an optional compilation feature (this is where it hurts).
• Only allow a nint assembly to depend on another assembly if it's also using the new length types.
• But allow a global or per-reference "I know what I'm doing" override, in which case calling the old int32 Length property on an Array is undefined (or just wraps around the 64-bit value).
• Spend a decade refactoring for loops and var i = 0.

I understand this is far from ideal and can create uncomfortable ecosystem situations (Python2 vs Python3 anyone?). Open to suggestions on how to introduce size types in .NET in a way that doesn't leave .NET and C# more irrelevant on modern hardware each year.

[MichalStrehovsky]

If we can solve the issue with the GC not being tolerant of variable-length objects bigger than 2 GB, a couple things that might make LargeArray<T> with a native-word-sized-Length more palatable:

• Array length is already represented as a pointer-sized integer in the memory layout of arrays in .NET. On 64-bit platforms, the extra bytes serve as padding and are always zero.
• If we were to introduce a LargeArray<T>, we can give it the same memory layout as existing arrays. The only difference is that the bytes that serve as padding for normal arrays would have a meaning for LargeArray<T>.
• If the memory layout is the same, code that operates on LargeArray<T> can also operate on normal arrays (and for a smaller LargeArray<T>, vice-versa)
• We can enable casting between LargeArray<T> and normal arrays
• Casting from an array to LargeArray<T> is easy - we just follow the normal array casting rules (we could get rid of the covariance though, as @GrabYourPitchforks calls out). If we know the element types, it's basically a no-op.
• When casting from LargeArray<T> to normal arrays, we would additionally check the size and throw InvalidCastException if the LargeArray<T> instance is too long.
• Similar rules would apply when casting to collection types (e.g. you can cast LargeArray<T> to ICollection<T> only if the element count is less than MaxInt).
• Existing code operating on arrays doesn't have to worry about Length being longer than MaxInt. Casting rules guarantee this can't happen.
• LargeArray<T> would not derive from System.Array, but one could explicitly cast to it (the length-check throwing InvalidCastException would happen in that case). The same for non-generic collection types (e.g. casting to ICollection that has the Int32 Count property would only be allowed for a small LargeArray<T>).

This scheme would allow LargeArray<T> and normal arrays to co-exist pretty seamlessly.

[kasthack]

@MichalStrehovsky

Similar rules would apply when casting to collection types (e.g. you can cast LargeArray to ICollection only if the element count is less than MaxInt).

This would make LargeArray<T> incompatible with a lot of older code in these cases:

• Methods that currently accept a more specific type than they actually need(like ICollection<T> instead of IEnumerable<T> while the code just enumerates the values).

• Probably, some cases where ICollection<T> isn't used directly but inherited from. For instance, methods that accept ISet/IList/IDictionary<...>(I assume, that those interfaces and their implementations would be eventually updated for 64-bit lengths as well) which are inherited from ICollection<T>.

I would go with overflow checks when .Count is called to keep the compatibility and add .LongCount property with a default implementation to the old interfaces.

[MichalStrehovsky]

@kasthack I was trying to come up with a compatible solution where one never has to worry about getting OverflowException in the middle of a NuGet package that one doesn't have the source for. Allowing cast to ICollection<T> to succeed no matter the size is really no different from just allowing arrays to be longer than 2 GB (no need for a LargeArray<T> type). Some code will work, some won't.

With explicit casting, it's clear that we're crossing a boundary into "legacy land" and we need to make sure "legacy land" can do everything it would reasonably be able to do with a normal ICollection<T> before we do the cast.

methods that accept ISet/IList/IDictionary<...>

Arrays don't implement ISet/IDictionary so a cast to these would never succeed. For IList, the same rules would apply as for ICollection (ICollection was just an example above).

[philjdf]

@GPSnoopy's post makes me wonder whether the following variation might make sense:

1. Introduce new nint and nuint types, but don't change the signatures of anything to use them. Nothing breaks.
2. Introduce new arrays types (with fixed covariance), new span types, etc, which use nint and nuint. Keep the old ones and don't touch them. Make it fast and easy to convert between old and new versions of these types (with an exception if your 64-bit value is too big to fit into the 32-bit counterpart), but conversion should be explicit. Nothing breaks, type safety and all that.
3. Add a C# compiler switch /HeyEveryoneIts2019 which, when you write double[] you get the new type of array instead of the old one, everything's nint and nuint, and the compiler adds conservative/obvious stuff to convert to/from old-style arrays when you call outside assemblies which want old-style arrays. This way if it gets through the conversion without an exception, you won't break any old referenced code.

[GSPP]

It has been proposed that we could make Array.Length and array indices native-sized (nint or IntPtr).

This would be a portability issue. Code would need to be tested on both bitnesses which currently is rarely required for most codebases. Code on the internet would be subtly broken all the time because developers would only test their own bitness.

Likely, there will be language level awkwardness when nint and int come into contact. This awkwardness is a main reason unsigned types are not generally used.

In C languages the zoo of integer types with their loose size guarantees is a pain point.

I don't think we want to routinely use variable-length types in normal code. If nint is introduced as a type it should be for special situations. Likely, it is most useful as a performance optimization or when interoperating with native code.

---

All arrays should transparently support large sizes and large indexing. There should be no LargeArray<T> and no LargeSpan<T> so that we don't bifurcate the type system. This would entail an enormous duplication of APIs that operate on arrays and spans.

If the object size increase on 32 bit is considered a problem (it might well be) this could be behind a config switch.

---

Code, that cares about large arrays needs to switch to long.

Likely, it will be fine even in the very long term to keep most code on int. In my experience, over all the code I ever worked on, it is quite rare to have large collections. Most collections are somehow related to a concept that is inherently fairly limited. For example, a list of customers will not have billions of items in it except if you work for one of 10 companies in the entire world. They can use long. Luckily for us, our reality is structured so that most types of objects do not exist in amounts of billions.

I see no realistic way to upgrade the existing collection system to 64 bit indexes. It would create unacceptable compatibility issues. For example, if ICollection<T>.Count becomes 64 bit, all calling code is broken (all arithmetic but also storing indexes somewhere). This must be opt-in.

It would be nicer to have a more fundamental and more elegant solution. But I think this is the best tradeoff that we can achieve.

[FraserWaters-GR]

Just note there is already a case today where Length can throw. Multi-dimensional arrays can be large enough that the total number of elements is greater than Int.MaxValue. I think this is why LongCount was originally added.

[huoyaoyuan]

Note: Currently, when you write foreach over an array, the C# (and also VB) compiler actually generates a for loop, and store index in 32 bits. This means that existing code must break with array that > 2G, or at least a recompile is required.

I really hopes all size-like parameters among the ecosystem is using nuint (avoid checking for >= 0). There can be a [SizeAttribute] on all the parameters, and JIT generates the positive guard and bit expansion to allow existing int-size-compiled assembly to run with native-sized-corelib.

[lostmsu]

One option is to create a NuGet package for LargeArray<T>, and polish it in practical use. Once polished, make it part of the standard. This is what C++ did to parts of Boost.

But CLR should be ready by then.

[GPSnoopy]

But CLR should be ready by then.

@lostmsu Are you aware of ongoing work to fix this in CLR that I'm not aware of?

[lostmsu]

@GPSnoopy nope, that was an argument to start the CLR work ahead of time before BCL, so that BCL could catch up.

[GrabYourPitchforks]

I want to reiterate: there is little appetite within the .NET engineering team for allowing normal T[] / string / Span to be able to represent more than int.MaxValue items. The problems with shoehorning this in to the existing ecosystem have been described at length earlier in this thread.

Any proposal which modifies the implementation of these existing types is likely to stall. Alternative proposals which do not modify the implementation of these existing types are much more actionable.

[SommerEngineering]

Thanks @GrabYourPitchforks for the update. Although it's not good news: I am also currently reaching the limit of arrays because I am constructing large graphs. If the team does not want to remove the limit for T[], the only option is probably LargeArray<T>, based on an unsafe implementation. This would mean that we would have to free the memory manually e.g. via LargeArray<T>.FreeMemory(). Are there other proposals imaginable?

My opinion after 15 years as a C# developer: I would appreciate it if at least the limit of arrays T[] would be removed. At the same time it would make sense to use ulong as index instead of long.

[SommerEngineering]

It's me again. I've been trying to sketch a solution in a hurry. I put the code online at Github: https://gist.github.com/SommerEngineering/e8f61ada40b8ff3ecfe61f5263a666b9 (edit: see https://github.com/dotnet/runtime/issues/12221#issuecomment-665550339 for my ExaArray library)

This is a class for a one-dimensional array that pretends to be an array that might become larger than the arrays in C#. This is done using chunks of arrays and ulong as index.

Did I miss a technical detail here, or would that be a possible solution?

A wide range of applications in the field of graphs, big data or HPC should require one, two or three-dimensional arrays. According to this we might extend the chunk approach to these dimensions and would satisfy a wide range of developers without losing compatibility.

Dear .NET team: Couldn't you integrate such an approach into .NET 5? My code is obviously free to use without a license. I might write some unit tests tomorrow.

[lostmsu]

@SommerEngineering there are few reasons to use signed types to index arrays. For example, unsigned index makes it harder to iterate in reverse due to 0-1 >= 0.

[tannergooding]

It's also not strictly required to have an "unsafe" large array. You could, for example, imagine a LargeArray type which Array is implicitly convertible to and which has some specialized runtime/language semantics to make things work smoothly.

[Frassle]

I feel like given the underlying limitation to this is because of the GC there's really only two available options, assuming you want a container that has O(1) index access and is large (more than 2^32 entries): 1) You want a managed container that can hold references to objects, but doesn't need to be contiguous memory. I think this could be done via a container similar to the C++ deque where you have one array that holds references to other arrays, Given the largest array size is 2146435071 if you used the maximum size for parent array and the child arrays you could have up to 4607183514018775041 elements, which is about 2^62. 2) You want contiguous memory, and so because of underlying GC limitations we can't hold references in this but we could have a container that keeps a native allocation of memory. This would just be UnmanagedMemoryAccessor but with an array style interface instead of general read/write.

[SommerEngineering]

Note for all who are interested in a managed array with more than 2^32 elements: I published the library ExaArray on NuGet yesterday. In theory, it can hold up to 4.4 quintillion (4,410,000,000,000,000,000) elements. When using byte as T, this would require approx. 3.8 EB of memory. I tested it up to 32 GB of memory, though.

NuGet: https://www.nuget.org/packages/ExaArray/ Github: https://github.com/SommerEngineering/ExaArray Main repo: https://code.tsommer.org/thorsten/ExaArray License: BSD-3-Clause

Right now only a one-dimensional array is implemented. Thus, jagged arrays are working fine.

This library might helps someone until .NET supports larger arrays.

[Frassle]

@SommerEngineering the limits you've used in that library are a little off, you should be able to allocate up to 2,146,435,071 elements in a (non-byte) array, not just 2,100,000,000. Gives you an extra 46 million to work with.

[jkotas]

It would be better to use power of two as max chunk capacity to avoid expensive division. Power of two division is optimized into a shift that is a lot cheaper than arbitrary divide operation.

[Frassle]

The largest power of two that's smaller than 2,146,435,071 (a) is 1,073,741,824 (b) a^2) 4,607,183,514,018,775,041 b^2) 1,152,921,504,606,846,976

I mean probably still more than enough elements but is a lot smaller.

[SommerEngineering]

Thanks @Frassle and @jkotas for the feedback.

@jkotas: I was not yet familiar with the automatic optimization of divisions of powers of two. Thanks for that. @Frassle Thanks for the exact limit that .NET accepts. I determined 2,100,000,000 by trial-and-error.

Alright... now we have a classic trade-off situation: Either slightly more performance but with "only" 1.1 quintillion elements or less performance and 4.6 quintillion elements instead. Yes, 1.1 quintillion is still very much. Especially if we put that in relation to the currently available memory: Azure offers a maximum of 5.7TB, Google 11.7TB and Amazon 3.9TB per machine. None of these offers are sufficient to load 1.1 quintillion elements on one machine.

Thus: I added an option to chose a strategy. Default is maximizing performance. I added a unit test for measure the performance difference: Power of two divisions are approx. 1% faster in average. @jkotas: Is this the expected level of improvement? I mean one percent savings is a lot for a scientific simulation that runs for several months.

I released version 1.1.0 with this changes.

[jkotas]

The dividend has to be constant to make the optimization kick in. The configurability that you have introduced defeats the point of the optimization.

[SommerEngineering]

Thanks @jkotas ... I tested it once quickly: The performance improves up to about 70% (depending on the number of get/set calls). This is such a significant difference that I will give up the option to choose for this. Especially because I will use the library for scientific simulations, this performance difference is priceless. Thanks again for the advice. This evening I will release a new version 1.1.1 with the appropriate changes.

[verelpode]

Bonus points for extending 64-bit support to Span and ReadOnlySpan.

Following is an implementation that you can use immediately, today, with the preexisting NETFW 5.0. Interestingly, this does create a Span<T> even without any modifications to the NETFW 5.0 source code. "What's the catch?" The catch is either nothing or a big problem, depending on the needs of your particular application. For some applications, the following is sufficient, but for some other apps, it's insufficient.

readonly struct BigArray<T> where T : unmanaged
{
    private readonly LargeArrayElement[] _UnderlyingArray;
    private readonly Int64 _Length;

    public BigArray(Int64 inLength)
    {
        // Checks (such as negative length, overflow, maximum) are omitted -- this is only an example.
        _Length = inLength;
        int largeElementSize = Unsafe.SizeOf<LargeArrayElement>();
        _UnderlyingArray = new LargeArrayElement[(inLength + (largeElementSize - 1)) / largeElementSize];
    }

    public Span<T> Slice(Int64 inOffset, Int32 inLength)
    {
        if (unchecked((UInt64)inOffset > (UInt64)_Length | (UInt64)(Int64)inLength > (UInt64)(_Length - inOffset)))
            throw new System.ArgumentOutOfRangeException();

        ref T arrayDataRef = ref Unsafe.As<LargeArrayElement, T>(ref MemoryMarshal.GetArrayDataReference(_UnderlyingArray));
        return MemoryMarshal.CreateSpan<T>(ref Unsafe.Add(ref arrayDataRef, new IntPtr(inOffset)), inLength);
    }

    public T this[Int64 inOrdinal]
    {
        get
        {   // Checks are omitted.
            ref T arrayDataRef = ref Unsafe.As<LargeArrayElement, T>(ref MemoryMarshal.GetArrayDataReference(_UnderlyingArray));
            return Unsafe.Add(ref arrayDataRef, new IntPtr(inOrdinal));
        }
    }

    public Int64 Length { get { return _Length; } }

    [System.Runtime.InteropServices.StructLayoutAttribute(System.Runtime.InteropServices.LayoutKind.Sequential, Size = 64, Pack = 8)]
    private struct LargeArrayElement { UInt64 d0, d1, d2, d3, d4, d5, d6, d7; }

}

As you can see, the above Slice method has a 64-bit offset but a 32-bit length. Some applications only need the offset to be 64-bit. Such applications are perfectly happy having a 32-bit "window" into a large array that has a 64-bit length. Yes ofcourse some other applications require both the offset and span/slice length to be 64-bit, so it depends. For apps that only need the 64-bit offset, the above can be used. It's also possible in some cases to find it reasonably acceptable to modify the app/algorithms to work with 32-bit slices of a 64-bit array -- sometimes that's practical; sometimes that's impractical.

A second reason why this solution is only suitable for some apps/algorithms is that it uses where T : unmanaged as you can see above. In my particular case, this is no problem.

If your particular needs exceed the limitations of the above technique, then ofcourse @SommerEngineering's chunked technique is an option that is more generally applicable. The chunked technique is a good idea with advantages and disadvantages. The advantages are clear. One of the disadvantages is that it doesn't allow you to create a regular Span<T> slice whenever the slice overlaps 2+ chunks. There's no single winning design -- there are multiple winners for multiple different circumstances of different apps/algorithms. Obviously, choose the one that best suits your particular needs.

I'm sorry if I'm slow to reply -- I'm overloaded currently.

[verelpode]

@GrabYourPitchforks wrote:

there is little appetite within the .NET engineering team for allowing normal T[] / string / Span to be able to represent more than int.MaxValue items.

I find that understandable. What if the solution would be that no change is made to the .NETFW other than a compile-time #if that produces a "scientific edition" of .NETFW + runtime? In this idea, the normal edition would continue to use Int32 for Span<T>.Length, and the normal runtime would continue to limit arrays to approx Int32.MaxValue, whereas the scientific edition would use Int64 or nint for:

• Span<T>.Length
• Memory<T>.Length
• Array.Length
• ArraySegment<T>.Offset
• ArraySegment<T>.Count
• List<T>.Count
• List<T>.Capacity
• SortedSet<T>.Count (rb-tree)
• HashSet<T>.Count
• Array.BinarySearch<T>(T[] array, int index, int length, T value)
• Array.Sort<T>(T[] array, int index, int length)
• Array.IndexOf<T>(T[] array, T value, int startIndex, int count)
• Array.Resize<T>(ref T[] array, int newSize)

This doesn't mean creating a mess of hundreds of #if directives scattered in awkward places. Not at all. The using alias feature is helpful here -- a single #if directive changes the entire .cs file in every place where the applicable integer type is used. For example:

#if SCIENTIFIC_EDITION
using OrdinalType = System.Int64;
#else
using OrdinalType = System.Int32;
#endif

namespace System
{
    public readonly ref struct Span<T>
    {
        internal readonly ByReference<T> _pointer;
        private readonly OrdinalType _length;

        public OrdinalType Length { get => _length; }
        // ...
        public Span<T> Slice(OrdinalType offset, OrdinalType length) { ... }
    }
}

Usage of a couple of other similar basic tricks would also be desirable to keep the number of occurrences of #if SCIENTIFIC_EDITION to a minimum. There is only a small quantity of patterns affected by this issue, such as checking for an invalid slice offset + length pair, and these patterns could be moved to standardized static methods in a static class (a good practice in any event), instead of every ".cs" file containing its own custom non-standard implementation of the same patterns. This way, the number of occurrences of #if SCIENTIFIC_EDITION (across various .cs files) would be kept to a fairly small number.

If desired, an actual ordinal type could be defined as more than only a using alias. I'm unsure whether this is good, but it would be possible to consider defining an ordinal type that is visible at runtime similar to System.IntPtr:

namespace System
{
    public readonly struct OrdinalType
    {
    #if SCIENTIFIC_EDITION
        private readonly System.Int64 _value;
    #else
        private readonly System.Int32 _value;
    #endif
    }

    public readonly ref struct Span<T>
    {
        private readonly OrdinalType _length;       
        public OrdinalType Length { get => _length; }
        // ...
        public Span<T> Slice(OrdinalType offset, OrdinalType length) { ... }
    }
}

See also the unfortunate System.Index and System.Range. This is debatable, but in my personal opinion, System.Index.IsFromEnd has much less value than what System.Index could have been: A simple index/ordinal type that conditionally supports 64-bit, without any support for the lesser-value and strangely-one-based Index.IsFromEnd feature.

I would think it's reasonable if scientific applications are required to use a scientific edition in order to get the 64-bit Span<T>.Length etc. It could even be viewed as a badge of honor for a program/algorithm that consumes so much data that it requires a scientific edition of the framework and runtime.

[GrabYourPitchforks]

The problem isn't really the runtime or SDK. We can make whatever changes we want to System.Private.CoreLib.dll, System.Memory.dll, and friends.

The problem is the entire ecosystem of .NET assemblies and packages that exist around the SDK. When suggesting a fundamental breaking change to the shape of System.Array or T[], the result would end up bifurcating the ecosystem. All packages target .NET5 or earlier (int32 lengths), or they target .NET6 or later (nuint lengths). You cannot bring a package compiled for one world into the other world. It's hitting a giant reset button that invalidates every DLL that has ever been compiled over the past 20 years.

Perhaps there's a user base that would be willing to rewrite / recompile every single DLL (and every single dependency!) they have in their project. But given the amount of work this would entail I have to imagine that potential user base would be very, very small.

[verelpode]

You cannot bring a package compiled for one world into the other world. It's hitting a giant reset button that invalidates every DLL that has ever been compiled over the past 20 years.

You're absolutely right ofcourse, but the more you think about hitting that "giant reset button", the more appetizing it becomes, doesn't it? At first I thought, "Oh my god! It's terrible!", but after an hour, I thought, "It's not as bad as it sounds", and after 2 hours and considering the alternatives, I thought, "Yes! Hit it! Why not?!"

The alternatives, such as a runtime that "would allow LargeArray and normal arrays to co-exist pretty seamlessly" (@MichalStrehovsky ), cause much more work and trouble, and as you said, the team has little appetite for it, understandably.

Not only does the .NET team have little appetite for it, but people like @GPSnoopy and @GSPP and nearly everyone doing scientific or big-data work ALSO have little appetite for co-existence of normal and big arrays. Nobody truly has big appetite for simultaneous usage of normal and big arrays. The people who want big arrays would either immediately or eventually be happy with using big arrays for every array in their entire program.

So the problem you mentioned is a real problem that would continue to exist, but I think people like @GPSnoopy and @GSPP would surely end up accepting it anyway. Either they'll happily accept it immediately, OR they'll think about it and discuss it and eventually decide that they want it, regardless of the limitation that you mentioned.

Isn't that right, @GPSnoopy and @GSPP ? There's a high risk of receiving nothing because the team said they have little appetite for it. Alternatively, instead of nothing happening, you could potentially have this:

• 64-bit Span<T>.Length, Array.Length, etc.
• Only supported in a scientific edition (compile-time #if) of runtime and SDK.
• Requires you to "recompile every single DLL (and every single dependency!)" as @GrabYourPitchforks said.
• No support for "small" arrays. All arrays are big arrays meaning Array.Length would always be Int64 in the scientific edition.
• "Small" arrays have very slightly higher overhead in the scientific edition than in the normal edition, because all arrays are big arrays.

The recompiling of DLL's is a chore, yes, but it's still much better / less bad than having NO support for big arrays. It's also a chore to install a special edition of the runtime and SDK, but in return for these chores, you get big arrays.

@GPSnoopy and @GSPP, currently all of your pointers and references are 64-bit pointers/references, regardless of whether they're pointers to small or big pieces of data, and regardless of their storage location in RAM. Do you want a feature that gives you simultaneous co-existence of 32-bit and 64-bit pointers, that allows you to use 32-bit pointers in some cases and 64-bit pointers in some other cases? No, you don't want that, understandably. It's much easier for you to just simply use 64-bit pointers for everything, regardless of the fact that each 64-bit pointer consumes 4 more bytes of RAM than 32-bit pointers.

Now, similar question, do you want co-existence of 32-bit and 64-bit arrays? Your answer is "Don't care", isn't it? You're happy to use 64-bit arrays for everything, including small arrays, aren't you? Just like how you already use 64-bit pointers for everything. The extra overhead of 64-bit pointers and 64-bit arrays is negligible in your viewpoint, because more importantly you need the ability to just HAVE these big things functioning. It doesn't matter that 64-bit consumes 1% more of your RAM or processor power when you simply use 64-bit for everything big AND small -- that's a price well-worth paying.
Better to focus on doing the actual science work rather than juggling co-existence of 32-bit and 64-bit things for little benefit in return for all the juggling work. Easier to make it all 64-bit, even though it means recompiling DLL's and installing a special scientific edition of the runtime.

[GPSnoopy]

@verelpode Pretty much, yes 😄. We would move our entire code base to 64-bit arrays. I also agree that having two array implementations (normal and large) is just asking for trouble, we have no interest in that.

Microsoft missed a big opportunity to hit the reset button when introducing .NET Core.

Having a big reset is not as bad as it seems at first. Nuget packages could dual-target legacy arrays and 64-bit arrays for a while. Apple has pretty much nailed this one with x86, x64 and ARM transitions.

If Array.Length became nint, most of the code out there would still compile and work out of the box (maybe some warnings though if the loop is using int instead of nint?). The world would continue to spin. Sure, you'll hit an infinite loop if you pass a array whose length is greater than 2^31 to one of these loops; but the reality is that only a small subset of our dependencies actually would need to deal with 64-bit arrays with length greater than the 32-bit limit.

There are no perfect solutions. But I'm indeed surprised that Microsoft developers are so fearful of this fundamental change given that they have made pretty disruptive changes in the past (e.g. .NET Core vs .NET Framework, nullables, float string representations, etc).

[GPSnoopy]

Like nullables, the compiler could warn on boundaries between assemblies (e.g. calling assemblies is compiled with 64-bit arrays and callee is legacy). Then the user has the option to override the compiler warning (e.g. ! when using nullables).

[verelpode]

@GPSnoopy

HPC frameworks expects data to be contiguous in memory (i.e. the data cannot be split into separate arrays). E.g. BLAS libraries.

As a workaround in the meantime, here's an example of how the BLAS asum function can be executed with my aforementioned BigArray<T> example. The following supports both 64-bit offset and 64-bit length, and executes the asum over the entire 64-bit length of the array, or over a slice of the array if you use the second method. The entire array is contiguous in RAM. Automatic garbage collection operates in the normal manner, as no unmanaged memory is allocated here.

static class BLAS
{
    public static unsafe double asum(BigArray<double> x, Int64 incx)
    {
        fixed (double* p = &x.GetItemReference(0))
        {
            return cblas_dasum(x.Length, p, incx);
        }
    }

    public static unsafe double asum(BigArray<double> x, Int64 x_sliceOffset, Int64 x_sliceLength, Int64 incx)
    {
        // This example omits the check of the validity of x_sliceOffset and x_sliceLength.
        fixed (double* p = &x.GetItemReference(x_sliceOffset))
        {
            return cblas_dasum(x_sliceLength, p, incx);
        }
    }

    [System.Runtime.InteropServices.DllImportAttribute("mkl.dll")]
    private static extern unsafe double cblas_dasum(Int64 n, double* x, Int64 incx);
}

// The following contains an additional method "GetItemReference" that was not included in my previous message, and more info about StructLayoutAttribute.
readonly struct BigArray<T> where T : unmanaged
{
    private readonly LargeArrayElement[] _UnderlyingArray;
    private readonly Int64 _Length;

    public BigArray(Int64 inLength)
    {
        // Checks (such as negative length, overflow, maximum) are omitted -- this is only an example.
        _Length = inLength;
        int largeElementSize = Unsafe.SizeOf<LargeArrayElement>();
        _UnderlyingArray = new LargeArrayElement[(inLength + (largeElementSize - 1)) / largeElementSize];
    }

    public Span<T> Slice(Int64 inOffset, Int32 inLength)
    {
        if (unchecked((UInt64)inOffset > (UInt64)_Length | (UInt64)(Int64)inLength > (UInt64)(_Length - inOffset)))
            throw new System.ArgumentOutOfRangeException();

        ref T arrayDataRef = ref Unsafe.As<LargeArrayElement, T>(ref MemoryMarshal.GetArrayDataReference(_UnderlyingArray));
        return MemoryMarshal.CreateSpan<T>(ref Unsafe.Add(ref arrayDataRef, new IntPtr(inOffset)), inLength);
    }

    public T this[Int64 inOrdinal]
    {
        get
        {   // Checks are omitted.
            ref T arrayDataRef = ref Unsafe.As<LargeArrayElement, T>(ref MemoryMarshal.GetArrayDataReference(_UnderlyingArray));
            return Unsafe.Add(ref arrayDataRef, new IntPtr(inOrdinal));
        }
    }

    public ref T GetItemReference(Int64 inOrdinal)
    {
        if (inOrdinal < 0 | inOrdinal >= _Length) throw new System.ArgumentOutOfRangeException();
        return ref Unsafe.Add(ref Unsafe.As<LargeArrayElement, T>(ref MemoryMarshal.GetArrayDataReference(_UnderlyingArray)), new IntPtr(inOrdinal));
    }

    public Int64 Length { get { return _Length; } }

    // Note in the StructLayoutAttribute, you can increase "Size" to, for example, 512 bytes, even without adding any additional fields to the struct. 
    [System.Runtime.InteropServices.StructLayoutAttribute(System.Runtime.InteropServices.LayoutKind.Sequential, Size = 64, Pack = 8)]
    private struct LargeArrayElement { UInt64 d0, d1, d2, d3, d4, d5, d6, d7; }
}

[GrabYourPitchforks]

If Array.Length became nint, most of the code out there would still compile and work out of the box.... Sure, you'll hit an infinite loop if you pass a array whose length is greater than 2^31 to one of these loops....

These two sentences seem to be in contradiction to one another. If your application relies on some third-party library, how do you know if that library has been enlightened for large arrays? This puts the onus on the app developer to exercise edge cases in every single one of their code paths and in every single one of their dependencies' code paths. Yes, technically the app developer is ultimately responsible for all the code they ship, but realistically nobody has the time to be this diligent. The end result would be difficult-to-debug problems in production environments and a frustrating developer experience. And I know the "things are limited to 32 bits" situation now is frustrating, but at least it falls under the "you know about this limitation ahead of time" umbrella of frustration.

But I'm indeed surprised that Microsoft developers are so fearful of this fundamental change given that they have made pretty disruptive changes in the past (e.g. .NET Core vs .NET Framework, nullables, float string representations, etc)

.NET Core 1.x was a truly fundamental shift from the way .NET Framework operated. Many APIs were blown apart and refactored, functionality was moved to many different assemblies in a pay-for-play model, and we removed APIs that we didn't want to bring forward to the .NET Core world. For instance, System.AppDomain was absent from .NET Core 1.x because .NET Core didn't (and still doesn't, and never will) support the concept of app domains. It was supposed to be a utopia of keeping only what we wanted while letting legacy APIs and outdated paradigms stay behind.

The feedback was overwhelmingly negative.

Developers needed to be able to cross-target between .NET Core and .NET Framework. Or they had deadlines that required them to migrate their code now and not take weeks to excise legacy components. Or they had existing DLLs targeting .NET Framework but had since lost the source to them, so they needed to bring those DLLs forward to Core without recompiling. Or any number of scenarios where they absolutely could not move to Core until we brought back some feature they [rightly or wrongly] depended on.

This isn't to say that we're fearful of disruption. You've pointed out some good examples with floating-point representation and nullability (though nullability annotations truly are opt-in). In .NET 5 we switched our globalization stack from NLS to ICU (though this brought with it a whole slew of complaints, tracked at the top of https://github.com/dotnet/runtime/issues/43956). The goal is to be smart about any disruption we cause. When weighing any disruptive idea, we'll consider how many people will benefit from it in practice. The examples I gave in this paragraph are situations where we determined that the disruption was worth it because it would benefit the ecosystem at large, perhaps by making something more standards-compliant or by ironing out the differences between Linux and Windows when people deploy their applications across different OSes.

If we introduce disruption that affects the developer ecosystem at large but benefits only a small handful of people, then we might be more inclined against an idea, since we might not see it as a net benefit to the ecosystem. Of course, I admit our value judgment could be wrong. If we've misjudged the number of people who would benefit or the actual disruption that would be caused, then our weightings would lead to an incorrect decision. I don't know for certain if that's the case here, but given the pros and cons stated so far I suspect we're weighting correctly.

[verelpode]

@GrabYourPitchforks

The feedback was overwhelmingly negative.

Indeed. For example, in my case, I didn't use .NET Core for anything other than experiments and internal testing, until .NET 5.0 was released. I played with .NET Core, but for production purposes, for the end-users, I published the software using NETFW 4.8 followed by NETFW 5.0 and ASP.NET 5.0, without ever publishing any version that used .NET Core 1, 2, or 3. For me, .NET Core 1, 2, and 3 were effectively alpha and beta versions, and NETFW 5.0 was finally the production version that I took seriously. And today I'm still calling it NETFW 5.0 even though it's not called that anymore.

A far worse version of the same type of problem was UWP formerly WinRT formerly WinPhone 8. Again I only ever used UWP for internal experiments and never actually published anything with it, because, unfortunately, to my great disappointment, UWP never really felt like it moved past the beta stage.

So I'm very much one of the people you're talking about. But here's the thing: It's not the changes that bothered me. For example, Azure nuget packages made breaking changes multiple times and it didn't bother me.

So what's going on here, why did I publish using the latest versions of Azure packages including their breaking changes, yet published nothing using .NET Core 1, 2, 3 and UWP ? Here's the critical difference:

• Making disruptive/breaking changes
• ...versus...
• Removing functionality!!

Azure "broke" things with their changes but it was no problem for me, because they didn't remove functionality. The new version with breaking changes still allowed me to do what I needed to do. The functionality was not removed or missing; it was just changed. In contrast, the huge problem with UWP was that the functionality was just GONE and there was no replacement for it.

A new/changed version of the functionality is OK, but you can't remove the old version before the new version is ready! But that's exactly what they did with UWP. They removed an incredibly large amount of functionality before the replacements were made available. In some cases, the replacements were never delivered. So with UWP, the functionality was just GONE and us developers were left wondering, "uh what now? It's gone and there's no real replacement". In contrast, in Azure, things didn't go missing, rather it was just changed, so it was straightforward to adapt to these changes. I can adapt to changes, but I can't just suddenly adapt to having a bunch of components suddenly disappear even before true non-beta replacements are delivered.

Another big problem with UWP was being practically almost-forced to upgrade to the latest version immediately, even when the timing is terrible because of other business/development reasons. In contrast, this was no problem with Azure packages because if the new Azure package (including breaking changes) is released and the timing is bad for upgrading to it immediately, then no problem at all, just simply wait until you're ready to click the "Upgrade" button in the Nuget Package Manager. This worked for Azure because it was delivered as a Nuget package, whereas UWP was not delivered as a Nuget package.

That said, I don't ask for Array.Length to be changed from Int32 to Int64 in the normal edition. I only support the idea in a #if SCIENTIFIC_EDITION. I think it's increasingly important for scientific customers. It's also relevant in apps that don't seem scientific, such as opening a high resolution x-ray or computer tomography scan in Adobe Photoshop. These kinds of apps are willing and able to accept a "scientific" or "big" edition that supports 64-bit array lengths. For other developers, especially developers running apps in tablets, give them the normal edition with 32-bit array length as-is.

BTW, thank you very much to the .NET Team for adding the System.Runtime.CompilerServices.Unsafe, which did not exist in .NET Framework 4.8. I find this Unsafe class very helpful for specialized situations. Ofcourse I follow the recommended practice of trying to avoid using it (use only when necessary), but in these special cases, it's very nice to have it available, and to know that C# can achieve "native code" performance, entirely in C#, without being forced to resort to writing part of the program in a C++ .DLL and the rest in C#. It's great to have in the toolbox.

[verelpode]

@GrabYourPitchforks

If we had a theoretical LargeArray class, it could be created from the beginning with a "correct" API surface, including even using nuint instead of long for the indexer. If it allowed T to be a reference type, we could also eliminate the weird pseudo-covariance / contravariance behavior that existing Array instances have, which would make writing to a LargeArray potentially cheaper than writing to a normal Array.

Unfortunately that sweet idea doesn't work in practice, as far as I know. I could be misunderstanding you, but if you mean what I think you mean, then it doesn't work. You can eliminate that weird pseudo-covariance/contravariance by using multiple interfaces, but you said class not interface. It's impractical or impossible to make a single class that eliminates that weird pseudo-covariance/contravariance behavior. If you wish, you could provide several interfaces that eliminate it, but not a class.

Thus the proposed LargeArray<T> class would not eliminate it, but this class could optionally support interfaces that eliminate it, but this doesn't achieve much because people aren't going to suffer the constant hassle of typecasting a LargeArray<T> class to a particular interface every time they use it covariantly, and then typecast it again to a different interface merely to access the same array contravariantly in the next line of code. People aren't going to tolerate this inconvenience:

LargeArray<T> myArray = new LargeArray<T>();
ICovariantLargeArray myArrayCovariantly = myArray;
myArrayCovariantly.DoSomething(...);
IContravariantLargeArray myArrayContravariantly = myArray;
myArrayContravariantly.DoSomething(...);

That's a constant hassle for too little benefit.

To better see why the sweet idea doesn't work, look at System.Collections.Generic.IReadOnlyList<T> and IReadOnlyCollection<T>. They both define their T as covariant. This gives them an advantage/flexibility. However, this flexibility comes at a high cost because it means that IReadOnlyList<T> cannot support important methods such as:

bool Contains(T inValue);
int FindItem(T inValue);
void CopyTo(T[] inDestinationArray);
void CopyTo(ArraySegment<T> inDestinationSegment);

Imagine a class List<T> that doesn't support Contains(T)! People won't accept this. I'm not saying IReadOnlyList<T> is wrong. It's not wrong because it's an interface. It's alright to do that with an interface; one of multiple interfaces. But LargeArray<T> would be a class. I don't see how it's possible to give such a class purity in co- and contra-covariance. I think, if you try to "eliminate the weird pseudo-covariance / contravariance" in a LargeArray<T> class, it would cause more hassle than the benefit it delivers.

But maybe I'm mistaken because maybe you meant something else than what I've presumed above.

It seems like the option that causes the least amount of work/difficulty for the .NET Team is the #if SCIENTIFIC_EDITION idea, and the customers appear to be satisfied with this solution, so it seems like a win-win solution, and also good for Microsoft's marketing in attracting big-data, scientific, engineering customers. The idea doesn't have zero problems, but the problem of recompiling DLL's/dependencies appears to be a price that big-data customers are willing to pay, in order to get a functioning solution.

[GrabYourPitchforks]

It's also relevant in apps that don't seem scientific, such as opening a high resolution x-ray or computer tomography scan in Adobe Photoshop. These kinds of apps are willing and able to accept a "scientific" or "big" edition that supports 64-bit array lengths.

I understand the scenario. However, I'm trying to convey that this is viral. Now Photoshop would become a "scientific edition" application, which means that all of its plugins would also have to become so. The end effect is that in one action, it invalidates the entire rich ecosystem of plugins. You'll have to go to all of your plugin authors and ask them to produce new builds for you if you want to continue consuming them. (And this doesn't even touch on what a time sink it would be for an app as large as Photoshop - with likely hundreds of third-party dependencies - to make sure that all of its dependencies are also "scientific edition" enlightened.)

Re: covariance / contravariance, I was referring to the following example.

string[] strArray = new string[] { /* ... */ };
object[] objArray = (object[])(object)strArrray; // cast succeeds at runtime!
objArray[0] = new Person(); // uh-oh (compile successfully, throws exception at runtime)

For any T[] instance where T is a reference type and not sealed, all not-provably-nullptr writes to the array incur a type check in addition to the standard bounds check. These checks prevent storing an invalid element into the backing array, even if the backing array is currently projected as an array of a supertype.

This is not an issue with Span<T>, since Span<T> doesn't provide covariance / contravariance. (ReadOnlySpan<T> does, but that's a different story.)

string[] strArray = new string[] { /* ... */ };
object[] objArray = (object[])(object)strArrray; // cast succeeds at runtime!
Span<object> objSpan = objArray; // implicit conversion throws ArrayTypeMismatchException at runtime
objArray[0] = new Person(); // the app never even gets to this point

Since Span<T> doesn't support this type of variance, writes will never incur a type check. In certain cases, this can lead to significant performance wins.

using System;
using System.Diagnostics;

namespace MyApp
{
    class Program
    {
        static void Main(string[] args)
        {
            const int BATT_COUNT = 1_000_000_000;
            var sw = new Stopwatch();

            while (true)
            {
                IConvertible[] arr = new IConvertible[1];
                Span<IConvertible> span = arr;

                sw.Restart();
                for (int i = 0; i < BATT_COUNT; i++)
                {
                    WriteToArray(arr);
                }
                Console.WriteLine(sw.Elapsed);

                sw.Restart();
                for (int i = 0; i < BATT_COUNT; i++)
                {
                    WriteToSpan(span);
                }
                Console.WriteLine(sw.Elapsed);
                Console.WriteLine();
            }
        }

        static void WriteToArray(IConvertible[] array)
        {
            array[0] = "Hello!";
        }

        static void WriteToSpan(Span<IConvertible> span)
        {
            span[0] = "Hello!";
        }
    }
}

Sample output on my machine, showing 70% reduction in runtime:

00:00:05.3400232
00:00:01.6970198

00:00:05.2633273
00:00:01.6461046

00:00:05.2352003
00:00:01.6390955

So the proposal for a LargeArray type would be that it not implement any type of variance. This also limits allowable interfaces to pretty much IEnumerable<T>. But maybe that's ok for our scenario.

[verelpode]

Since Span doesn't support this type of variance, writes will never incur a type check. In certain cases, this can lead to significant performance wins.

hmm, that seems like more of an argument in favor of the opinions of @GPSnoopy and @GSPP. As @GPSnoopy said in the first message: "Bonus points for extending 64-bit support to Span and ReadOnlySpan."

i.e. you're saying Span<T> has significant performance wins, and @GPSnoopy did ask for 64-bit versions of Span and ReadOnlySpan, and now there's another reason for wanting 64-bit Span -- 70% reduction in runtime while also supporting 64-bit lengths -- wouldn't that be great to have?

I love the 70% reduction in runtime, that's compelling, and I didn't know that there was such a big speed difference between Span and Array under those circumstances, and I'm glad you informed me about it, but shouldn't this speed gain remain the same as it is now, meaning available via Span<T> and not System.Array?

Making this speed gain available via a LargeArray<T> class sounds great at first, but the catch seems awfully hard to accept:

This also limits allowable interfaces to pretty much IEnumerable.

I don't think I could work with that in practice. Only IEnumerable<T> and no other interfaces? It goes against what I've been doing in a many places in many .cs files for years. It goes against my whole trend. Over time I've actually been adding more interfaces to collection objects, not reducing the number of implemented interfaces. I find these interfaces immensely useful. I believe I'm willing to continue to pay the performance price, even if it's as high as 70%, in order to retain the ability to implement several interfaces in collection objects not only IEnumerable<T>.

It's like I said in my previous message about IReadOnlyList<T> -- I see the advantage of it, and it's not wrong, but it comes at a high cost -- inability to support important methods. Or in the case of the proposed LargeArray type, inability to support any of the methods that exist in any and all of the collection interfaces in the entire .NET framework other than IEnumerable<T>. That's a high price to pay for the sweet 70% performance gain. I think in the end, functionality has to take priority over speed when both can't be maximized.

Slower (or even slow) success in doing everything we need to do is better than very quickly doing only half of what we need to do. But it wouldn't be slow. Actually the speed would be fast, AND it would support 64-bit, if Span<T>.Length and Array.Length were conditionally compiled as 64-bit, and if the developers know the important point you made -- that in places where you require max speed with arrays of references, then you should preferably make a Span from the Array and write to the Span instead of directly writing to the Array. Very good to know! But I wouldn't change how this speed gain is made available via Span<T> not Array.

And this doesn't even touch on what a time sink it would be for an app as large as Photoshop - with likely hundreds of third-party dependencies - to make sure that all of its dependencies are also "scientific edition" enlightened.

I know you're right, but aren't you putting scientific developers between a rock and a hard place? It's true what you say about that Photoshop example, but what's the alternative? It seems like big-data developers end up being stuck with:

1. No big arrays via a "scientific edition", because of the true problem you described.
2. No big arrays via any other solution, because the .NET Team seems to have little motivation to implement these other solutions, for reasons that are also valid and understandable.

You're right on point 1 and you're also right on point 2, but unfortunately this practically means the big arrays will probably be stalled forever, never delivered, making C# unattractive for sci/big-data developers.

True what you say about Photoshop plugins, but what if nearly all of those plugin developers say they're willing to accept the recompile anyway? This is the case with the sci/big-data -- they're apparently willing to accept the downside of the recompile. Yes it's a problem, you're right, but they accept this problem anyway. It's a price they're willing to pay.