Advanced Extents & Segments Proposal re C# 8.0 Ranges

[verelpode]

Advanced Extents & Segments Proposal

Rough sketch -- Version 1 (2019/05/07)

The potential for advanced Extents and Segments in C# is exciting and under-estimated. I am writing this proposal because I feel that the current version of the C# 8.0 Range proposal is not yet ready for release. It would be sad if a half-baked version is released prematurely and set in concrete forever. Although future extensions are possible in C# 8.5, this becomes difficult when it breaks compatibility with the syntax and design in C# 8.0, therefore it would be best to design the advanced scenarios NOW not later.

This is a rough sketch and I've been forced to rush it, thus I'm sure it contains some mistakes and omissions, but it is nevertheless worthwhile to present. Example:

int myOffset = ...;
int myLimitOffset = ...;
Segment<char> mySegment = ...;
Segment<char> resultSegment = mySegment[myOffset from start until element is CharMatchers.VerticalWhiteSpace 
     && element != '\u2029' with fence myLimitOffset with maxLength 5000*2];

The above example does this: Beginning with the Segment<char> (or Span<char> or Array or List<char>) named "mySegment", make a sub-segment of mySegment (alternatively an Extent alone) that begins at offset "myOffset" from the start, and continues until an element (char) is a vertical whitespace character and does not equal "paragraph separator" (U+2029), but do not extend the Extent or Segment any further than "myLimitOffset", and limit Extent.Length to 5000*2.

If you want to get the Extent (offset and length) alone without making a new Segment<T> corresponding to that Extent, then you can write:

Extent x = [mySegment at myOffset from start until element is CharMatchers.VerticalWhiteSpace 
     && element != '\u2029' with fence myLimitOffset with maxLength 5000*2];

Relative offsets: The "from" keyword makes an offset relative to something else. The default is "from start" thus writing "myOffset from start" is the same as writing simply "myOffset" without any "from" keyword. Supported "from" keywords include:

    100 from start
    100 from end
    100 from myPositionX forwards
    100 from myPositionX backwards
    100 from myPositionX backwards for 300+10
    100 from last '%' backwards
    100 from first CharMatchers.UppercaseLetter forwards
    100 from first CharMatchers.UppercaseLetter || CharMatchers.WhiteSpace forwards

The "for" keyword makes an Extent. For example, 100 for 20 means Extent.Offset = 100 and Extent.Length = 20.
100 from myPositionX backwards for 300+10 means the Extent.Length is 300+10, and the Extent.Offset is calculated like this: Begin at offset myPositionX then slide the offset backwards by 100. The calculation is simply:

    Extent.Offset = myPositionX - 100;
    Extent.Length = 300+10;

Supported "with" keywords include:

    with fence myLimitOffset
    with maxLength myMaxLength
    with minLength myMinLength
    with length myLength    // maybe, see also "for".
    with direction backwards
    with direction forwards

CharMatchers.VerticalWhiteSpace is an immutable singleton class that implements IElementMatcher<char>. Anyone can write their own classes that implement IElementMatcher<T> and match elements using any criteria desired.

Naturally if you want the opposite of a particular matcher, you can use the logical-not operator with the "is" operator:

element is !CharMatchers.Alphabetic && element is !CharMatchers.WhiteSpace

Without using a matcher, the following example begins at offset "myStartOffset" and continues until an element (char) is the carriage-return character or the newline character:

Segment<char> resultSegment = mySegment[myStartOffset until element == '\r' || element == '\n'];
// Or if you only want the Extent:
Extent x = [mySegment at myStartOffset until element == '\r' || element == '\n'];

"while" keyword

Note the difference between the "until" and "while" keywords. The following example begins at "myStartOffset" and continues while the elements are decimal digits or underscore:

Segment<char> resultSegment = mySegment[myStartOffset while element is CharMatchers.DecimalDigit || element == '_'];
// Or if you only want the Extent:
Extent x = [mySegment at myStartOffset while element is CharMatchers.DecimalDigit || element == '_'];

"where" keyword

The following loop iterates through each segment/run of consecutive whitespace characters:

foreach (Segment<char> whitespaceSeg in mySegment[myStartOffset where element is CharMatchers.WhiteSpace])
{ ... }

The foreach loops functions with the "where" keyword because "where" produces IEnumerable<Segment<T>>:

IEnumerable<Segment<char>> results = mySegment[myStartOffset where element is CharMatchers.WhiteSpace];

Note that the returned IEnumerable instance is not a pre-generated array, rather it produces its enumerable elements on-demand like how System.Linq.Enumerable does. This can be implemented using the preexisting yield return feature of C#. Also on the topic of efficiency, note that Segment<T> is a struct, thus the where keyword does not create thousands of objects in the heap.

TO DO: The "where" keyword gives you the matching segments. Decide whether to make an option or variation of "where" that gives you both the matching and non-matching segments.

Moving Average

A "moving average" example was provided on the following webpage, but the version there as of 2019/05/07 looks defective unfortunately. https://docs.microsoft.com/en-us/dotnet/csharp/tutorials/ranges-indexes Many elements are missed/ignored in that moving average because it does start += 100" yet "Range r = start..start+10; Furthermore, it has a function named "MovingAverage" that doesn't return a MovingAverage at all, surprisingly. It is named MovingAverage but it returns the same as System.Linq.Enumerable.Average.

The following design of MovingAverage attempts to eliminate the defects. In addition, it is better because it smooths-out the average by overlapping the segments.

using System.Linq.Enumerable;
static IEnumerable<double> MovingAverage(Segment<int> inData)
{
    foreach (Segment<int> seg in inData.CutAndSplice(inPieceLength: 20, inOverlapLength: 10))
    {
        yield return seg.Average();
    }
}

See also: https://en.wikipedia.org/wiki/Moving_average

Factionalize

The "factionalize" operation splits a segment into a sequence of "factionalized segments". A "factionalized segment" is a Segment<T> with a faction value assigned. For example, factionalization could be used as a stage in syntax parsing of text that is formatted to obey particular syntax rules. Example:

enum MyFaction { ... }
Segment<char> mySegment = ...;
foreach (FactionalizedSegment<char,MyFaction> in mySegment.Factionalize<MyFaction>(MyFactionalizer))
{ ... }

The foreach loops functions with Segment<T>.Factionalize because Factionalize returns IEnumerable<FactionalizedSegment<TElement,TFaction>>.

If Factionalize is used as a stage in syntax parsing of a programming, scripting, or data language, then TFaction could be defined as an enumeration like this:

enum TokenKind
{
    None = 0,
    WhiteSpace,
    Number,
    Word,
    Keyword,
    PlusSymbol,
    MinusSymbol,
    MultiplySymbol,
    ...
}

An instance of the delegate DlgFactionalizer is given to the Segment<T>.Factionalize method. See definition of this delegate within the .cs file.

Other slicing operations

See the .cs file for other slicing operations in Segment<T> such as:

public IEnumerable<Segment<T>> CutUp(int inPieceLength)
public static IEnumerable<Segment<T>> operator / (Segment<T> inSegment, int inPieceLength)
public IEnumerable<Segment<T>> CutAndSplice(int inPieceLength, int inOverlapLength)
public IEnumerable<Segment<T>> Dice(int inPieceLength, int inSkipLength, bool inInvert = false)
public IEnumerable<Segment<T>> Split(DlgSegmentSplitter<T> inSplitter)
public static IEnumerable<Segment<T>> Interleave(int inLength, params Segment<T>[] inSegments)
public Segment<T> ChopOffStart(int inPieceLength)
public Segment<T> ChopOffEnd(int inPieceLength)
public Segment<T> CleaveStart(int inPieceLength)
public Segment<T> CleaveEnd(int inPieceLength)
public void SeverStart(int inPieceLength, out Segment<T> outStartPiece, out Segment<T> outRemainder)
public void SeverEnd(int inPieceLength, out Segment<T> outEndPiece, out Segment<T> outRemainder)
public void Sever(Extent inExtent, out Segment<T> outStart, out Segment<T> outMiddle, out Segment<T> outEnd)
public (Segment<T> partA, Segment<T> partB) Bisect(int inOffset)
public IEnumerable<FactionalizedSegment<T,TFaction>> Factionalize<TFaction>(DlgFactionalizer<T,TFaction> inFactionalizer)
public Segment<T> Truncate(int inMaxLength)
public Segment<T> GetExtent(Extent inExtent)
public Segment<T> GetExtent(int inOffset, int inLength)
public Segment<T> GetExtentToEnd(int inStartOffset)
public Segment<T> GetExtentToStart(int inOffset)
public Segment<T> GetExtentAtStart(int inLength)  // might be renamed to GetPrefix
public Segment<T> GetExtentAtEnd(int inLength)    // might be renamed to GetSuffix
public Segment<T> GetRange(int inStartOffset, int inEndOffset)
public Segment<T> GetRangeInclusive(int inStartOffset, int inEndOffset)

Syntax for basic extents and ranges

Basic examples following (basic meaning not using the until, while keywords etc).

Extent x = [20 for 30];   // makes Extent.Offset=20, Extent.Length=30
Extent rangeA = [20..50]; // makes Extent.Offset=20, Extent.EndOffset=50
Extent rangeToEnd = [20..];
Extent rangeToStart = [..20];
Extent singleElement = [5];  // makes Extent.Offset=5, Extent.Length=1 or maybe Length=0
Extent negativeOrdinal = -[5]; // Extent.Offset == DataLength - 5
Extent reverseOrdinal = ~[5];  // makes 0-based ordinal in reverse, as if System.Array.Reverse was executed.
Extent rangeB = [20 .. -[5]];  // Extent.EndOffset == DataLength - 5

The following are the same, which means that -[5] actually invokes the operator - (Extent) overload method that is defined in struct Extent.

Extent negativeOrdinal = -[5];
Extent negativeOrdinal = -(new Extent(5, 1));

Likewise ~[5] invokes the operator ~ (Extent) overload method that is defined in struct Extent.

Iteration in reverse order

Example:

Segment<char> seg = ...;
int len = seg.Length;
for (int i = 0; i < len; i++)
{
    char ch = seg.InReverse[i];
    // Also supported:
    char ch = seg.GetItemReverse(i);
}

Builder

See Segment<T>.Builder in the .cs file.

https://github.com/verelpode/Advanced-Extents-Segments

[HaloFour]

What makes a Segment<T> or Extent<T> different from a Span<T>? Most of this seems like an API that could be furnished by a library. The proposed language changes seem ... odd. A reinvention of LINQ but specific for finding an offset? That also feels like it could be easily furnished by an API using fluent methods rather than so many permutations of combining keywords.

[verelpode]

@HaloFour

What makes a Segment<T> or Extent<T> different from a Span<T>?

Extent (non-generic) is instead of System.Range, thus Extent and System.Range don't contain any array elements whereas Span<T> does point to elements. Segment<T> is the one that is comparable with Span<T>. The difference is that Segment<T> includes much more functionality than Span<T> but I'd also be happy if the final design is to add the new functionality to Span<T> and/or System.Memory<T> rather than creating Segment<T> separately. However for the purposes of compiling and testing the .cs file, it is easier to test it as struct Segment<T> independent of Span<T>.

Most of this seems like an API that could be furnished by a library. ... That also feels like it could be easily furnished by an API using fluent methods rather than so many permutations of combining keywords.

Yes as I said in a previous message: "I'm unconvinced that it should be a language feature"

However language/syntax ideas were specifically requested of me, so that's what I did. For example, @CyrusNajmabadi requested "explanation of what that means at a language level". Also we're discussing this in dotnet/csharplang therefore the proposal should include language/syntax ideas at least for brainstorming and inspiration purposes, regardless of whether some of those language features are deleted or modified before the final release.

The proposed language changes seem ... odd.

As you said in your previous message: "They'll learn it once and move on. Any new feature requires getting used to."

A reinvention of LINQ but specific for finding an offset?

Linq is part of the C# language. Linq is not a separate programming language. In fact, Linq is an acronym for "Language Integrated Query", and what language is it integrated into? C# ofcourse. Linq is C#. Thus it makes perfect sense for my proposal to remain consistent with C# by reusing preexisting keywords in C#. Why should I invent new different keywords when the preexisting keywords in C# are already good? The similarity with Linq is an advantage of my proposal, not at all something strange.

[CyrusNajmabadi]

This seems incredibly esoteric and niche to be at the language level. I second the question about why this simply isn't an API. The examples shown don't seem to be compelling enough to warrant a language change here...

[CyrusNajmabadi]

Linq is part of the C# language. Linq is not a separate programming language.

Linq introduces patterns and practices that are felt to be wildly applicable to large swaths of programming domains. I'm not seeing that here. You're introducing a complex and esoteric syntax... for a use case that i can't recall a single person before you asking for.

Is this modeled after some existing successful system? Why would this particular API warrant such a large language change, given the limited use and applicability to the ecosystem?

[CyrusNajmabadi]

Also... i see nothing here that would have any impact on the existing Range/Slicing feature. You're proposing an altogether different sort of API with different language integration that seems entirely unrelated. So i don't see anything that would warrant not doing the existing range feature just as it's being planned today.

i.e. ranges are well served by syntax like a..b. Even if i thought that something to help with your 'extent' domain was warranted in the language (which i don't), it would be simple to just pick a different syntax here instead of using the simple and easy to grok range syntax.

[CyrusNajmabadi]

I'm not seeing how this really helps at a language level. You could just write:

// Your approach:
Segment<char> resultSegment = mySegment[myOffset from start until element is CharMatchers.VerticalWhiteSpace  && element != '\u2029' with fence myLimitOffset with maxLength 5000*2];

// Simple flexible API approach:
Segment<char> resultSegment = GetSubSpan(mySegment, from: start, until: e => CharMatchers.VerticalWhiteSpace(e) && e != '\u2029', fence: myLimitOffset, maxLength: 5000*2)

The API approach is actually simpler and clearer (imo). You have your domain specific API and your domain specific concepts. You can expand on it as much as you want with whatever operators are appropriate (sounds like you have a lot).

[HaloFour]

@verelpode

The difference is that Segment<T> includes much more functionality than Span<T> but I'd also be happy if the final design is to add the new functionality to Span<T> and/or System.Memory<T> rather than creating Segment<T> separately.

Seems like this could all easily be extension methods for Span<T> and for IEnumerable<T> in general. You could write them and publish them to NuGet.

I'm unconvinced that it should be a language feature

The LDM's default position is to not add features. Ranges had to meet a fairly high bar to get as far as it did, as it seems as far as the LDM is concerned it is worthy of being a language feature.

[verelpode]

@CyrusNajmabadi

I second the question about why this simply isn't an API.

And I'll third the question. Yes, some features are better when they are simply methods in a class in a library. I mentioned this previously. On the other hand, some features are good to integrate into the language, thus it's a judgement call as always.

As for the connection with Linq, I'm not the person who started that. Linq is already used in the current version of the C# 8.0 Range feature. For example, the C# 8 "Indices and ranges" tutorial uses System.Linq.Enumerable.Range and System.Linq.Enumerable.Average.

You're introducing a complex and esoteric syntax...

Well which is it? On one hand, I'm hearing that its Linq-like, and on the other hand, I'm hearing that it's "esoteric", but it can't be both of those, so therefore I ask: Well which is it? If you'd like to increase the consistency with C# 7.x by modifying my proposal to bring the syntax even closer to Linq, then I'd be happy to go along with that.

for a use case that i can't recall a single person before you asking for.

I'm surprised to read that, because I thought about which example to place as the very first example in my proposal, and I chose that one specifically for the reason that it's a common operation. So I'm quite surprised to read your message describing it as the exact opposite.

To be precise, ofcourse it's not common if you use it exactly as per my example without changing any of the parameters. But if you do change the parameters and change the match criteria to a different expression, then the pattern is a common pattern in programming. The one specific example is uncommon but the general pattern is common.

i see nothing here that would have any impact on the existing Range/Slicing feature.

The syntax in my proposal is different and incompatible with the syntax in the current C# 8 Range proposal. One example of the multiple differences/incompatibilities is that the C# 8 Range proposal says...

Range r = 20 .. ^5:

whereas my proposal says the same thing would use this syntax:

Extent x = [20 .. -[5]];

For multiple reasons, my proposal impacts in breaking ways on the current version of the C# 8 Range proposal.

[CyrusNajmabadi]

However language/syntax ideas were specifically requested of me, so that's what I did. For example, @CyrusNajmabadi requested "explanation of what that means at a language level".

It was requested of you not to say that it was good to have syntax changes. But because it sounded like you wanted syntax changes, and it was really impossible to tell what you were asking for. I was thinking you wanted some small tweak. I didn't realize you wanted to make a new large subdialect in the language for this small API domain.

Note that the burden here to get this cahnge is enormous, as per our previous conversation:

1. you'd have to demonstrate that this would be quite useful for a wide segment of users (i.e. >50%).
2. you could say this was beneficial for a smaller group. But then you'd have to show that group is SOL with existing solutions and this approach at hte language is fundamentally necessary for them to be effective.

I think showing either is going to be very difficult to do. For one, i'm not sure i can see even 5% of users using this new feature. Second, even if there was a group that would find 'extent-based programming' useful, they'd be served completely well by just doing this as a library/API as i showed above (and not requiring an actual language change).

Note that the slice/range feature was felt to hit '1'. This was based on actually examining codebases out there and dealing with the consistent and steady feedback from many groups across many domains that this would be useful.

[CyrusNajmabadi]

Well which is it? On one hand, I'm hearing that its Linq-like,

Linq itself was, and remains to be, highly esoteric. But we accepted that because of the enormous value it presented to the ecosystem.

and on the other hand, I'm hearing that it's "esoteric", but it can't be both of those,

Yes it can. Note that the language has never since taken on anything 'linq-like'. That's very intentional. Nothing has been able to show it warrants the type of approach we took there.

If you could somehow demonstrate value on the level of, or higher than, Linq, then we could potentially justify another esoteric embedded API domain language like this. But it would be a very tall order.

[CyrusNajmabadi]

To be precise, ofcourse it's not common if you use it exactly as per my example without changing any of the parameters. But if you do change the parameters and change the match criteria to a different expression, then the pattern is a common pattern in programming. The one specific example is uncommon but the general pattern is common.

Can you summarize what you think "the general pattern" is here that you're trying to design an API/langauge-feature around? It sounds like a general "allow me to to compute start/end points of my range using combinators", but i that's not right, please let me know. Thanks!

[CyrusNajmabadi]

The syntax in my proposal is different and incompatible with the syntax in the current C# 8 Range proposal. One example of the multiple differences/incompatibilities is that the C# 8 Range proposal says...

So don't use .. Now you're no longer conflicting with Ranges. Why is it so imperative for your feature that it must use ..? Why can't it use some other syntactic form that hasn't been used? Thanks!

[CyrusNajmabadi]

Also, @verelpode Have you considered hashing some of this out on gitter? it might be easier to converse on it, rather than the async nature of all these github messages.

[verelpode]

@HaloFour

Ranges had to meet a fairly high bar to get as far as it did,

Normally the C# team does have high bars and high standards, but this Range feature is an exception because it is unusually sloppy, unusually poorly described, and unusually poorly justified, and unusually poorly accompanied by practical examples. For example, the provided "moving average" example is terribly sloppy, so bad that it should actually be deleted or rewritten. In fact, I don't think I've ever seen such a sloppy example coming from the C# team. Normally they have high standards and do impressive work. I don't know how this mistake happened, but everyone makes mistakes from time to time, so it's no big deal. Mistakes are only bad if they remain uncorrected. I know I'm also guilty of sloppy work on a few past occasions, but I went back and fixed my mistakes and ended up proud of my work.

[CyrusNajmabadi]

because it is unusually sloppy

What is sloppy about ranges/slices?

unusually poorly described

What is poorly described?

and unusually poorly justified

I have no idea waht this means. What are you looking for in a justification?

and unusually poorly accompanied by practical examples

it literally hasn't shipped yet.

For example, the provided "moving average" example is terribly sloppy, so bad that it should actually be deleted or rewritten. In fact, I don't think I've ever seen such a sloppy example coming from the C# team.

What on earth does this sample have to do with the language feature? Why do you think it's something the C# team wrote?

Mistakes are only bad if they remain uncorrected.

Even if tehre is a problem with this sample, that has nothing to do with any sort of claim that there's a problem with the actual feature.

[verelpode]

@CyrusNajmabadi

you'd have to demonstrate that this would be quite useful for a wide segment of users (i.e. >50%).

My proposal is relevant for 90% of software engineers. On second thought, it's probably fair to say that my proposal is relevant to 99% of software engineers because a software engineer who doesn't use extents just doesn't exist. OK maybe in Antarctica there exists a software engineer who doesn't use extents, but 99% of software engineers in the world use extents, although with different names for this pattern. "Extent" isn't a common name but the pattern is extremely common. The .NET Framework contains hundreds of usages of extents. Here's just a few of the many examples:

System.String.Substring(int startIndex, int length)
System.Array.Copy(Array sourceArray, int sourceIndex, Array destinationArray, int destinationIndex, int length)
System.Array.Clear(Array array, int index, int length)
System.Array.IndexOf<T>(T[ ] array, T value, int startIndex, int count)
System.Array.BinarySearch<T>(T[ ] array, int index, int length, T value)
.....and many more.....

Can you summarize what you think "the general pattern" is here

Another general pattern is the element matcher. For example, open the Object Browser window in Visual Studio, and set it to browse .NET Framework 4.7.2, and then search for "TryParse". Many examples appear for "TryParse", and that's not the only thing. Many other examples of this general pattern exist in real-world software such as the .NET Framework / Core.

[CyrusNajmabadi]

@verelpode You realize that this feature hasn't even been completed, let alone shipped. By what basis are you judging how the feature is coming along versus other features in the past? I can't recall any sort of approach with features in the last 5 years that did things differently here.

Looking at teh C# proposals, this seems in line with the rest: https://github.com/dotnet/csharplang/tree/master/proposals/csharp-8.0

[CyrusNajmabadi]

Here's just a few of the many examples:

All of your examples are served fine by ranges... so i'm not sure what the problem is... :)

You're proposing a vast API here and a large set of language changes to integrate them into the language. The onus is on you to demonstrate that this will provide clear and unambiguous value to the greater C# ecosystem.

[CyrusNajmabadi]

Another general pattern is the element matcher. For example, open the Object Browser window in Visual Studio, and set it to browse .NET Framework 4.7.2, and then search for "TryParse". Many examples appear for "TryParse", and that's not the only thing. Many other examples of this general pattern exist in real-world software such as the .NET Framework / Core.

I don't see how your 'extent' language feature helps out all for any of the TryXXX apis. Can you clarify how those would benefit?

[verelpode]

Why do you think it's something the C# team wrote?

OK, well, I'm sorry, I didn't know that you wrote it. But it's no big deal. I've read multiple messages from you in various other threads and I found those comments good and helpful. Certainly I also produce sloppy work on occasions, but I go back and fix it.

[CyrusNajmabadi]

My proposal is relevant for 90% of software engineers. On second thought, it's probably fair to say that my proposal is relevant to 99% of software engineers because a software engineer who doesn't use extents just doesn't exist. OK maybe in Antarctica there exists a software engineer who doesn't use extents, but 99% of software engineers in the world use extents, although with different names for this pattern.

I honestly don't know how you're going from the idea that programmers use some APIs that specify a 'start+length' pair into thus, there should be this extremely large API and assorted helper functions provided here *and* that API should be mapped into C# with a large embedded DSL.

Do you see the leap you're making, and how you haven't justified it? It's like saying developers use integers, therefor it's necessary for C# to have a symbolic mathematical system built into it a-la Mathematica.

You're making a logical leap that you're not supporting.

[CyrusNajmabadi]

OK, well, I'm sorry, I didn't know that you wrote it. But it's no big deal. I've read multiple messages from you in various other threads and I found those comments good and helpful. Certainly I also produce sloppy work on occasions, but I go back and fix it.

Again, i don't have any idea who wrote this. Most likely it was the MSDN docs team. You're welcome to give them feedback that the example shoudl be improved (the page itself allows you to give feedback). I don't believe the C# team writes these docs.

However, regardless, i don't see what that has to do with the actual language feature itself. Even if this one example page is bad, that has no bearing on the language feature itself and whether or not it has been properly designed and is the right feature to ship.

[CyrusNajmabadi]

@verelpode As before, gitter is likely a better medium for this. However, i strongly recommend you read: https://blogs.msdn.microsoft.com/ericgu/2004/01/12/minus-100-points/

The onus on someone proposing a language change (especially one as large and esoteric as this) is to provide an enormous amount of evidence to justify the value of the cahnge. Language changes start in a position of being assumed to be a negative. So, in order to be done, they not only have to outweigh their substantial initial negativeness, but they also have to demonstrate a high enough value to warrant doing ahead of everything else that can be done.

In your case, you've stated you want this very large feature, but you haven't demonstrated at all why it's appropriate to exist in the language, or why it's even something that valuable to the C# ecosystem as a whole.

I'm a compiler writer (among many other things), and i'm commonly working with streams of text. And i can tell you that i can only think of a tiny handful of cases this would be helpful. And that means adding a massive feature for niche applicability. On the other hand, if you wanted to put this in a Nuget package, i think you might get some adoption. THat would likely help drive a more focused API that might eventually have some impact back on the language itself.

[CyrusNajmabadi]

///  The .NET Framework contains hundreds of usages of extents. Here's just a few of the many examples:

System.String.Substring(int startIndex, int length)
System.Array.Copy(Array sourceArray, int sourceIndex, Array destinationArray, int destinationIndex, int length)
System.Array.Clear(Array array, int index, int length)
System.Array.IndexOf<T>(T[ ] array, T value, int startIndex, int count)
System.Array.BinarySearch<T>(T[ ] array, int index, int length, T value)
.....and many more.....

All of these can be suitably handled well with Ranges. Indeed, they were many of the motivating examples for Range by both the C# LDM and CoreFx. Both sides felt that Range was a suitable abstraction here. i.e. String.Substring(Range) would work (along with just having a slicing indexer). The same would hold for the rest of those.

So the above justification really does not help your case. Because it's already believed to be the case that the Range/Slice feature is a very good solution here, and your proposal hasn't indicated why it is not, or why your proposal is substantively more desirable.

--

Note: as before, i recommend proceeding on gitter.

[verelpode]

I've made a significant effort to accommodate both of you (even though you're not MS staff members), so it would be nice if the favor is returned. Originally I didn't want the Pascal ordinals in the proposal at all, but to accommodate you two highly vocal participants, I've gone and done it anyway -- for you. The Pascal ordinals are included in my proposal document against my wish, in an attempt to satisfy you guys. I searched for a win-win solution and the best I could find is this syntax with the arity-1 negation operator with square brackets:

10 .. -[5]

I'm so accommodating and open-minded and team-friendly that I've included it in the proposal despite the fact that I still think it's complete nonsense to make C# 8 use both C and Pascal ordinals.

Originally I said that I'm unconvinced that the C# 8 Ranges should be a language feature as opposed to API. But you guys wanted language features super strongly, so I went and put it in the proposal to the best of my ability, in an attempt to accommodate you guys.

I'm searching for win-win solutions. Are you?

[verelpode]

The onus on someone proposing a language change (especially one as large and esoteric as this) is to provide an enormous amount of evidence to justify the value of the cahnge.

Exactly, and that's the huge problem here, because that principle has been followed for other C# features but not for the Range feature. Somehow the Range feature has become an emotional religious-style issue. Emotional decisions are inappropriate in the design of C# 8.

[svick]

@verelpode

But you guys wanted language features super strongly

I believe people asked about language features, because discussing language features is the purpose of this repository and so they assumed that if you want to discuss something, it's a language feature.

I understand that you don't like Ranges. That's unfortunate, but no feature can satisfy everyone. You also think that Advanced Extents & Segments would be valuable, but it seems you don't think it has to be a language feature. But doesn't that mean you can create what you want as an API and ignore Ranges in your own code? Isn't that win-win, or at least as close to win-win we can get when people have different opinions?

[HaloFour]

@verelpode

You also explicitly stated that this proposal would be incompatible with ranges, so I assumed that you had an alternate syntax in mind. But I don't see that as being the case here. In fact, I think that these APIs would benefit from ranges, especially where it comes to calculating extents.

[verelpode]

@svick

it seems you don't think it has to be a language feature. But doesn't that mean you can create what you want as an API and ignore Ranges in your own code?

Maybe but it's too early to determine that. Some parts of my proposal may well be justifiable as language features, either in their current form or in a tweaked form.

System.Range can be used as an API instead of a language feature. It may well function better as an API. For example, Range.All is better than writing 0..^0 and Index.End is better than ^0 in my opinion. It is important for every new language feature to begin at minus 100 points and then work its way up. Somehow this minus 100 points procedure has been circumvented or bypassed for the Range feature and instead it has become an emotionally-charged debate. I also feel concerned about the possibility of fanaticism/fanboyism creeping into the decision-making process. I don't know how this happened, but it would be sensible to wait until the emotions subside. Defer the feature to C# 8.1 and then try to make a cold logical non-emotional analysis of the pros and cons of the proposal and the alternative proposals.

A large amount of evidence to justify the value of the language change is needed. In particular, multiple real-world practical non-academic examples that demonstrate that the new syntax is significantly superior than the previous way of writing the same thing. Credible justifications of why the Range feature should be a syntax and not an API. Multiple real examples demonstrating practical advantages in real-life scenarios, not emotions, and not sloppy examples like the broken "MovingAverage" example that was provided.

[verelpode]

@HaloFour

I assumed that you had an alternate syntax in mind. But I don't see that as being the case here.

Have a look at the first section of my proposal for several examples of the alternate syntax. See also the section titled "Syntax for basic extents and ranges" for further examples of the alternate syntax.

[HaloFour]

@verelpode

It is important for every new language feature to begin at minus 100 points and then work its way up. Somehow this minus 100 points procedure has been circumvented or bypassed for the Range feature and instead it has become an emotionally-charged debate.

No, it hasn't. You're just not the score keeper. The people that are the scope keepers disagree with you.

As for the charged emotions, I totally agree. I suggest that you calm down before you continue to make such accusations. Stop taking the reality that some people disagree with you so personally. It's going to happen, especially when it comes to language design.

[HaloFour]

@verelpode

Have a look at the first section of my proposal for several examples of the alternate syntax. See also the section titled "Syntax for basic extents and ranges" for further examples of the alternate syntax.

What I see there is a proposal for an alternative form of reverse indexing using -[x] vs ^x, but I don't see much in the way of justification as to why that's better. And you not liking ^ doesn't make it better.

[HaloFour]

@verelpode

but it would be sensible to wait until the emotions subside.

I'd also like to point out that the language team has been discussing this feature for over four years now:

https://github.com/dotnet/roslyn/issues/98 https://github.com/dotnet/roslyn/issues/120

Nobody is trying to cram this into C#. It has been carefully considered, probably longer than many of the smaller features added to the language.

[CyrusNajmabadi]

It is important for every new language feature to begin at minus 100 points and then work its way up. Somehow this minus 100 points procedure has been circumvented or bypassed for the Range

This feature was one we worked on for years. It got tons of thought and had many proposals and designs we examined and tried out.

and instead it has become an emotionally-charged debate.

It really hasnt Maybe that's just your own feelings here?

I also feel concerned about the possibility of fanaticism/fanboyism creeping into the decision-making process.

I was part of this decision making process. We examined the options, weighed then out, discussed things heavily, and then made an informed decision about the direction we wanted to take.

try to make a cold logical non-emotional analysis of the pros and cons of the proposal and the alternative proposals.

I can't promise to was done without bias. But this is what was done. We spent years on this and tried many approaches exploring many codebases and domains. This was the result that had the best balance of pros and cons for all of us.

[CyrusNajmabadi]

Originally I didn't want the Pascal ordinals in the proposal at all, but to accommodate you two highly vocal participants, I've gone and done it anyway -- for you. The Pascal ordinals are included in my proposal document against my wish, in an attempt to satisfy you guys.

Fwiw, I didn't ask for this (and I don't believe Halo asked either)

You were the one complaining about the language feature, and stating you wanted something else. I thought that meant you wanted a different language feature, do that's why I asked to find out what you wanted.

If you just want an API, then just write the API. It sounds like there's no problem having ranges and your feature.

[CyrusNajmabadi]

I'm searching for win-win solutions. Are you?

I think the current approach is already a win. I'm happy to help you explore this space. But perhaps gitter would be a better venue as this is much more explorative.

[CyrusNajmabadi]

System.Range can be used as an API instead of a language feature.

Yes. We considered that. We felt the feature was valuable enough to get dedicated syntax. It has more than enough value to make it over the -100 hump. As one of the people who decided on this, this was an easy choice.

My personal expectation is that this will get 90%+ adoption in codebases that are being maintained. So, it's a small improvement to the language, but with broad value to the ecosystem as a whole

[CyrusNajmabadi]

A large amount of evidence to justify the value of the language change is needed.

That evidence was presented during the years where we were designing this. This came from our own experience, corefx, many API groups, and many partner groups (internal and external)

In particular, multiple real-world practical non-academic examples that demonstrate that the new syntax is significantly superior than the previous way of writing the same thing.

We did all that. It met our needs for finding this feature valuable enough.

Credible justifications of why the Range feature should be a syntax and not an API.

We did that. That's why we accepted the feature.

and not sloppy examples like the broken "MovingAverage" example that was provided.

You are not owed anything. I get that you don't like that made doc. So take it up with them. That's not the Roslyn team. The ldm was more than satisfied with all the examples and use cases.

The feature hasnt even shipped, but you're complaining about things that generally come later (i.e. at launch)

[CyrusNajmabadi]

Finally: this isn't a democracy, and no one is really going to be spending any effort to make you happy or meet your demands. If you want anything to change, your own chance is to present such a massively compelling argument and proposal to turn some heads.

No one is really going to work on convincing you of the current POR. It's already the direction the team is taking. If you want that direction to change, that will only happen through your own legwork, people won't do it for you.

[verelpode]

@CyrusNajmabadi

this isn't a democracy

Right, this isn't a democracy. As jmn2 said, it's a benevolent dictatorship. Thus in the end, if the Microsoft staff members decide against your wish, then you'll need to learn to accept it and move on, regardless of whether you like or dislike the final decision, regardless of whether you agree or disagree with it. That's the nature of it. But it's not the end of the world if this happens. You can still release your baby/project as a Nuget package. The same applies for me.

[verelpode]

@CyrusNajmabadi

You are not owed anything.

I never said anything like that, but now that you mention it, Microsoft does clearly owe my employer something in return for the money that my employer pays to Microsoft every month of every year. I think you might not have realized that Microsoft and the C# team do not owe you anything in return for your unpaid participation in this project. It was your own free choice to participate without a salary. Nobody forced you to do it. Microsoft does not owe you any ownership nor leadership of C#, Roslyn etc. It's not your baby now. You have no rights to Roslyn etc. If you wanted something, then you needed to get an agreement before you participated, not afterwards. You are not owed anything.

Could you please refrain from posting so many off-topic emotional comments in this thread? The topic of this thread is: "Advanced Extents & Segments Proposal re C# 8.0 Ranges". Too many of your comments are unrelated to the topic of extents/segments, or only superficially related but not truly. A lot of your comments are fundamentally on the topic of yourself (the comments where you describe your own behavior), but this thread isn't for the purpose of discussing YOU. This thread is for the purpose of discussing extents, ranges, segments. Please stay on-topic. Thank you for your consideration.

[HaloFour]

@verelpode

Microsoft does clearly owe my employer something in return for the money that my employer pays to Microsoft every month of every year.

They got the product they paid for, just like everyone else.

Microsoft does not owe you any ownership nor leadership of C#

No, they don't owe him anything, but as a previous member and as continuing contributor they do have a great deal of respect for his input. Infinitely moreso than yours.

Could you please refrain from posting so many off-topic emotional comments in this thread?

There is only one individual on this thread who is railing emotionally against everyone, and it's not Cyrus. You seem to have a serious axe to grind and want to have some special treatment to the evolution of the language. You're not going to get it. You're not going to derail four years of discussions with, "I don't wanna!", especially if your best argument is that the team is sloppy and emotionally-charged (which is an emotionally-charged accusation).

[verelpode]

@HaloFour As jmn2 said, this is tiring and absorbs a lot of time and energy, and creates frustrating repetition. I would enjoy reading your real opinions and I'd like to hear your critique, but you haven't given me your real opinions so far. Instead I've received many messages from you and CyrusNajmabadi that are expressions of your emotions instead of your analytical abilities. It would be helpful for every participant if you could please make an effort to keep your emotional attachment out of the design process and remain analytical, non-emotional, logical, and practical.

Maybe the name "Roslyn" was a mistake because it's a female name and thus makes it more difficult for males to remain non-emotional and to avoid engaging in imaginary wars over self-claimed turf.

[HaloFour]

@verelpode

Maybe the name "Roslyn" was a mistake because it's a female name and thus makes it more difficult for males to remain non-emotional and to avoid engaging in imaginary wars over self-claimed turf.

And you've just immediately and completely shut down any chance of having a proper discussion. Considering this my last response to you, ever. I'd suggest some serious introspection into this worldview you hold.

[verelpode]

@HaloFour @CyrusNajmabadi I want to read critique of my ideas but what you're engaging in is actually more like a form of harassment. I would appreciate it if you would read the Wikipedia article on the topic of Mobbing:

https://en.wikipedia.org/wiki/Mobbing

[CyrusNajmabadi]

I think you might not have realized that Microsoft and the C# team do not owe you anything in return for your unpaid participation in this project.

Here's the thing: I never have expected otherwise. I even said that explicitly. After I explained what you could expect from a proposal, I also said: "The same applies for me."

Feel free to take it to gitter if you want to converse more.

[CyrusNajmabadi]

I want to read critique of my ideas

Please read my posts. There were several critiques posted

[CyrusNajmabadi]

A lot of your comments are fundamentally on the topic of yourself (the comments where you describe your own behavior),

I was one of the people who designed the range/slice feature. I'm giving you insight into the thinking and process that went through it from my direct experience there.

I'm explaining what it takes to make a feature a reality to help you understand what is necessary on your side to do the same.

[CyrusNajmabadi]

FYI: Roslyn was named after https://en.m.wikipedia.org/wiki/Roslyn,_Washington

A town outside seattle (where MS is based)

[CyrusNajmabadi]

It would be helpful for every participant if you could please make an effort to keep your emotional attachment out of the design process and remain analytical, non-emotional, logical, and practical.

Sure. Feel free to respond to the direct critiques I've made of the proposal. There are several points that haven't been responded to.

That said, I would highly highly highly recommend taking this to gitter. It's just a better medium for hashing out ideas like this.