Span<T> etc. support

[dsyme]

I propose we make the necessary language changes to support Span

See https://github.com/Microsoft/visualfsharp/issues/4166#issuecomment-370047591

Interop:

C# 7.2: Ref extension methods on structs dotnet/csharplang#186 dotnet/roslyn#165

Safety:

C# 7.2: The "readonly references" feature is actually a group of features that leverage the efficiency of passing variables by reference, but without exposing the data to modifications. https://github.com/dotnet/csharplang/blob/master/proposals/csharp-7.2/readonly-ref.md

C# 7.2: Compile time enforcement of safety for ref-like types. The main reason for the additional safety rules when dealing with types like Span and ReadOnlySpan is that such types must be confined to the execution stack. https://github.com/dotnet/csharplang/blob/master/proposals/csharp-7.2/span-safety.md

Performance:

C# 7.2: https://github.com/dotnet/csharplang/blob/master/proposals/csharp-7.2/conditional-ref.md

Proposal: Struct Lambdas dotnet/csharplang#1060

Proposal: Static Delegates https://github.com/dotnet/csharplang/blob/master/proposals/static-delegates.md

Proposal: Blittable Types https://github.com/dotnet/csharplang/blob/master/proposals/blittable.md

[realvictorprm]

I'm absolutely in, now we need to figure out what the preferred way is to implement the necessary changes.

I already tried to implement an error if a method returns a span type but this would require that span somehow is in the standard library too.

Maybe we could add another special type kind just called "stack-only-type" because value types must not be always on stack (so far I understood, please correct me if I'm wrong).

[zpodlovics]

I did some digging to how the runtime implemented it, and what checks they use for type loading / instrisincs. It's probably worth checking and linking the Roslyn compiler checks too.

CoreCLR Span issue tracking: https://github.com/dotnet/coreclr/issues/5851

Representation:

    // ByReference<T> is meant to be used to represent "ref T" fields. It is working
    // around lack of first class support for byref fields in C# and IL. The JIT and 
    // type loader has special handling for it that turns it into a thin wrapper around ref T.
    [NonVersionable]
    internal ref struct ByReference<T>
    {
        private IntPtr _value;

        public ByReference(ref T value)
        {
            // Implemented as a JIT intrinsic - This default implementation is for 
            // completeness and to provide a concrete error if called via reflection
            // or if intrinsic is missed.
            throw new System.PlatformNotSupportedException();
        }

Source: https://github.com/dotnet/coreclr/blob/6c12105bb8cc1821ba5d5c3d36aad609a44308e0/src/mscorlib/src/System/ByReference.cs

JIT Intrinsics for ByReference/Span/ReadOnlySpan: https://github.com/dotnet/coreclr/blob/1a4a2d5b3121b6559dd864e15c9ae264d35b125a/src/vm/method.cpp#L2278

Fast Span:

    /// <summary>
    /// Span represents a contiguous region of arbitrary memory. Unlike arrays, it can point to either managed
    /// or native memory, or to memory allocated on the stack. It is type- and memory-safe.
    /// </summary>
    [DebuggerTypeProxy(typeof(SpanDebugView<>))]
    [DebuggerDisplay("{ToString(),raw}")]
    [NonVersionable]
    public readonly ref partial struct Span<T>
    {
        /// <summary>A byref or a native ptr.</summary>
        internal readonly ByReference<T> _pointer;
        /// <summary>The number of elements this Span contains.</summary>
#if PROJECTN
        [Bound]
#endif
        private readonly int _length;

Source: https://github.com/dotnet/coreclr/blob/master/src/mscorlib/shared/System/Span.Fast.cs

Span loader/jit checks: https://github.com/dotnet/coreclr/issues/8516 https://github.com/dotnet/coreclr/pull/9061 https://github.com/dotnet/coreclr/pull/15746

Span cannot be used for generics:

"Span is a stackonly (by-ref like) type and cannot be used as a generic as that would box it. If you need a type that needs to live on the heap, consider using Memory"

Source: https://github.com/dotnet/corefx/issues/25669#issuecomment-349147531

[realvictorprm]

@dsyme would this include support for readonly byref types?

[manofstick]

Can we have some integrated pattern matching support too?

I played a little with Span in C# a while back, looping on a continually shrinking by 1 Span (i.e. the equivalent of taking a tail of a list) and found very good performance.

So maybe something like (ug; running out of special characters.... not recommending [/ /] but () as [//] might get parser thinking its a comment without effort? but just as example where I can't think of something else good...)

match someSpan to
| [/ /] // match empty span
| [/ e0 /] // match length = 1
| [/ hd; (... as tail) /] // match length > 0; e0 binds to element 0, tail as Span (1 to Length-1)
| [/ (... as root); eN /] // match length > 0; eN binds to element Length-1, root as Span (0 to Length-2)
| [/ e0; (... as middle); eN /] // match length > 1; e0 to 0, eN to Length-1, middle as Span (1 to Length-2)

[haf]

What can be done from libraries, such as Logary and Suave to start using this type directly? What are the minimal requirements to use it? .Net 4.7.1? netstandard2.0? What are the library author recommendations to be future-proof? Also, is it already obsolete https://docs.microsoft.com/en-us/dotnet/api/system.span-1?view=netcore-2.0 ?

[TIHan]

@haf, in latest version of System.Memory, Span<'T> is marked obsolete which will prevent F# from compiling code that uses it because F# does not implement any rules surrounding Span<'T>. I'm guessing F# needs to opt out of the error in order just to compile successfully.

In order to use Span today, you need a lesser version of System.Memory that didn't add the Obsolete attribute.

[Rickasaurus]

Apparently this "Obsolete" stuff is some kind of protection for older compilers? https://twitter.com/James_M_South/status/983480529591791616

An additional measure will be taken to prevent the use of ref-like structs in compilers not familiar with the safety rules (this includes C# compilers prior to the one in which this feature is implemented).

Having no other good alternatives that work in old compilers without servicing, an Obsolete attribute with a known string will be added to all ref-like structs. Compilers that know how to use ref-like types will ignore this particular form of Obsolete

What an insane and horrible hack.

[zpodlovics]

As Span require full byref review (stack only refs), I would like to suggest to extend the byref review to readonly byrefs, in this way we could avoid defensive struct copy:

https://blogs.msdn.microsoft.com/mazhou/2018/01/08/c-7-series-part-8-in-parameters/

Right now this is an System.Runtime.CompilerServices.IsReadOnlyAttribute custom attribute for the arguments:

using System;

namespace inref
{
    class Program
    {
        public int Increment(in int value)
        {
            // Reassignment is not ok, "value" is passed by ref and read-only.
            int returnValue = value + 1;
            return returnValue;
        }

        static void Main(string[] args)
        {
            Console.WriteLine("Hello World!");
        }
    }
}

IL:

   .method public hidebysig 
           instance default int32 Increment (int32& 'value')  cil managed 
    {
    .param [1]
    .custom instance void [System.Runtime]System.Runtime.CompilerServices.IsReadOnlyAttribute::.ctor() =  (01 00 00 00 ) // ....

        // Method begins at RVA 0x2050
    // Code size 5 (0x5)
    .maxstack 8
    IL_0000:  ldarg.1 
    IL_0001:  ldind.i4 
    IL_0002:  ldc.i4.1 
    IL_0003:  add 
    IL_0004:  ret 
    } // end of method Program::Increment

I am not an expert of F# compiler internals, but I guess it would be as the ByRef path + additional restrictions due the read-only nature + some optimizations to eliminate "defensive" struct copying.

And here is the hard part, how it should be named? How about inbyref<'T> and robyref<'T> or readonlybyref<'T> for naming? Span will require some additional types too eg.: stack only types Span<'T>. How about using span<'T> for this purpose? And special types will needed for ref struct and readonly ref struct types too. The C# compiler also create these structs with System.Runtime.CompilerServices.IsByRefLikeAttribute.

https://blogs.msdn.microsoft.com/mazhou/2018/03/02/c-7-series-part-9-ref-structs/

How it should be named? How about [<StackOnlyStruct>] or [<ByRefStruct>] and [<ReadOnlyStackOnlyStruct>] or ```[] attribute?

[<ByRefStruct>]
type Point =
 val X: int
 val Y: int
 new(x,y) = {X=x;Y=y}

[<ReadOnlyByRefStruct>]
type Point =
 val X: int
 val Y: int
 new(x,y) = {X=x;Y=y}

vs

[<StackOnlyStruct>]
type Point =
 val X: int
 val Y: int
 new(x,y) = {X=x;Y=y}

[<ReadOnlyStackOnlyStruct>]
type Point =
 val X: int
 val Y: int
 new(x,y) = {X=x;Y=y}

A unified and terminology would be good to have for this, using similar terminology mapping where C# concepts mapped to F# use would be preferred (eg.: ref -> byref).

[zpodlovics]

Any plan how to keep up with C# improvements? I am afraid that at this rate of roslyn/coreclr progress the F# language would be unusable within a few years due the unsupported features required even for interop.

https://github.com/dotnet/roslyn/blob/master/docs/Language%20Feature%20Status.md

[ghost]

I think that while it is important to maintain compatibility with C# in terms of interop, we need to be careful to not just implement things because C# is doing it.

One of the big selling points of F# is that most of the system is explainable on a napkin. This becomes useful when you can keep an entire system written in F# in your head. As someone who has used F# for close to 7 years now, I fear that the language is starting to move away from its original simplicity (example, struct syntax for tuples, as if the compiler couldn't automatically optimise this allocation pattern), all for the optics of staying on par with C#.

It is fine to have competition, but it is a dangerous precedent we are setting where C# is the driver for language innovation (for a long time it was the other way around). I am not saying we shouldn't keep an eye on things that C# is doing well, we just need to pick the things that will benefit the F# ecosystem the most, and I have yet to see evidence that using read only stack based structs provides a language level benefit (no doubt it provides a framework (i.e. dotnet) benefit, but this shouldn't be something that concerns F# developers, outside the compiler and FCS) .

As a thought experiment, can anyone provide a demonstrable use case where these new C# language features would benefit either the F# compiler, or F# code?

[zpodlovics]

@abk-x Yes, F# should have simple concepts. However simple does not mean it's easy (because easy=familiar), simplicity comes from conceptual integrity: https://www.infoq.com/presentations/Simple-Made-Easy-QCon-London-2012

I would be happy to have more higher level concepts such as staging in F# something like [1] [2] that could provide domain specific high level of optimalizations ("abstraction without regret"), however we are not there yet. At the meantime the language still have to provide a somewhat competetive performance.

Here is a scientific/data processing task that should be trivial in F#: How fast you can process a simple 100GiB structured CSV file with F# (each fields must be strongly typed, straight line and singe line CSV, 100 fields with numeric types and/or codes, without any external library/type provider/native library)? How fast you can do simple query or aggregation with this data eg.: single field min/max/avg? You will immediately see how easilty heap allocation and GC could become a huge problem.

[1] https://infoscience.epfl.ch/record/180642/files/EPFL_TH5456.pdf [2] https://www.cs.purdue.edu/homes/rompf/papers/rompf-icfp15.pdf [3] https://youtube.com/watch?v=NTAJNYcsAEE

[zpodlovics]

@abk-x As F# already runs on different runtime or language environment eg: Fable - JS, Fez - BEAM, and other future runtimes or language it will be worth investigate what language features and how it can be implemented in different runtime or language environments. One possible option is to walk the Haxe way, it has a core library, that was implemented on every runtime or language [1], and specialized library for each runtime environment or language (C++,C#,Flash,HL,Java,JS,Lua,Neko,PHP, Python).

The more runtime environment and language the F# supports the more higher level and pure F# library will exists. At least this was the OCaml experience after the introduction of MirageOS (unikernel library operating system written mostly OCaml) [2][3]

[1] https://api.haxe.org/ [2] https://mirage.io [3] http://anil.recoil.org/papers/2015-usenixsec-nqsb.pdf

[ghost]

Playing the devils advocate here, I think while span offers some really awesome benefits, it needs to be implemented in F# with the least amount of user pains, we don't want another type like struct tuples that we need to be conscious of, it should be something that is behind the scenes and only used when required (like NativePtr).

To your point of parsing a big CSV file, I see no reason why the non-niave implementation should cause heap allocations at all? I admit I haven't tried this before :)

[<Struct>]
type CsvToken = { start : int; end':int } 

[<Struct>]
type CsvLine = { 
    tokens : nativeptr<CsvToken>
    Length : int
}

[<Struct>]
type CsvFile = {
   lines : nativeptr<CsvLine>
   Length : int
}

[ghost]

p.s. in case this didn't come across on the internet, @zpodlovics, you are really correct with your examples.

[ghost]

What would the compiler do in this scenario, would it throw a compile time error?:

[<StackOnlyStruct>]
type Point =
 val X: int
 val Y: Dictionary<a,b>
 new(x,y) = {X=x;Y=y}

[zpodlovics]

@abk-x It's interesting how the programmers understands the code, it's still an active research area, it would be also interesting the understanding of functional code compared to other paradigms: https://www.infosun.fim.uni-passau.de/cl/publications/docs/SKA+14.pdf

According to this, learning to implement a functional language have an excellent result of learning the paradigm:

"Although this language is minimal, our compiler generates as fast code as standard compilers like Objective Caml and GCC for several applications including ray tracing, written in the opti- mal style of each language implementation. Our primary purpose is education at undergraduate level to convince students—as well as average programmers—that functional languages are simple and efficient." https://esumii.github.io/min-caml/paper.pdf

How and why hiding the details will help the users? Using the same name for different things ("overloading concepts") will generate lot more confusion, and even more surprises when somebody use it in code.

Take a look at K language(from APL+LISP family of languages), they only have a few concept (but they use it for different things). However reading the code - unless you use it daily and you are familiar with the idioms - is a bit harder than F#, especially [3] [4]. As these examples only a few characters a new user - literally - can keep it in their minds.

[1] https://github.com/kevinlawler/kona/wiki/Tutorial [2] https://github.com/kevinlawler/kona/wiki/Project-Euler-Code-Golf [3] http://www.nsl.com/k/ray/raya.k [4] http://www.nsl.com/k/ray/rayq.k

[ghost]

@zpodlovics would you take a crack at the RFC for span<'a>? I am struggling to find any argument with many of your points.

[zpodlovics]

@abk-x If you have some spare time, you should really try out the 100GB CSV parse example, even with a simplified (10column, the number of columns and the content type is fixed) case. Please note columns in the CSV must have their own fixed number non-dynamic individual type in F# (eg.: Col1,Col2,Col3...). You are free to choose colum / row oriented approach, but they must be strongly typed too. You can even try out the plain tuple vs struct tuple approach. Unless you did similar things before, you'll be suprised with the results. Small things could matter a lot if you do it in billion times within a small timespan.

Example processing pipeline pseudocode (Seq could be replaced with other processing pipeline abstractions):

Row oriented:

readLines()
|> Seq.map lineToRow
|> Seq.mapFold f
|> Seq.filter filter

Column oriented (one column at a time):

readLines()
|> Seq.map lineToCol1
|> Seq.mapFold f
|> Seq.filter filter

[cartermp]

@abk-x I'll offer my philosophical stance on this, as I've mailed @dsyme about this earlier and offered a similar argument.

Fundamentally, F# is a .NET language. Fable and Fez make it run elsewhere, but they also inherit some ".NET-isms" by virtue of F# being a .NET language. Span<'T> is fundamentally a .NET concept. The motivations behind it were driven by the .NET runtime and ASP.NET teams to address performance concerns that they've had for a long time. It is an abstraction that is core to performance-oriented programming in .NET. The C# team was influential in the shape of the abstraction, but this is not a C# feature.

Span<'T> will be a part of .NET Standard 3.0, and is "a new .NET mainstay". For F# to continue successfully living atop .NET, it must adopt the features necessary for interoperation and writing of performance-oriented code that uses Span<'T>. These features may not be generally useful for most application developers (or even most F# developers at all), but they will be critical for library authors whose code requires better performance and developers of "middleware" or other components in the web server space.

[ghost]

@zpodlovics I only have time to complain about things on the internet, not actually write code :)

@cartermp fully understand, and since span is going to be part of netstandard 3, this is a very important thing for F#.

My worries with span:

Span originally required post compile IL rewrite (with ILSub) so that efficient code could be used, as opposed to what the C# compiler emitted at the time. When the corefx(lab?) took it over span from Joe Duffy, there was improvements made to the C# compiler to avoid this IL rewrite. The compiler changes came thick and fast, often the packages available in myget for span would be broken unless you had a specific C# compiler version (see https://github.com/dotnet/corefx/issues/25669). This is completely understandable as the library was in preview at the time, but I think it shows potential pain points for F#, F# has quite a checkered history in regards to FSharp.Core versioning (admittedly less an issue now that Core is in nuget).

Don't get me wrong, I think span support is very much needed at some point, I guess I am just curious to see how it can be done with minimal headache from the user perspective.

[zpodlovics]

Update: The C# -> F# and F# -> C# overload resolution question moved to https://github.com/fsharp/fslang-design/issues/287#issuecomment-390203219

[dsyme]

RFC is here: https://github.com/fsharp/fslang-design/blob/master/RFCs/FS-1053-span.md

@zpodlovics Please put discussion on the RFC thread or open a PR to the RFC listing an unresolved issue, thanks

[xperiandri]

By the way what about F# collections upgrade to Span and Memory under the hood? Does it make sense?

[cartermp]

@xperiandri We'll take a look at that once the feature is in and baked. There are also multiple areas in the compiler itself that can benefit from this.

[zpodlovics]

@xperiandri Something like the Array module should be possible for span, with functions that works directly on reference passed value types as inref<'T> instead of passed as a copy of the value type.

@cartermp When we can expect (if possible to predict/schedule) the span support? I would like to try it out if possible, in fact I have checked the CI builds / CI wiki to try it out, but not yet found a way to do it on linux + dotnet core. The /usr/share/dotnet/sdk/2.1.4/FSharp/ directory have a really different set of files.

[cartermp]

@zpodlovics We're aiming to release with VS 15.8, which will have a corresponding release of the .NET SDK where we will have a merged compiler with F# 4.5 features.

Between now and then, there may be another release of the .NET SDK where we can try to insert a newer compiler. Unfortunately, because F# is necessarily cross-cutting across products on different schedules, and VS has the largest population of users, we're having to align with the VS schedule.

[zpodlovics]

@cartermp I understand, and it's perfectly fine to target the main product schedule / user base. What do you mean by the ".NET SDK"? You mean the Windows10 SDK or the .NET Core SDK?

Please if you can insert the newer compiler into any point release (eg.: SDK 2.1.5, SDK dotnet-sdk-2.1.500-rc1 ). Please let me know if you have any alpha/beta/prerelease that I can try it out and test it on linux + dotnet core.

[dsyme]

By the way what about F# collections upgrade to Span and Memory under the hood? Does it make sense?

You mean collection functions like Span.map etc? I think it would be better to define them in a new nuget package.

[ghost]

I am not sure if this is the right place to post this, but for people googling F# + Span, this seems to be the first results, so, if you are using NetCore 2.1 (which has Span and friends enabled by default), and targeting netcoreapp2.1, the following will not build:

   Directory.GetFiles(dir, "*.*proj", SearchOption.AllDirectories)
   |> Seq.map (fun p ->
       let p = Path.GetDirectoryName p 
       Path.Join(p, "obj"))

/tmp/test/Program.fs(12,18): error FS0001: This expression was expected to have type    'ReadOnlySpan<char>'    but here has type    'string' [/tmp/test/test.fsproj]
...

Using a type annotation on the type will not work either (e.g. let p : ReadOnlySpan<char> = Path.GetDirectoryName p) with "This construct is deprecated. Types with embedded references are not supported in this version of your compiler." (this is starting to sound familiar)

Workaround

  Directory.GetFiles(dir, "*.*proj", SearchOption.AllDirectories)
  |> Seq.map (fun p ->
      let p = Path.GetDirectoryName p
      Path.Join(p.AsSpan(), "obj".AsSpan()))

From my perspective this is a breaking change. The CLR team has chosen to not break backwards compatibility in a way that precludes languages that do not rely on the crutch of implicit conversion :(

https://github.com/dotnet/coreclr/blob/master/src/System.Private.CoreLib/shared/System/IO/Path.cs, Instead of leaving the old String.Join(string,string) methods and marking them obsolete, they appear to be nuked entirely. Nice.

EDIT: @cartermp pointed out that Path.Join is actually new in netcore 2.1, (an entirely new API), so the actual changes do not break existing code bases. The workaround still stands but it is not as dramatic as it sounds.

[dsyme]

@abk-x Thanks for the heads up. cc @cartermp @terrajobst

As I've said elsewhere I think it is very, very wrong for .NET and C# (let alone F#) if basic, simple, clear entry-level APIs like System.IO are using Span or ReadOnlySpan in a way that is in-the-face of programmers. This appears to be just one example, but on a first glance it is, in my humble opinion, a clear case of a misuse of Span in API design.

IMHO Span should never, ever be used in API design as a catch-all replacement for the ubiquitous and simple string. I wonder if we will see a lot more of this, before people realise they have sacrificed simplicity (for, say, teenagers or data scientists or Python programmers or F# programmers...). You should never accidentally find yourself reading a tutorial on high-performance memory primitives just because you tried to join two paths.

It is very common to see this: people get a thing and they start hammering everything with that thing. If that goes ahead then the thing that ultimately gets hammered is usability and simplicity. For .NET, 99% of programmers should never need to know or think about Span. If we force that 99% to think about it, we will lose 50% of them, since the thing that the vast majority of programmers value above everything else is simplicity.

[dsyme]

In case anyone is getting worried about my comment above, here's the response from one of the core .NET API reviewers:

BTW, I agree that we should not have common scenario APIs where the developer has to use Span<T>.

[smoothdeveloper]

some context about Path.Join api: https://github.com/dotnet/corefx/issues/25536

[zpodlovics]

It seems that Span (~ memory views) will be new central concept in .NET really soon. I guess thinking in terms of memory views will be unfamiliar for most new developers, so it will be not easy to use (easy to use means familiar here).

It still has a potential to simplify things in long run, but I still have my own doubts about it: https://github.com/matthiasn/talk-transcripts/blob/master/Hickey_Rich/SimpleMadeEasy.md

There is no clear path here, as an alternative F# introduce their own string library as an easy to use high level library for beginners, but that could create even more confusion for the beginners in longer run when they need .NET interop...

[cartermp]

@zpodlovics I think that's a bit of a hasty view. Span is not going to be a cornucopia, nor does the .NET team feel it should be. It's currently only properly available in .NET Core 2.1, with no current plans to move it to a .NET Standard until 3.0 (but this is pending user feedback). As you pointed out, working in terms of memory views is not easy and it they have no plans to hamstring .NET by making that the default way to use things.

I'm not sure where F# comes into this. We're quite close to merging our work and the "Span and friends" features. And today without that work in, you can use some of the few Span-only APIs, e.g.:

Path.Join(p1.AsSpan(), p2.AsSpan())

Is there a more specific issue you have in mind?

[cartermp]

Closing as implemented.

[haf]

How do you convert a Span to a ReadOnlySpan in F#? All docs talk of C# and AsSpan() (which doesn't show up on code completion) and it being an implicit cast (Figure 1). I'm targeting netcoreapp2.1 here.

I've tried:

• .AsSpan(): overload resolution fails
• :> ReadOnlySpan<_>: no proper subtypes...
• :?> ReadOnlySpan<_>: no proper subtypes...
• upcast ...Slice(): same as above
• Passing it in as-is: see screenshot
• ReadOnlySpan.op_Implicit(...Slice())

[svick]

@haf

The following code compiles for me:

let spanToROSpan (span : Span<'a>) : ReadOnlySpan<'a> =
    Span<_>.op_Implicit(span)

I don't know if there's a better way and I agree there should be one.

[haf]

@svick That function cannot be used; gives

error FS0412: A type instantiation involves a byref type. This is not permitted by the rules of Common IL

when I use it...

[svick]

@haf

It works for me. The following code runs fine on .Net Core SDK 2.1.403:

open System

let spanToROSpan (span : Span<'a>) : ReadOnlySpan<'a> =
    Span<_>.op_Implicit(span)

[<EntryPoint>]
let main argv =
    let span = Span<int>([| 42 |])
    let roSpan = spanToROSpan span
    printf "%A" roSpan.[0]
    0

[haf]

EDIT: compilers compilers... It would seem JetBrains Rider doesn't compile with the installed latest F# compiler, so I get different compiler results depending on whether I compile in the IDE or through the command line.

EDIT 3: You can select this here:

---

EDIT 2:

You're right, this compiles:

let private readOnly (span: Span<'a>): ReadOnlySpan<'a> =
  Span<_>.op_Implicit(span)

let xx () =
  let hash = IncrementalHash.CreateHash HashAlgorithmName.SHA1
  let value = "abc"
  let buf = Span<byte> [| 42uy |]
  let read = utf8.GetBytes(value.AsSpan(), buf)
  let s = buf.Slice(0, read)
  hash.AppendData(readOnly s)

But not

let private readOnly (span: Span<'a>): ReadOnlySpan<'a> =
  Span<_>.op_Implicit(span)

let xx () =
  let hash = IncrementalHash.CreateHash HashAlgorithmName.SHA1
  let value = "abc"
  let buf = Span<byte> [| 42uy |]
  let read = utf8.GetBytes(value.AsSpan(), buf)
  hash.AppendData(buf.Slice(0, read) |> readOnly)

...which is what I used. Perhaps this is a bug in the implementation then.

[svick]

@haf My guess is that that's related to the fact that byref types can't be used as first-class functions. E.g., this code:

let f = spanToROSpan

produces:

error FS0425: The type of a first-class function cannot contain byrefs

I don't know if using |> with byref functions could or should work, but maybe you should open a separate issue about that.

[cartermp]

@haf We have some documentation on this now, though my cursory glance with a search engine shows it's not that discoverable yet...

Span is a byref-like struct (and ReadonlySpan is a readonly byref-like struct), which means they are subject to the rules listed. The relevant one is this:

They cannot be used as a generic parameter.

This last point is crucial for F# pipeline-style programming, as |> is a generic function that parameterizes its input types. This restriction may be relaxed for |> in the future, as it is inline and does not make any calls to non-inlined generic functions in its body.

This is tracked by #688.

As for the op_implicit, maybe this is something we could look at. Thoughts @dsyme?

[dsyme]

I don't know if using |> with byref functions could or should work, but maybe you should open a separate issue about that.

It is by design that this doesn't work. |> is a generic operator in F#. But agreed #688 tracks the suggestion though I don't think that will be implemented.

Pipeline programming is only an idiom in F#, it is not essential

As for the op_implicit, maybe this is something we could look at. Thoughts @dsyme?

I would like to make some kind of change for better support for op_Implicit and op_Explicit. Exactly what to do remains TBD

[KySpace]

Why is this closed. I don't see it working at all. Does this count as inner function? Partial application does not work. Recursion does not work. How is this closure or inner function though? Piping does not work. Please...

[TIHan]

@KySpace

Does this count as inner function?

Yes, rsc is an inner function that is capturing Sample which is a byref-like (byref or Span) type which is not allowed. The error message describes exactly that. This is by design.

Partial application does not work.

This is by design. We do not permit partial application on functions that have a byref-like parameter type. However, the error message isn't that descriptive.

Recursion does not work. How is this closure or inner function though?

Recursion does work, but we have a limitation on local functions having byref-like parameter types. We were recently discussing this: https://github.com/fsharp/fslang-suggestions/issues/805 - I have a draft of a potential design in my head that I want to share soon.

Piping does not work.

As @dsyme mentioned above: It is by design that this doesn't work. |> is a generic operator in F#. Byref-like types cannot be used as type arguments for generic functions or operators. There was talk about having it work for inline functions, https://github.com/fsharp/fslang-suggestions/issues/688, but in my opinion, it should not be implemented.

[cartermp]

@KySpace Span is a special primitive with a very restrictive programming model that allows the runtime to elide bounds checking and assume that data is only on the stack. Many normal F# programming idioms would violate the latter, hence they are not supported. The generalized concept, ByRefLike structs, are the same. You can read more about it here: https://docs.microsoft.com/en-us/dotnet/fsharp/language-reference/byrefs#byref-like-structs

[KySpace]

@TIHan Thanks for the explanation. I hope local function will work soon.

[KySpace]

@cartermp @TIHan But this also prohibit encapsulating Span data and isolate it from other functions right? Or is there any workarounds? Implementing the encapsulation in C# code maybe?

[TIHan]

@KySpace It is possible to encapsulate Span if you create a type, assuming you are on NetStandard 2.1, NetCore 2.0+ or NetFramework4.7.1+, with the IsByRefLike attribute:

open System.Runtime.CompilerServices
[<Struct;IsByRefLike>]
type EncapsulateSpan (x: Span<int>) =
    member _.X = x

This makes the type, EncapsulateSpan, have the same restrictive behavior as Span though.

Lambdas are semantically not byref-like, therefore, they cannot capture byrefs or Spans. C# has the same limitations.

Just as a FYI regarding local functions, your original example showed a local function capturing a byref-like type. This will still be invalid even when we start allowing byref-like parameter types for local functions. Once we fix the limitation with local functions, you just need to pass the byref-like type as an argument instead of the lambda capturing it.

[KySpace]

@TIHan Hi, I don't think I quite understand the word "capture". I see why others won't work, but to my understanding, I have passed a span as argument in the recursion example: let pos, data = read s pos raw, both pos and data should be evaluated as values at this point now. Is there a difference between the term parameter and argument here? BTW I decide to use Stream instead.