New Library - Wraps repetitive code with LanguageExt

[gregberns]

Hi Paul, Recently I've joined a company as tech lead and am rebuilding a platform with C#. Obviously LanguageExt is playing a centrall roll in the code base.

After the 5th small service I found I was re-writing/copy-pasting a large portion of the code like: wrapping GetEnvironmentalVariable with Validation, wrapping HttpClient, JSON and Yaml deserialization, and a paging in REST APIs.

So I took all the common code and aggregated it into: LanguageExtCommon. We've now used it for the last couple weeks and cut out a ton of repetitive code in our services.

Its still very much in alpha, but I'd image we'll add things like database wrappers (maybe as separate nuget's), Azure SDK, who knows...

Anyways, I just wanted to share this and get any feedback, suggestions, or ideas you may have. Hell, if there's a better name for the project I'm open to it!

https://github.com/gregberns/LanguageExtCommon

cc @StefanBertels (since you seem very active the last couple years)

[StefanBertels]

Hi Greg,

IMHO: I think a single helper library is problematic because it might result in "endless scope" and incompleteness.

If there is a general feature which seems missing in core (or parsec) this should be discussed/added via a PR to Pauls core library. Of course some features might be too special / opinionated / adding unwanted dependencies or not general enough. E.g. I would like to have functions in LanguageExt.Core for getting e.g. Either<Error, T> from any T returning function without the "need" to use custom or third party code to build an expression like try/catch => Try<T> => Either<Exception,T> => Either<Error,T>. If the built-in way is too much boilerplate, let's make it easier. But maybe it's not boiler plate but just clarity/explicitness.

If you solved a specific thing like e.g. interoperability with Azure SDK I suggest to create a helper package specifically for that use case and make them available independently of each other. (Probably you should check with Paul about naming it LanguageExt*.) I created some similar classlibs (internally, non-public) adding e.g. support for Option types and parsers in CsvHelper or get json reader/writer support for Json.NET and System.Text.Json (supporting more familar json formats). Doing this in a public library IMHO should aim to build a thing that isn't "incomplete" and should try to avoid being opinionated and avoid adding new types (like HttpResponse).

Third way IMHO is to try to add features in other libraries (via PR's) that make it easier to wire functional / LanguageExt code into it -- reducing the need to have overhead code like wrappers. Many things are already supported by LanguageExt out of the box like turning exception throwing code into Try or converting Action to Func<Unit> or having IDictionary.TryGetValue or turning char predicates into char parser using satisfy in Parsec etc.etc.

Finally I myself often end up with a collection of extension methods that I just "copy & paste" between solutions (buying the disadvantages of ignoring DRY principle). I still think this is a valid way to learn whether the code is universal enough to be a candidate for some own separate library or a PR to the core library.

Just my 2 cents.

[gregberns]

@StefanBertels Thanks for the feedback, thats exactly what I was looking for and many of your concerns I've been thinking about as well.

1) Missing from Core - Personally I haven't found anything missing from the core library. Only thing would be more examples/how to do particular things, but thats another discussion.

2) Separating Interop libraries - Yes, I also had concerns about building in support for specific things like CsvHelper, Azure SDK, any database specific libs, etc. I like the idea of separating those out into separate packages. It would be really nice to have consistent namespace conventions for these too.

I created some similar classlibs... Doing this in a public library IMHO should aim to build a thing that isn't "incomplete"

100% agree. Completeness is generally the first thing I've been pushing toward, but I hadn't thought of your other two points:

• Ensure completeness
• Avoid being opinionated
• Avoid adding new types

I probably haven't put enough thought into it, but for something like the HttpClient I don't know how you avoid adding new types, maybe extension methods?? Seem like there may be challenges there.

Also - how to handle System.* interfaces. Core handles plenty of System, but it might be helpful to have other parts like System.IO (File access) and other parts wrapped up. That could obviously be a lot of work...

3) I don't disagree, but doing that at scale seems challenging to say the least. It would be awesome to have something like Fantasy Land that others could build their interfaces to, but I don't see many libraries integrating in LanguageExt. Try does help but the interop code still needs to be written and copy/paste.

4) Copy/Paste Extension Methods - I've done the same for years and I agree it definitely helps prove out the interfaces/usefulness. Its exactly why I'd like to get that type of stuff into a standardized library.

@louthy - Any thoughts on the subject or a namespace convention that could be used to wrap System.* and any other packages or functionality (CsvHelper, Azure SDK, any database specific libs, etc)? Or is there a way you deal with things like this internally?

[louthy]

This overlaps somewhat with what I'm trying to achieve with the new Aff and Eff monads (asynchronous and synchronous effects, i.e. IO).

The Core library will provide wrappers for all IO in the BCL as a base-line, but it will be extensible to support 'plugin' IO. This is done via a Runtime that is the environment for the Aff and Eff monads (like the Reader monad environment). By, using a runtime that is passed to the computation, we get the opportunity to make all IO mockable without resorting to hideous DI frameworks. It also makes functions declare that they use IO, because they'll explicitly be marked Aff or Eff. Additionally the type of IO will be declarative.

To give you an example, I recently used the new effect monads in a internal project and needed something that converts text to a Json union-type, so I declared the general interface, but also a 'trait':

    /// <summary>
    /// Interface that specifies the behaviour of a JSON IO sub-system 
    /// </summary>
    public interface JsonIO
    {
        /// <summary>
        /// Convert a string to a Json
        /// </summary>
        Json Convert(string json);
    }

    /// <summary>
    /// Trait that tells a runtime it has the capacity to deal with JSON
    /// </summary>
    /// <typeparam name="RT">Runtime</typeparam>
    public interface HasJson<RT>
    {
        Eff<RT, JsonIO> Json { get; }
    }

The Json union-type looks like this:

    /// <summary>
    /// JSON union
    /// </summary>
    [Union]
    public interface Json
    {
        Json Record(HashMap<string, Json> Fields);
        Json StringValue(string Value);
        Json IntValue(int Value);
        Json LongValue(long Value);
        Json DoubleValue(double Value);
        Json BoolValue(bool Value);
        Json DateValue(DateTime Value);
        Json GuidValue(Guid Value);
        Json NoValue();
        Json Array(Seq<Json> Values);
    }

There then needs to be a 'live' implementation of the JsonIO (which uses Json.NET):

    public struct LiveJsonIO : JsonIO
    {
        public static readonly JsonIO Default = new LiveJsonIO();

        public Json Convert(string value) =>
            Walk(JsonConvert.DeserializeObject(value));

        Json Walk(object obj) =>
            obj switch
            {
                null         => new NoValue(),
                JArray jarr  => WalkArray(jarr),
                JObject jobj => WalkObject(jobj),
                JValue jval  => WalkValue(jval),
                _            => throw new NotImplementedException()
            };

        Json WalkValue(JValue val) =>
            val.Type switch
            {
                JTokenType.Boolean => new BoolValue((bool)val),
                JTokenType.Date    => new DateValue((DateTime)val),
                JTokenType.Float   => new DoubleValue((double)val),
                JTokenType.Guid    => new GuidValue((Guid)val),
                JTokenType.Integer => WalkInt((long)val),
                JTokenType.String  => new StringValue((string)val),
                _                  => new NoValue()
            };

        Json WalkInt(long val) =>
            val < (long) Int32.MinValue || val > (long) Int32.MaxValue
                ? JsonCon.LongValue(val)
                : JsonCon.IntValue((int) val);

        Json WalkArray(JArray jarr) =>
            new Array(jarr.Map(Walk).ToSeq().Strict());

        Json WalkObject(IDictionary<string, JToken> jobj) =>
            new Record(toHashMap(
                           jobj.AsEnumerable()
                               .Map(pair => (pair.Key, Walk(pair.Value)))));
    }

Access to a generic Convert function that works for an injectable runtime then can be wrapped up:

    /// <summary>
    /// JSON IO
    /// </summary>
    public static class JsonEff
    {
        /// <summary>
        /// Convert a string to a `Json` 
        /// </summary>
        public static Eff<RT, Json> convert<RT>(string json) where RT : struct, HasJson<RT> =>
            default(RT).Json.Map(j => j.Convert(json));
    }

That constrains RT to be a HasJson. HasJson provides a JsonIO wrapped in an Eff monad and that gives direct access to the injected subsystem.

And finally, the project will need a runtime that will make all effects and IO available:

    /// <summary>
    /// IO runtime
    /// </summary>
    public struct Runtime : HasCancel<Runtime>, HasJson<Runtime>, HasFile<Runtime>
    {
        readonly CancellationTokenSource cancellationTokenSource;
        public CancellationToken CancellationToken { get; }

        /// <summary>
        /// Construct a runtime
        /// </summary>
        /// <param name="cancellationTokenSource"></param>
        public Runtime(CancellationTokenSource cancellationTokenSource) =>
            (this.cancellationTokenSource, CancellationToken) = (cancellationTokenSource, cancellationTokenSource.Token);

        /// <summary>
        /// Cancellation token source
        /// </summary>
        public Eff<Runtime, CancellationTokenSource> CancellationTokenSource =>
            SuccessEff(cancellationTokenSource);

        /// <summary>
        /// Make a new cancellation toke
        /// </summary>
        public Runtime LocalCancel =>
            new Runtime(new CancellationTokenSource());

        /// <summary>
        /// JSON serialisation provider
        /// </summary>
        public Eff<Runtime, JsonIO> Json =>
            SuccessEff(LiveJsonIO.Default);

        /// <summary>
        /// Text encoding
        /// </summary>
        public Eff<Runtime, Encoding> Encoding =>
            SuccessEff(System.Text.Encoding.Default);

        /// <summary>
        /// File IO
        /// </summary>
        public Aff<Runtime, FileIO> FileAff =>
            SuccessAff(LanguageExt.LiveIO.FileIO.Default);

        /// <summary>
        /// File IO
        /// </summary>
        public Eff<Runtime, FileIO> FileEff =>
            SuccessEff(LanguageExt.LiveIO.FileIO.Default);
    }

You can see the traits in the inheritance list: HasCancel and HasFile from Lang-ext, and HasJson from this project. That allows for composition of effect systems from various libraries into a single runtime for any one project. Or, there could be several runtimes for layers within a system (like UI, Business Logic, Data Access, etc.)

If you look at couple of functions that use this:

static Aff<RT, Unit> Do<RT>(string path, string rootClass) 
    where RT : struct, HasCancel<RT>, HasJson<RT>, HasFile<RT> =>
        from tx in IO.File.readAllText<RT>(path)
        from js in JsonEff.convert<RT>(JsonEff.many(tx))
        from sc in EffMaybe(() => Generate(js, rootClass))
        from _1 in IO.File.writeAllText<RT>(Path.ChangeExtension(path, "cs"), sc)
        from _2 in IO.File.writeAllText<RT>(Path.ChangeExtension(path, "csproj"), MakeProjectFile(path))
        select unit;

static Fin<string> Generate(Json json, string rootClass) =>
    from tys in Infer.Run(json, rootClass, default)
    from src in Generator.Run(tys)
    select src;

You can see that Do needs the following traits: HasCancel, HasJson, HasFile (otherwise it wouldn't compile). And so, not only does it declare that it has side-effects inside, it declares exactly what they are. Note also that the Generate function is obviously pure, there are no side-effects.

These are the BCL methods I've done so far, that will grow as I build up to v4 of lang-ext (and so will cover the System.Environment and System.Net.Http wrappers you've built). But the idea is to map the namespace pretty closely, like IO.File.readAllText, but also fixup dishonest type-signatures (use of Option, or Seq instead of arrays, etc.), and to bring async forward as the default where there's sync and async implementations.

It's really to start wrapping up some of the difficult stuff with IO in a C# functional world, and to be a bit more opinionated on how to do it (questions regarding IO come up all the time).

To get back to the point of this request. I would expect that for anything outside of the BCL that is IO or a side-effect of any kind, or might require application wide injection (like my JSON example), that separate libraries would be created that would provide the traits needed to compose runtimes (like LanguageExt.IO.Azure, with traits like HasAzure, and implementations like LiveAzure and TestAzure).

In terms of name-spacing, I'm replacing System with LanguageExt for the BCL stuff; but I think it'd be up to you to decide how to achieve your goals. I think if you're making it for public consumption, then keeping it focussed, either around a subsystem (like 'Environment') or an area (like 'Web' for example) makes most sense.

[gregberns]

@louthy Holy shit. Just spent the night working through your examples. I heard about Effects in 2018 from John DeGoes in Scala and probably used them in Purescript... but damn, I never understood them until now. It reminds me of learning Dependency Injection... seemed complicated but provided IoC. This'll take a day to grok it - but without the shitty magic of DI !!!!

This is also awesome because I've been working on a data and logic layer to an API... and I'm pretty sure this solves all the issues I've been trying to deal with trying to do FP. I'll play with this in that project and get back to you. The only concern (initially) is that it seems like you have to buy in fully to Eff/Aff + Runtime, instead of just pulling it in when needed, which may makes a whole project require it and training FP nubes is even harder, but seems like the tradeoff is worth it.

Here's an even simpler version using pwd (present working directory) that seems to illustrate all the parts needed.

using System;
using System.Linq;
using LanguageExt;
using LanguageExt.Common;
using static LanguageExt.Prelude;
using LanguageExtCommon;
using Xunit;

namespace LanguageExt.HotDamn
{
    // specifies the behaviour of IO sub-system
    public interface PwdIO
    {
        string Pwd();
    }
    // Trait
    public interface HasPwd<RT>
    {
        Eff<RT, PwdIO> EffPwd { get; }
    }

    public struct LivePwdIO : PwdIO
    {
        public static readonly PwdIO Default = new LivePwdIO();

        public string Pwd() =>
            System.IO.Directory.GetCurrentDirectory();
            // throw new Exception("AHHHHHHHHH!!!");
    }

    public static class PwdEff
    {
        public static Eff<RT, string> pwd<RT>() where RT : struct, HasPwd<RT> =>
            default(RT).EffPwd.Map(p => p.Pwd());
    }

    public struct Runtime2 : HasPwd<Runtime2>
    {
        public Eff<Runtime2, PwdIO> EffPwd =>
            SuccessEff(LivePwdIO.Default);
    }

    public class PwdEffTests
    {
        static Eff<RT, string> DoThePwd<RT>()
            where RT : struct, HasPwd<RT> =>
                from pwd in PwdEff.pwd<RT>()
                select pwd;

        [Fact]
        public void Test()
        {
            var env = new Runtime2();
            var a = DoThePwd<Runtime2>();

            var b = a.RunIO(env);

            Assert.True(b.IsSucc);

            Assert.Equal(FinSucc("/Users/.../project/test/bin/Debug/netcoreapp3.1"), b.First());
        }
    }
}

[louthy]

@gregberns

I heard about Effects in 2018 from John DeGoes in Scala and probably used them in Purescript...

Yes, this overlaps with ZIO in Scala, and the Eff and Aff monads in PureScript, as well as the IO monad in Haskell. Of course, the implementation won't be the same, and is very much hand-optimised to be quick in C# (lock-free, use of ValueTask for all async work, support for memoised results, etc.).

The only concern (initially) is that it seems like you have to buy in fully to Eff/Aff + Runtime, instead of just pulling it in when needed, which may makes a whole project require it and training FP nubes is even harder, but seems like the tradeoff is worth it.

In some ways that's the point. It's to be opinionated about dealing with the world's side-effects. It's been a long running theme on this repo where I get questions about how to do IO and deal with state and side-effects. This is one way, but it will be the most supported and optimised way. It is of-course possible to just call ma.RunIO(runtime) to get out of the computation, or just use this for a subsystem, but you gain more by constraining yourself to work within it.

In Haskell, for example, there's your IO code and then there's your pure-code. Very much Haskell engineers try to push the IO to the edges so it doesn't pollute their nice pure code. This is kind-of the point of the Aff and Eff, it's to make you feel a bit dirty for using it (and the constraints are a bit awkward), it's trying to encourage you to segregate your IO and non-IO code. Which is what happened naturally for me in the example I posted above.

It can also be used to completely limit what a subsystem can do. Here we dictate the capabilities of the various layers within our enterprise application. This groups the traits together:

    // UI layer capabilities
    public interface HasUI<RT> : HasCancel<RT>, HasJson<RT>, HasWeb<RT>
    {
    }

    // Business logic layer capabilities
    public interface HasBusLayer<RT> : HasCancel<RT>, HasFile<RT>, HasMessaging<RT>
    {
    }

    // Data layer capabilities
    public interface HasDataLayer<RT> : HasCancel<RT>, HasDatabase<RT>, HasMessaging<RT>
    {
    }

We can then still just define one runtime, but use the layer traits instead

    public struct Runtime : HasUI<Runtime>, HasBusLayer<Runtime>, HasDataLayer<Runtime>
    {
        ...
    }

That means the constraints are then easier to work with (if slightly less declarative):

static Aff<RT, Unit> Do<RT>(string path, string rootClass) 
    where RT : struct, HasDataLayer<RT>=>
        ...;

[gregberns]

In some ways that's the point. It's to be opinionated about dealing with the world's side-effects.

Love it, sounds great to me.

Looks like I'll be spending the morning playing with this, but three more questions at your convenience:

• How do you run an Aff or Eff, other than ma.RunIO(runtime)?
• A Runtime may need environmental variables like a DB Connection string to run. Does the runtime fetch those values? Are they supplied via constructor? I assume the later. Do you need another runtime to initialize a set of runtimes? Or just stitch them together at the top level and kick them off?
• When I assert (in XUnit) on Fin (and probably others) and there is a difference, I just see Fin<String> [[[...]]], but not the inner details? Is this expected? Should I open an issue to try and improve this?

  Error Message:
  Assert.Equal() Failure
  Expected: Fin<String> [[[...]]]
  Actual:   Fin<String> [[[...]]]

If you have any more sample code/projects you could dump my way, I may put a project together to illustrate this and get some feedback?? - and submit as a PR??

[louthy]

How do you run an Aff or Eff, other than ma.RunIO(runtime)

That's the way. But, in theory, an application only ever needs one call to RunIO in Main. In reality, you may need it per web call, or wherever you have a boundary that you want to package up.

A Runtime may need environmental variables like a DB Connection string to run. Does the runtime fetch those values? Are they supplied via constructor? I assume the later. Do you need another runtime to initialize a set of runtimes? Or just stitch them together at the top level and kick them off?

The Runtime can contain data too. So, you can put configuration data, or anything you like into it. So, it's not just limited to injectable functionality. Another method is to leverage the Atom and Ref systems for managing global state. They have been updated to have Eff and Aff support, so managing global state (a side-effect), now has first-class support in the effect monads. And finally, you could have any external form of persistence (app.config or web.config for example) that has its own trait in the Runtime.

I started a massive refactor of my echo-process project to use Aff and Eff, so there's quite a few examples in there.

• Actor.cs - This is the per-process actor class that encapsulates state and manages the invocation of Aff and Eff when messages arrive.
• ActorContext.cs - This is an example of dealing with some global state (the running process-systems and the default process-system); it shows the use of Atom with Eff
• ProcessSystemConfigAff.cs - This controls the configuration for the processes. Note how the RT runtime is part of the static class's generic argument list, rather than the individual functions (as you've seen so far), that makes it easier to work with generics, because lots of stuff can be inferred which couldn't when providing the RT on a per-method call.
• EchoIO.cs - This is where the IO specific to the process-system, and the basis of the runtime is defined.
• RedisEchoIO.cs - This is the live implementation of the Redis IO for the process system
• RedisRuntime.cs - And the runtime so it can be used with Redis
  • Note how there's an EchoEnv in there as a property - this is some global data that flows through the system, but can also be updated for child-operations by calling localAff or localEff

I'm midway through this refactor, so I wouldn't try running it (or even compiling it), but it's a useful reference.

[gregberns]

Update: Sample project started, initially with the data layer. Very much a work in progress.

Everything interesting is in src/DataLayer.cs, and a test to run it is in test/DataLayerE2ETest.cs.

I used an Atom to store the connection string, but couldn't figure out how to pass the connection string into the instance, because static class SqlDbAff uses default(RT).AffSqlDb.MapAsync, and couldn't figure out how to not use default. Something to look at later. Update #2: I setup a global config item like in the Echo Cluster Configuration. I kinda like how that works - then the config(conn string) just needs to be set once when the system is initialized.

[louthy]

Some ideas to show how you could do the config different:

    public struct DataLayerRuntime : HasCancel<DataLayerRuntime>, HasSqlDb<DataLayerRuntime>
    {
        readonly CancellationTokenSource cancellationTokenSource;
        readonly string connectionString;

        public CancellationToken CancellationToken { get; }

        public static DataLayerRuntime New(string connectionString) =>
            new DataLayerRuntime(connectionString, new CancellationTokenSource());

        DataLayerRuntime(string connectionString, CancellationTokenSource cancellationTokenSource) =>
            (this.connectionString, this.cancellationTokenSource, CancellationToken) =
               (connectionString, cancellationTokenSource, cancellationTokenSource.Token);

        public DataLayerRuntime LocalCancel =>
            new DataLayerRuntime(connectionString, new CancellationTokenSource());

        public Eff<DataLayerRuntime, CancellationTokenSource> CancellationTokenSource =>
            Eff<DataLayerRuntime, CancellationTokenSource>(env => env.cancellationTokenSource);

        public Aff<DataLayerRuntime, SqlDbIO> AffSqlDb =>
            EffSqlDb.ToAsync();

        public Eff<DataLayerRuntime, SqlDbIO> EffSqlDb =>
            Eff<DataLayerRuntime, SqlDbIO>(env => LiveSqlDbIO.New(env.connectionString));
    }

This means the global environment is part of your runtime and is automatically threaded through the system.

The global atoms are valid too (although they do make future assumptions for you, that there will only ever be one configuration needed). A better way to access the configurations is to realise that reading a global value is also a side-effect, and so instead of configOrThrow (the exception being a side-effect), wrap the access in an Eff - you then benefit from all the automatic error handling built into the Eff.

    public static class ConfigurationStore
    {
        static readonly Atom<Option<Configuration>> configMap = Atom(Option<Configuration>.None);

        public static Eff<Unit> SetConfig(Configuration config) =>
            Eff(() => ignore(configMap.Swap(_ => config)));

        static Eff<A> NotInitialised<A>() =>
            FailEff<A>(Error.New("Configuration not initialised"));

        public static Eff<string> ConnectionString =>
            configMap.Value
                     .Map(c => c.ConnectionString)
                     .Match(SuccessEff, NotInitialised<string>);
    }

NOTE: That the Atom and Ref features don't need a runtime, as they're always working with transactional memory. They work they same in all scenarios, and so no injectable behaviour is needed. Therefore they work with Aff<A> and Eff<A>, not Aff<RT, A> and Eff<RT, A>

Then the runtime access to the SqlDbIO can be done like so:

        public Aff<DataLayerRuntime, SqlDbIO> AffSqlDb =>
            from conn in ConfigurationStore.ConnectionString
            select LiveSqlDbIO.New(conn);

        public Eff<DataLayerRuntime, SqlDbIO> EffSqlDb =>
            from conn in ConfigurationStore.ConnectionString
            select LiveSqlDbIO.New(conn);

Or alternatively, use Bind:

        public Aff<DataLayerRuntime, SqlDbIO> AffSqlDb =>
            ConfigurationStore.ConnectionString.Bind(LiveSqlDbIO.New);

        public Eff<DataLayerRuntime, SqlDbIO> EffSqlDb =>
            ConfigurationStore.ConnectionString.Bind(LiveSqlDbIO.New);

[louthy]

This is an alternative approach to the SqlDbAff. I have put the RT as an generic argument for the class rather than the methods.

    public static class SqlDbAff<RT>
        where RT : struct, HasSqlDb<RT>
    {
        public static Eff<RT, string> pwd =>
            default(RT).SqlDb.Map(p => p.Pwd());

        public static Aff<RT, T> querySingle<T>(string query, object param) =>
            default(RT).SqlDb.MapAsync(p => p.QuerySingle<T>(query, param));
    }

This can have some benefits if you ever want to fix the RT type with a using static:

    using static SqlDbAff<YourRuntime>;

It can also make usage of the functions a little clearer and use of other generic arguments easier.

[gregberns]

Fantastic! I can't decide whether I like adding the connection string from the constructor or global state. Conceptually passing it via the constructor removes any dependencies, which I like. But the last several small jobs I've used a pattern to pull EnvVars and add them to a Configuration object which is fine being global.

Representing the config access as an Effect made sense too, I think I was going that direction and got stuck. Exposing the ConnectionString explicitly is really nice too, hadn't thought about that.

I guess my next step is one of two things: to try and compose two runtimes, or add an API and see how to get the runtime stitched in.

Does it make any sense to put AspNet (the API system) into a Runtime?? Seems like it would be a pain/not feasible. Or is there a way we can represent/handle its Effects? Or do we just have it call into Runtimes ... via DI?? eek.

As always, thanks for all the help.

[louthy]

Does it make any sense to put AspNet (the API system) into a Runtime??

Obviously it depends entirely on how much you want to buy into a system like this, but side-effects are:

• Exceptions / Errors / No-value (i.e. None in Option)
• Access to global state
• Access to IO

I don't use ASP.NET Core on a regular basis, so I don't know what's involved there. But an abstraction layer on-top of any system like that will always give you options further down the track. My goal is to wrap up all the BCL, so I guess that gives an idea of the potential for a LanguageExt.AspNetCore library that does everything in an Eff or Aff way.

The other thing to note is that you don't have to use the Runtime for all IO. You can just lift existing IO into an Aff<A> or Eff<A>. i.e.

    public static class File
    {
        public static Eff<string> readAllText(string path) =>
            Eff(() => System.IO.File.ReadAllText(path));
    }

    public static class Math
    {
        public static Eff<double> divide(double x, double y) =>
            EffMaybe(() => y == 0 
                               ? FinFail<double>(Error.New("Divide by zero")) 
                               : FinSucc(x / y ));
    }

That doesn't have the dependency injection benefits, but allows you to easily lift behaviour into the effect system - you could create a trait for those later.

You could even do this:

    public static class File
    {
        public static Func<string, Eff<string>> readAllText =
            path =>
                Eff(() => System.IO.File.ReadAllText(path));
    }

    public static class Math
    {
        public static Func<double, double, Eff<double>> divide =
            (x, y) =>
                EffMaybe(() => y == 0
                                   ? FinFail<double>(Error.New("Divide by zero"))
                                   : FinSucc(x / y));
    }

Which would allow for the static functions to be set (in unit tests for example). Not exactly pretty, but avoids the need for runtime traits.

The use of the runtime environment to store config is valuable because it allows you to switch it depending on context. This is an example of a runtime that allows a user to be picked by ID, which can then be used to put the rest of the computation in the context of the user.

    [Record]
    public class UserEnv
    {
        public readonly string Name;
        public readonly string ConnectionString;
    }

    [Record]
    public class Env
    {
        public readonly HashMap<int, UserEnv> Users;
        public readonly Option<UserEnv> User;
    }

    public interface HasUsers<RT>
    {
        Fin<RT> SetCurrentUser(int id);
        Eff<RT, UserEnv> CurrentUser { get; }
        Eff<RT, Unit> AssertUserExists(int id);
    }

    public struct Runtime : HasCancel<Runtime>, HasUsers<Runtime>
    {
        readonly CancellationTokenSource cancellationTokenSource;
        readonly Env env;

        public CancellationToken CancellationToken { get; }

        public static Runtime New(Env env) =>
            new Runtime(env, new CancellationTokenSource());

        Runtime(Env env, CancellationTokenSource cancellationTokenSource) =>
            (this.env, this.cancellationTokenSource, CancellationToken) =
                (env, cancellationTokenSource, cancellationTokenSource.Token);

        public Runtime LocalCancel =>
            new Runtime(env, new CancellationTokenSource());

        public Eff<Runtime, CancellationTokenSource> CancellationTokenSource =>
            Eff<Runtime, CancellationTokenSource>(env => env.cancellationTokenSource);

        public Fin<Runtime> SetCurrentUser(int id) =>
            env.Users.ContainsKey(id)
                ? FinSucc(new Runtime(env.With(User: env.Users[id])))
                : FinFail<Runtime>(Error.New("Invalid user ID"));

        public Eff<Runtime, UserEnv> CurrentUser =>
            EffMaybe<Runtime, UserEnv>(
                env => 
                   env.env.User.Match(
                       Some: FinSucc,
                       None: FinFail<UserEnv>(Error.New("Current user not set"))));

        public Eff<Runtime, Unit> AssertUserExists(int id) =>
            EffMaybe<Runtime, Unit>(
                rt => rt.env.Users.ContainsKey(id)
                          ? FinSucc(unit)
                          : FinFail<Unit>(Error.New("User doesn't exist")));
    }

    public static class Users<RT> where RT : struct, HasCancel<RT>, HasUsers<RT>
    {
        public static Aff<RT, A> withUser<A>(int id, Aff<RT, A> ma) =>
            from _ in default(RT).AssertUserExists(id)
            from r in localAff<RT, RT, A>(env => env.SetCurrentUser(id).IfFail(env), ma)
            select r;

        public static Eff<RT, A> withUser<A>(int id, Eff<RT, A> ma) =>
            from _ in default(RT).AssertUserExists(id)
            from r in localEff<RT, RT, A>(env => env.SetCurrentUser(id).IfFail(env), ma)
            select r;

        public static Eff<RT, UserEnv> user =>
            from env in runtime<RT>()
            from usr in env.CurrentUser
            select usr;

        public static Eff<RT, string> userName =>
            from usr in user
            select usr.Name;

        public static Eff<RT, string> userConnectingString =>
            from usr in user
            select usr.ConnectionString;
    }

If we define an operation that expects a user to be set:

     var userOperation = from name in Users<Runtime>.userName
                         from conn in Users<Runtime>.userConnectingString
                         select (name, conn);

Then it will only work if it's been wrapped in something that has picked the user:

    var res = Users<Runtime>.withUser(123, userOperation);

So you could imagine a whole request/response loop having its context set. Including details of the request, details of the logged-in user, if any, etc. If there isn't an available user, it automatically fails gracefully, which could have security benefits. But the code itself does't care where or how that is managed, it becomes entirely part of the built-in behaviour of the monadic computation.

In the echo-process project I'm going all-in. That's for a couple of reasons:

• Almost everything it does is managing global state and IO
• I want to give the users of the library the ability to make their actor processes use Aff and Eff monads in the most efficient way
  • This leads to a much more powerful model than if I just allowed async processes
• I want to dogfood the Aff and Eff monads in the most extreme scenario

[gregberns]

Do you use something other than ASPNet? Or is most your work just back end without much API. Just wondering.

That gives me a ton to think about. That explanation is helpful - I definitely still don't fully grok the implications but that helps and more implementation will too. The user example was a perfect illustration because I'm working on a Shopping Cart/ECommerce platform right now which all revolves around a user. I'm going to take that and pull it into the example app as the business layer to see if I can better understand what you discussed.

Regarding the BCL work, last night I worked on seeing how an Environmental Variables interface would work. In the project that kicked off this conversation, I used the Validation applicative to grab a set of environmental variables, potentially parse them (int, bool, Url), and put them into a Configuration object (previously) discussed. The objective being to get all configuration on start of a process and fail immediately if the configuration was not set. Usage:

public static Validation<string, Configuration> GetConfig() =>
            (
                GetEnvUriAbs("ICOLOR_API_URL"),
                GetEnv("ICOLOR_API_AUTH_USERNAME"),
                GetEnv("ICOLOR_API_AUTH_PASSWORD"),
                GetEnv("ICOLOR_API_AUTH_SUBSCRIBERID"),
                GetEnv("ICOLOR_API_AUTH_CONSUMERID"),
                GetEnvInt("ICOLOR_API_PAGE_SIZE"),
                GetEnv("AZURE_STORAGE_CONNECTION_STRING"),
                GetEnv("AZURE_CONTAINER_NAME")
            ).Apply(
                (a, b, c, d, e, f, g, h) => new Configuration
                {
                    IColorUrl = a,
                    IColorUsername = b,
                    IColorPassword = c,
                    IColorSubscriberId = d,
                    IColorConsumerId = e,
                    IColorPageSize = f,
                    AzureStorageConnectionString = g,
                    AzureContainerName = h,
                    StartDate = DateTime.Today.AddDays(-1),
                    EndDate = DateTime.Today,
                }
            );

It appears that Eff and Aff are monadic but not applicative (or maybe apply just hasn't been written). Correction: Eff does have apply, but (Eff,Eff) does not appear to have it. Is there a way that we could compose multiple Effs, gather their results, and return as a compound object similar to Validation?

[louthy]

Do you use something other than ASPNet? Or is most your work just back end without much API. Just wondering.

Our system is 16 years old, so it was built on the original ASP.NET. But very early on I stripped it back to the bare minimum and built a bespoke type-safe UI system on-top of it. And so, day-to-day, we don't see ASP.NET front and centre.

That gives me a ton to think about. That explanation is helpful - I definitely still don't fully grok the implications but that helps and more implementation will too

There's a more elegant example in the echo-process system to set the context for the running processes. It knows about the message received, the sender, details of the process itself, all from static Eff and Affs.

If you look at RedisRuntime.cs, you'll see:

        /// <summary>
        /// Access to the echo environment
        /// </summary>
        public EchoEnv EchoEnv { get; }

        /// <summary>
        /// Set the SystemName
        /// </summary>
        public RedisRuntime SetEchoEnv(EchoEnv echoEnv) =>
            new RedisRuntime(
                source,
                echoEnv);

        /// <summary>
        /// Use a local environment
        /// </summary>
        /// <remarks>This is used as the echo system steps into the scope of various processes to set the context
        /// for those processes</remarks>
        public RedisRuntime LocalEchoEnv(EchoEnv echoEnv) =>
            new RedisRuntime(
                source,
                echoEnv);

Then in EchoIO.cs you can see the definition of the runtime environment state.

    public class EchoEnv
    {
        public readonly static EchoEnv Default = new EchoEnv(default, default, default, default);

        /// <summary>
        /// Current system
        /// </summary>
        public readonly SystemName SystemName;

        /// <summary>
        /// Current session ID
        /// </summary>
        internal readonly Option<SessionId> SessionId;

        /// <summary>
        /// True if we're currently in a message-loop
        /// </summary>
        internal readonly bool InMessageLoop;

        /// <summary>
        /// Current request
        /// </summary>
        internal readonly Option<ActorRequestContext> Request;

        /// <summary>
        /// Ctor
        /// </summary>
        internal EchoEnv(
            SystemName systemName,
            Option<SessionId> sessionId,
            bool inMessageLoop,
            Option<ActorRequestContext> request) =>
            (SystemName, SessionId, InMessageLoop, Request) = 
                (systemName, sessionId, inMessageLoop, request);

        /// <summary>
        /// Constructor function 
        /// </summary>
        public static EchoEnv New(SystemName systemName) =>
            new EchoEnv(systemName, None, false, None);

        /// <summary>
        /// Set the request
        /// </summary>
        internal EchoEnv WithRequest(ActorRequestContext request) =>
            new EchoEnv(SystemName, SessionId, true, request);

        /// <summary>
        /// Set the session
        /// </summary>
        internal EchoEnv WithSession(SessionId sid) =>
            new EchoEnv(SystemName, sid, InMessageLoop, Request);

        /// <summary>
        /// Set the session
        /// </summary>
        internal EchoEnv WithSystem(SystemName system) =>
            new EchoEnv(system, SessionId, InMessageLoop, Request);
    }

Which is then made visible in the HasEcho trait:

    public interface HasEcho<RT> : HasCancel<RT>, HasEncoding<RT>, HasTime<RT>, HasFile<RT>, HasSerialisation<RT>
        where RT : 
            struct, 
            HasCancel<RT>, 
            HasEncoding<RT>,
            HasTime<RT>,
            HasFile<RT>,
            HasSerialisation<RT>
    {
        /// <summary>
        /// Access to the echo environment
        /// </summary>
        EchoEnv EchoEnv { get; }

        /// <summary>
        /// Access to the echo IO
        /// </summary>
        Aff<RT, EchoIO> EchoAff { get; }

        /// <summary>
        /// Access to the echo IO
        /// </summary>
        Eff<RT, EchoIO> EchoEff { get; }

        /// <summary>
        /// Use a local environment
        /// </summary>
        /// <remarks>This is used as the echo system steps into the scope of various processes to set the context
        /// for those processes</remarks>
        RT LocalEchoEnv(EchoEnv echoEnv);
    }

Then these functions in ActorContext.cs allow the various contexts to be set for the passed Aff or Eff.

        public static Aff<RT, A> localSystem<A>(SystemName system, Aff<RT, A> ma) =>
            localAff<RT, RT, A>(env => env.LocalEchoEnv(env.EchoEnv.WithSystem(system)), ma);

        public static Eff<RT, A> localSystem<A>(SystemName system, Eff<RT, A> ma) =>
            localEff<RT, RT, A>(env => env.LocalEchoEnv(env.EchoEnv.WithSystem(system)), ma);

        public static Aff<RT, A> localSystem<A>(ProcessId pid, Aff<RT, A> ma) =>
            localSystem(pid.System, ma);

        public static Eff<RT, A> localSystem<A>(ProcessId pid, Eff<RT, A> ma) =>
            localSystem(pid.System, ma);

        public static Aff<RT, A> localSystem<A>(ActorSystem system, Aff<RT, A> ma) =>
            localAff<RT, RT, A>(env => env.LocalEchoEnv(env.EchoEnv.WithSystem(system.SystemName)), ma);

        public static Eff<RT, A> localSystem<A>(ActorSystem system, Eff<RT, A> ma) =>
            localEff<RT, RT, A>(env => env.LocalEchoEnv(env.EchoEnv.WithSystem(system.SystemName)), ma);

        public static Aff<RT, A> localContext<A>(ActorRequestContext requestContext, Aff<RT, A> ma) =>
            localAff<RT, RT, A>(env => env.LocalEchoEnv(env.EchoEnv.WithRequest(requestContext)), ma);

        public static Eff<RT, A> localContext<A>(ActorRequestContext requestContext, Eff<RT, A> ma) =>
            localEff<RT, RT, A>(env => env.LocalEchoEnv(env.EchoEnv.WithRequest(requestContext)), ma);

That then means static functions can be called when in-context, to get things like the current request:

        public static Eff<RT, ActorRequestContext> Request =>
            from e in EchoEnv
            from r in e.Request.Match(SuccessEff, FailEff<ActorRequestContext>(Error.New("Not in a message loop")))
            select r;

As you can see it can also gracefully fail when called out-of-context.

Remember it's an 'environment', not a 'state'. i.e. it works more like the Reader monad that can have local environments, but can't propagate state changes outside of its locale. State monads allow the internal state of the monad to leave the local area that it was set. This localised environment works really well for temporarily granting access, or temporarily creating something like a 'request'.

[louthy]

With regards to the validation question, and whether multiple errors could be carried, and applicative behaviour added: the short answer is no, it won't support that. The longer answer is I want to keep the effect system as lightweight and optimal as possible, so adding lots of bells and whistles compromises that. The single error type is a bit of a compromise, for sure, but there's ways of adding your own extensions to get around that:

public static Aff<Env, D> Apply<Env, A, B, C, D>(this (Aff<Env, A>, Aff<Env, B>, Aff<Env, C>) ms, Func<A, B, C, D> apply) 
    where Env : struct, HasCancel<Env> => 
        AffMaybe<Env, D>(async env =>
                                {
                                    var t1 = ms.Item1.RunIO(env).AsTask();
                                    var t2 = ms.Item2.RunIO(env).AsTask();
                                    var t3 = ms.Item3.RunIO(env).AsTask();

                                    var tasks = new Task[] {t1, t2, t3};
                                    await Task.WhenAll(tasks);
                                    return (t1.Result.ToValid(), t2.Result.ToValid(), t3.Result.ToValid())
                                                .Apply(apply)
                                                .Match(Succ: FinSucc, Fail: ErrorAggregate<D>);   
                                });

static Validation<Error, A> ToValid<A>(this Fin<A> ma) =>
    ma.Match(Succ: Success<Error, A>, Fail: Fail<Error, A>);

static Fin<A> ErrorAggregate<A>(Seq<Error> errs) =>
    FinFail<A>(Error.New(new AggregateException(errs.Map(e => (Exception)e))));

The Apply function here works with a tuple of three Aff effects, runs them in parallel, gets the result, and then runs the apply function, gathers the errors and puts them into an AggregateException, which is then returned ultimately as a single Aff.

That should give you the behaviour you want without anything additional in the core Aff or Eff types.

[gregberns]

Conceptually I get the 'env' idea, but I spent a day or two implementing some things and failed pretty hard. So I backed off and I wanted to get something very basic working. So the goal was to implement a simple http call to the Github API and deserialize the response. I created 3 IO layers: Json, HttpClient, and Github API. (Sorry, the code is really ugly right now with all the experimentation)

• JsonIO - Serializes Json, simple wrapper to handle the effects (exceptions)
• HttpClientIO - Manages an HttpClient, and sends an Http message and handles the side effects
• GithubIO - takes an HttpIO and JsonIO (and possibly credentials), and queries the GitHub API
• Runtime + Logic Layer - The runtime that ties the IO layers together, and a 'logic' layer of static methods to interact with
• Test - To show invoking it

To get it working initially, GithubIO takes JsonIO and HttpClientIO via the constructor. I don't like passing in the Json, but I'm not totally sure how to handle the HttpClient because that may need to handle a single HttpClient instance.

Also, in LiveHttpClientIO.HttpRequest(), I just returned EitherAsync<Error, HttpResponse>, but that makes using the interface a bit awkward. I'm a little confused about what an implementation should look like? Should exceptions not be caught at all? Should they be caught and transformed? Is there a way of returning the EitherAsync and transforming that to Aff?

Any other thoughts or changes would be very appreciated.

[gregberns]

I think it just clicked with this line of code:

public static Aff<RT, Lst<GithubOrg>> GetUserOrgs<RT>(string username)
    where RT : struct, HasCancel<RT>, HasHttpClient<RT>, HasJson<RT>, HasGithubCredentials<RT>
{
    return from credentials in Eff<RT, GithubCredentials>(env => env.GithubCredentials)
    ...
}

The runtime can have config and that config can be exposed via 'traits'/interfaces on the runtime. The static function knows nothing about the runtime, but can access the traits defined on the RT which was passed into the function. And the key is to use the Eff<RT, GithubCredentials>(env => ...) to access the value. I think it was the Echo code that gave me the idea to do that. Crazy!!

I also got rid of all the DI style crap. And realized that the Github module doesn't have effects, it just has logic and needs to use Http and Json which have effects, so there's no need for the runtime to know anything about Github (besides config credentials). Great day, only took a week :)

[louthy]

The runtime can have config and that config can be exposed via 'traits'/interfaces on the runtime. The static function knows nothing about the runtime, but can access the traits defined on the RT which was passed into the function. And the key is to use the Eff<RT, GithubCredentials>(env => ...) to access the value. I think it was the Echo code that gave me the idea to do that. Crazy!!

Yep. The constraints on RT specify what this function needs to be capable of (to perform its work). It doesn't care how you built it, or what additional features it has, as long as it has the minimum set of traits it needs to do the job.

NOTE: The trait model isn't totally needed, you could just have a runtime deriving from regular interfaces (rather than the HasX interfaces). The reason for the HasX model is to allow libraries to pre-build components of a runtime that you just plug in. Otherwise each time you wanted to, say have file access, you'd have to derive from FileIO and implement it all yourself.

So, the trait system is to provide that indirection. It facilitates the plug-in approach, but also has a nice side-effect of having names like HasFile, HasJson, etc. which I think is nicely declarative.

Great day, only took a week :)

It's a new way of thinking for the C# world (and therefore I will need to document the hell out of this); so kudos for pushing through with limited support! This fits with the shapes/roles/concepts investigation going on in the csharplang repo right now - these traits would be roles. In fact I would expect this system to gain super-powers if the C# language team come up with a good story around ad-hoc polymorphism.

[gregberns]

@louthy I implemented the System.Environment interface to understand better what it looks like. I did conversions of returning nulls to Options. I wasn't sure if we should accept nullables (string?). Let me know if there's anything I missed. If its helpful feel free to pull it into LangExt.

[StefanBertels]

Hi, looked at the implementation and have some questions:

1. GetEnvironmentVariables: Should this (and similar routines) use HashMap instead of IDictionary? I think it will be hard to change every type (might make switching from legacy to Eff style hard). But on the other hand code quality might be better (finally). This is similar to T? vs Option<T>.
2. What's the recommendation for exceptions? Let everything unchanged (never catch), catch non-deterministic runtime exceptions only (network broken / file not found), catch every runtime exception? What about invalid filename/path? I'm sure we won't want catch every possible exceptions. I personally removed most Try encapsulation in my own "library" functions because it gets to messy to have this always (I can always add a Try() around something). Hmm.

[louthy]

Let me know if there's anything I missed. If its helpful feel free to pull it into LangExt.

Rather than review this in-place, I think (if you're up for helping in general - and anyone else), I have setup a new project to manage the progress. I haven't done a full dig into the scope of what's going to be wrapped, but System.Environment is there. There's some examples already where I'm placing the final API surface within a partial class called LanguageExt.IO (I'm up for a discussion of that, although I think it will work for the System stuff).

Some complexities come around working with streams. I'm working on a resource management system for the Aff and Eff types (currently there's a use function, but I'd like something more advanced). There's also the new Pipes system, which is the ultimate side-effects wrapping system - it's an adaptation of the Haskell Pipes library - but it's pretty hard to use because of the limitation of C#'s generics. So, I'd like something more compelling for dealing with streams and long-lived side-effects

I wasn't sure if we should accept nullables (string?).

Obviously for your own code it's up to you. I think I'd prefer to stick with Option<A> for now, because not all consumers of this library are able to move to nullable references yet. Option<A> will accept null and deal with it properly, and will also accept Option (obviously), whereas nullable references won't accept Option. Still, happy to have a discussion about that. Don't return Option though, always return an Eff or Aff instead (in a fail or success state).

Should this (and similar routines) use HashMap instead of IDictionary

Yes. It's very much trying to leverage the toolkit of language-ext. Internally it can call hashMap.ToDictionary() which is a no-allocation conversion.

What's the recommendation for exceptions?

Exceptions are caught automatically and wrapped in an Error. But I think non-exceptional errors should leverage the FailEff and FailAff calls where appropriate.

[gregberns]

Alright, I created a PR for EnvironementIO. I haven't gone back to return HashMap's and did some work on the Option/nullables that'll need some changes. We can discuss the details further in there.

The pipes look interesting, once I get my head wrapped around this I'd like to spend some time understanding it better.

I took a stab at an HttpClient (partial) implementation but quickly got stuck primarily because the HttpClient returns objects that in turn have effects. My initial goal was to implement a simple workflow (line 129):

var client = new HttpClient();
var res = await client.GetAsync("url");
var body = await res.Content.ReadAsStringAsync();

My guess is that something like the HttpResponseMessage could go into a struct and then we can access the data in it via setting the context with localAff similar to the other examples we've looked at. But I'm struggling with how/if we try and keep the interfaces consistent with what exists in the underlying layer. This seems to work quite a bit differently than say FileIO or ConsoleIO, and its surface area has numerous classes. I'll spend some more time on it, but if you have any hints...

Edit: Note this is out of date since its actively being worked on and changes pushed. Look at main to see the most recent version.