Investigate use or pure powershell

[chriskuech]

Considering powershell can natively instantiate .NET classes, it would simplify development (no builds, better intellisense) and maintainability of the code base if Polaris was implemented in pure Powershell

[TylerLeonhardt]

This is something I've been thinking about since the beginning. It would be nice to remove the complexity of having 2 languages - and removing the need to build at all.

[chriskuech]

I just sifted through the C# code. It looks like you exclusively used encapsulation instead of inheritance, so I don't see anything that jumps out as incompatible with PowerShell. Are there any specific concerns you have that I should be aware of? If not, I might take up this task this weekend.

[TylerLeonhardt]

The only concern I have is that the Httplistener needs to be listening on another thread so that we can get the prompt back. Since PowerShell is single threaded, I'm not quite sure how that will work.

That said, I don't see anything out of the ordinary.

I would be extremely thankful (get it? Because it's Thanksgiving) for that PR! I'm on vacation this week so that would be awesome to come home to :)

[chriskuech]

I implemented Polaris using pure PowerShell in https://github.com/chriskuech/Polaris/tree/purepwsh. PowerShell does not support the same language constructs as C#, so it is not backward compatible with the current implementation of Polaris.

Start-PolarisServer starts the HttpListener in a background thread using a PowerShell Job. This requires that passed in data must be completely serialized, so the Polaris server can no longer support modifying the live server.

It also supports regex routing for a more idiomatic PowerShell solution, deprecating the need for ASP.NET routing.

A lot is changed and it is not yet feature-complete to be merged in, but I wanted to make sure you're ok with the changes required before continuing.

[TylerLeonhardt]

Wow @chriskuech! This is amazing progress!

Allow me to address your concerns:

live server editing

I really liked this feature... However, I like the thought of Polaris being in straight PowerShell more than live server editing. Plus not being able to edit the live server is parity with Express.js which was the inspiration for this project so I'm fine with loosing live server editing.

regex routing

Incredible! I've wanted to add this feature since day one. Thanks for that work!

other concerns

The biggest thing to keep in mind is that Polaris needs to work on both PowerShell Core 6 and Windows PowerShell. I do all my Polaris work on my Mac on purpose to embrace the new visions for PowerShell. I have not looked too much at your fork but do keep that in mind. If you'd like to run the Pester tests against Linux and don't have a system to test on, feel free to submit a PR preemptively just to trigger the CI builds!

Once again, thank you so much for giving Polaris a try and for attacking this pure PowerShell work head on!

I think if we get this rewritten in PowerShell, I'll finally get to throwing it in the PowerShell Gallery.

[TimCurwick]

If this is going to happen, the sooner the better as it is incompatible with any changes anybody else may be working on.

In Chris' current code, the server can only handle one request at a time. This is a significant and unacceptable loss of functionality. We need multithreading. I recommend using PowerShell objects for the multithreading rather than Jobs.

PowerShell wildcards would be better than regex and should at least be an option for route matching.

[chriskuech]

I recommend using PowerShell objects for the multithreading rather than Jobs.

Job is only used to free up the REPL thread, so we will still have a Job. I will port over multithreading to the degree that it's implemented in the C# version.

PowerShell wildcards would be better than regex and should at least be an option for route matching.

Globs can't replace regex completely due to lack of capture support. I do, however, agree that globs should be the default language for routing longterm. If you would like to implement this feature, please don't hesitate.

[TylerLeonhardt]

I will port over multithreading to the degree that it's implemented in the C# version

Awesome! I'm really looking forward to this. If you run into any issues understanding what I did in the C# version, I'm here to answer any questions.

[johnnygtech]

I think this will help with, if not completely preclude, the question about polaris running on window10 iot core! What do we need to keep this idea growing?

[TylerLeonhardt]

@chriskuech do you need any help getting multithreading working?

[majkinetor]

God damn guys this is awesome.

[chriskuech]

Sorry, this kinda dropped off my radar, but I can probably finish it in the next few days. Basically, I need to move around a bunch of code due to limitations with PowerShell's concurrency model.

[TylerLeonhardt]

No worries @chriskuech :) if you need any help, let us know. As you can see, a lot of folks are excited for this!

[chriskuech]

This isn't production ready yet, but I got it to a state where it runs and people can contribute to it and/or comment. https://github.com/chriskuech/Polaris/tree/purepwsh

Basically, I originally had a pure OOP solution that cleanly separated cmdlets from core logic and used attribute-based validation directly in the classes. Unfortunately, there is no way (that I'm aware of) to bind PowerShell classes to a namespace; so, unlike objects with namespaced types, we cannot move objects of our plain PowerShell class types between threads. As such, I had to remove all the class definitions and use hashmaps instead of custom classes.

Features

• Implemented in pure PowerShell (obviously)
• Handlers are processed on background threads, allowing multiple requests to be processed concurrently.
• Supports middleware pipelines. Every matching route will process the request unless the handler returns a value.
• Supports glob route matching by default
• If $Matches is one of the defined handler parameters, regex route matching is used (instead of glob matching) and $Matches is populated equivalent to using the -match operator. This should provide just as much power as any routing language (ie ASP.NET routing) without diverging from the simplicity that one would expect from a PowerShell-based server.

Outstanding issues

• For some unknown reason, Start-PolarisServer can't be run in a job, forcing the host to block
• CTRL+C doesn't kill the process when blocking on HttpListener.GetContext()
• No Pester tests yet (partially blocked by above issues)
• Feature comparison against existing solution is needed
• Convenience cmdlets (Add-PolarisRoute, Remove-PolarisRoute, Add-PolariStaticRoute) still need to be validated
• No input validation
• Static handler uses text/plain content type for all file types
• (potential feature) If content type is unspecified, content type is intelligently guessed

[jsuntheimer72]

Fantastic

On Dec 28, 2017 9:42 PM, "Chris Kuech" notifications@github.com wrote:

This isn't production ready yet, but I got it to a state where it runs and people can contribute to it and/or comment. https://github.com/chriskuech/ Polaris/tree/purepwsh

Basically, I originally had a pure OOP solution that cleanly separated cmdlets from core logic and used attribute-based validation directly in the classes. Unfortunately, there is no way (that I'm aware of) to bind PowerShell classes to a namespace; so, unlike objects with namespaced types, we cannot move objects of our plain PowerShell class types between threads. As such, I had to remove all the class definitions and use hashmaps instead of custom classes.

Features

• Implemented in pure PowerShell (obviously)
• Handlers are processed on background threads, allowing multiple requests to be processed concurrently.
• Supports middleware pipelines. Every matching route will process the request unless the handler returns a value.
• Supports glob route matching by default
• If $Matches is one of the defined handler parameters, regex route matching is used (instead of glob matching) and $Matches is populated equivalent to using the -match operator. This should provide just as much power as any routing language (ie ASP.NET routing) without diverging from the simplicity that one would expect from a PowerShell-based server.

Outstanding issues

• For some unknown reason, Start-PolarisServer can't be run in a job, forcing the host to block
• CTRL+C doesn't kill the process when blocking on HttpListener.GetContext()
• No Pester tests yet (partially blocked by above issues)
• Feature comparison against existing solution is needed
• Convenience cmdlets (Add-PolarisRoute, Remove-PolarisRoute, Add-PolariStaticRoute) still need to be validated
• No input validation

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/PowerShell/Polaris/issues/81#issuecomment-354389905, or mute the thread https://github.com/notifications/unsubscribe-auth/AhAGHyrNPWbG4OSZHiSaWyuCb4thN5ZGks5tFFGagaJpZM4Qjm8A .

[jsuntheimer72]

Is there a way to easily connect this to a protocol buffer/grpc endpoint?

On Dec 29, 2017 11:32 AM, "Jeremy Suntheimer" jsuntheimer72@gmail.com wrote:

Fantastic

On Dec 28, 2017 9:42 PM, "Chris Kuech" notifications@github.com wrote:

This isn't production ready yet, but I got it to a state where it runs and people can contribute to it and/or comment. https://github.com/chriskuech/Polaris/tree/purepwsh

Basically, I originally had a pure OOP solution that cleanly separated cmdlets from core logic and used attribute-based validation directly in the classes. Unfortunately, there is no way (that I'm aware of) to bind PowerShell classes to a namespace; so, unlike objects with namespaced types, we cannot move objects of our plain PowerShell class types between threads. As such, I had to remove all the class definitions and use hashmaps instead of custom classes.

Features

• Implemented in pure PowerShell (obviously)
• Handlers are processed on background threads, allowing multiple requests to be processed concurrently.
• Supports middleware pipelines. Every matching route will process the request unless the handler returns a value.
• Supports glob route matching by default
• If $Matches is one of the defined handler parameters, regex route matching is used (instead of glob matching) and $Matches is populated equivalent to using the -match operator. This should provide just as much power as any routing language (ie ASP.NET routing) without diverging from the simplicity that one would expect from a PowerShell-based server.

Outstanding issues

• For some unknown reason, Start-PolarisServer can't be run in a job, forcing the host to block
• CTRL+C doesn't kill the process when blocking on HttpListener.GetContext()
• No Pester tests yet (partially blocked by above issues)
• Feature comparison against existing solution is needed
• Convenience cmdlets (Add-PolarisRoute, Remove-PolarisRoute, Add-PolariStaticRoute) still need to be validated
• No input validation

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/PowerShell/Polaris/issues/81#issuecomment-354389905, or mute the thread https://github.com/notifications/unsubscribe-auth/AhAGHyrNPWbG4OSZHiSaWyuCb4thN5ZGks5tFFGagaJpZM4Qjm8A .

[TylerLeonhardt]

@jsuntheimer72 Without knowing the full ins-and-outs of HttpListener, I want to say no to grpc. I'm not aware of a way to use HttpListener to connect to a proto buff/grpc endpoint.

We are limited by what HttpListener supports: https://msdn.microsoft.com/en-us/library/system.net.httplistener(v=vs.110).aspx

I really hope someone can prove me wrong 😄

[TylerLeonhardt]

@chriskuech AWESOME progress!

Quick question: I'm a little confused on why you're not using HttpListener.GetContextAsync. With Tasks you can asynchronously handle requests that come in without having to spin up a thread per request.

All and all, this is really looking great!

[chriskuech]

I believe that's how I originally implemented it but received an error as PowerShell cannot have concurrent threads/processes without either Job or runspace isolation. This "feature" was the single most influential design restriction and caused much rewriting and complicated debugging since

PowerShell may be a .NET language, but it feels like interop only works from C# to PowerShell, whereas most multilanguage runtimes offer true interop between languages. PowerShell has some of the most fun and powerful language features, but things like lack of ability to join PowerShell classes to namespaces, export PowerShell classes from modules, and concurrency are preventing the language from being a true general purpose programming language with full .NET interop. Perhaps you can fast-track that feedback to the rest of PowerShell team and PowerShell 7 can finally convince people to stop using Python for DevOps? 😸 😸 😸

If everything looks good from a design standpoint, perhaps we can push this to a dev branch and people can help with some of the outstanding issues?

[TylerLeonhardt]

@chriskuech I created a branch for you called pure-pwsh: https://github.com/PowerShell/Polaris/tree/pure-pwsh

Feel free to send a PR there 😄

[chriskuech]

OK just did. Sorry the merge is so messy...

[johnnygtech]

Does a successful resolution of this issue indicate that we drop the Newtonsoft Json dll dependancies because Posh supports ConvertTO and ConvertFrom Json natively?

[TylerLeonhardt]

@johnnygtech Yep! Although under-the-hood pswh v6 uses Newtonsoft (pretty sure), so on that we'll still be using it under the hood 😄

[Tiberriver256]

Hey guys, I really like the direction @chriskuech is going but it looks like enough breaking changes to be an eternal pull request.

I attempted to do more straight conversion just moving the C# classes into PowerShell classes that seems to have worked pretty well. The tests and the cmdlets are mostly unchanged (although I did modify the static file server with a directory browser from PoshServer). The majority of the tests seem to be passing except for the PoshServer ones, something with the query strings isn't quite right. Live editing of the server still works using runspaces and a synchash (technique courtesy of Boe Prox)

I only get a free weekend to do hack away like this every once in a blue moon and it is nearing the end of the weekend so I thought I would post and let you guys know, in case you wanted to use it.

It's over in my fork here: https://github.com/Tiberriver256/Polaris

If I get some more time tonight I'll try and get the last few tests passing but it's probably time I spent some time with the family. It was a lot of fun and hopefully you guys can use some of it.

[TylerLeonhardt]

Awesome, @Tiberriver256!! I'm so happy to see so much interest and effort in making this pure PowerShell. I think it will help the product iterate faster.

Also, thanks for adding a static directory browser! I've been wanting to get to that :)

[hackjammer]

Would love to see this merged - agreed would help iteration

[TylerLeonhardt]

I prototyped a "full" PowerShell version of Polaris. It doesn't do EVERYTHING Polaris can, but the core of Polaris has been moved over.

There's still a single Add-Type... but at least there's no building required. It even leverages Register-ObjectEvent so no more Runspace management 😄

Here's what I've got so far:

Add-Type -TypeDefinition @'
using System;
using System.Net;
using System.Threading;
using System.Threading.Tasks;

namespace Polaris
{
    public class PolarisCore
    {
        public delegate void EventRaised(HttpListenerContext context);
        public event EventRaised myEvent;

        private async Task ListenerLoop(int Port)
        {
            var web = new HttpListener();
            web.Prefixes.Add($"http://localhost:{Port}/");
            Console.WriteLine("Listening..");
            web.Start();
            while (true)
            {
                var context = await web.GetContextAsync();
                myEvent(context);
            }
        }

        public void Start(int Port = 8080)
        {
            Thread listenerThread = new Thread(async () => { await ListenerLoop(Port); });
            listenerThread.Start();
        }
    }
}
'@

$ROUTES = [hashtable]@{}

function Register-PolarisRoute {
    param(
        [string]
        $Path,
        [string]
        $Method,
        [scriptblock]
        $ScriptBlock
    )

    if (-not $ROUTES[$Method]){
        $ROUTES[$Method] = [hashtable]@{}
    }
    $ROUTES[$Method][$Path.Trim("/")] = $ScriptBlock

}

function Start-Polaris {  
    param(
        [int]
        $Port = 8080
    )
    $polarisCore = [Polaris.PolarisCore]::new()
    $polarisCore.Start($Port)

    Register-ObjectEvent -InputObject $polarisCore -EventName myEvent -Action {
        $response = $Args[0].Response

        $Path = $Args[0].Request.RawUrl.Trim("/")
        $Method = $Args[0].Request.HttpMethod

        Write-Host "$Method $Path"

        if ($ROUTES[$Method] -and $ROUTES[$Method][$Path]) {
            $responseString = $ROUTES[$Method][$Path].Invoke()[0]
        }
        else {
            $responseString = "Not Found"
        }
        Write-Host $responseString
        $buffer = [System.Text.Encoding]::UTF8.GetBytes($responseString)
        $response.ContentLength64 = $buffer.Length
        $output = $response.OutputStream
        $output.Write($buffer, 0, $buffer.Length)
        $output.Close()
    }
}

This prototype lets you do:

Register-PolarisRoute -Path /hello -Method Get -ScriptBlock {
    return "wow"
}

Start-Polaris

Not only that, but a "new feature" of using the active runspace means scoping isn't as painful!

$hey = "hey"
Register-PolarisRoute -Path /hello -Method Get -ScriptBlock {
    return "wow $hey"
}

Start-Polaris -Port 8082

Works as you expect!

I'm really liking this model over the previous implementation. Would love some feedback :)

(NOTE: I know that that snippet of code doesn't do EVERYTHING that the current Polaris can, but it lays a foundation that can be built on top of to get to feature parity)

[Tiberriver256]

@tylerl0706 - Any reason to keep the c# in there instead of straight PowerShell?

[TylerLeonhardt]

AFAIK, it's for this line:

var context = await web.GetContextAsync();

Wanted to still take advantage of async/await. That said, if it works in pure powershell, that's fine too possibly. I just worry about perf. The listener has to be listening on a background thread. That way the prompt isn't blocked.

I really like this model using Register-ObjectEvent over managing a runspace pool. So much cleaner.

[Tiberriver256]

Gotcha yeah, I've fought the async fight in PowerShell for awhile and there really isn't any great equivalent. If we can limit the c# to just that single core class I think we should be fine right?

[TylerLeonhardt]

Oh yeah I totally agree. The C# isn't really doing anything complex - and it's super important to being async.

[Tiberriver256]

All set to close this one? I think we should open a new issue to talk about what we can do to reduce the complexity. As it stands we have:

----Main Thread----- ----ListenerRunspace---- ----ExecutionRunspace----

It would be really nice to only have two threads or keep everything in the main thread somehow. I would love to be able to set breakpoints and I'm sure others would too.

[TylerLeonhardt]

Yep! We can do that. I think if we went the Register-ObjectEvent route, we'd be able to do breakpoints and such

[TylerLeonhardt]

Updated #93 👍