Add support for .NET Core

[daveaglick]

Converted to netstandard2.0/netcoreapp2.0:

Clients

• [x] Cake.Wyam

2.x client strategy is discussed in #668

Core

• [x] Wyam.Common
• [x] Wyam.Configuration (need to look closely at custom assembly resolution)
• [x] Wyam.Core
• [x] Wyam.Hosting (need to investigate LiveReload support in .NET Standard)
• [x] Wyam.Testing
• [x] Wyam.Testing.JavaScript

Extensions

• [x] Wyam.AmazonWebServices
• [x] Wyam.CodeAnalysis
• [x] Wyam.Feeds
• [x] Wyam.GitHub
• [x] Wyam.Highlight
• [x] Wyam.Html
• [x] Wyam.Images
• [x] Wyam.Json
• [x] Wyam.Less
• [x] Wyam.Markdown
• [x] Wyam.Minification
• [x] Wyam.Razor
• [x] Wyam.Sass
• [x] Wyam.SearchIndex
• [x] Wyam.Tables
• [x] Wyam.Xmp
• [x] Wyam.Yaml
• [x] Wyam.YouTube

Recipes

• [x] Wyam.Blog
• [x] Wyam.BookSite
• [x] Wyam.Docs
• [x] Wyam.Web

Client Tests

• [x] Cake.Wyam.Tests

Core Tests

• [x] Wyam.Common.Tests
• [x] Wyam.Configuration.Tests
• [x] Wyam.Core.Tests
• [x] Wyam.Hosting.Tests

Extension Tests

• [x] Wyam.CodeAnalysis.Tests
• [x] Wyam.Highlight.Tests
• [x] Wyam.Html.Tests
• [x] Wyam.Json.Tests
• [x] Wyam.Markdown.Tests
• [x] Wyam.Minification.Tests
• [x] Wyam.Razor.Tests
• [x] Wyam.Sass.Tests
• [x] Wyam.Tables.Tests
• [x] Wyam.Xmp.Tests
• [x] Wyam.Yaml.Tests
• [x] Wyam.YouTube.Tests

Integration Tests

• [ ] Wyam.Examples.Tests

[gijswijs]

Just to be helpful: OctopusDeploy release a tool to help with migrating to Core:

https://icanhasdot.net/result?github=Wyamio~2FWyam

[daveaglick]

@gijswijs Thanks, I'll check that out! I'm hoping to start this effort fairly soon. It looks like many of the libraries we depend on have either ported or are in the process.

[airtonix]

Yep, i'd love to use this inside docker/linux

[Vladekk]

Any progress on this yet? Docker is on the rise, so Core support would be awesome.

[pauldotknopf]

I'd be willing to help out to add .NET Core support.

[williamwgant]

Is there a list floating around somewhere of what has to be fixed for Wyam to run on core? Is it possible to generate one somehow?

[pauldotknopf]

@daveaglick Can I get started on this now? I need to make wyam as a part of the build process, and I want to use a docker image for the build server :)

[pauldotknopf]

I'd probably get the core/tests done first. Then, module-by-module, we will inspect dependencies and determine if we can support .NET Core. I remember seeing a few dependencies that didn't have .NET Core support.

[daveaglick]

@pauldotknopf Be my guest! I haven't dug too deeply into it because I suspect some of the assembly context and application domain loading stuff may be a pain until those are introduced in netstandard 2.0. That said, if you're able to get around it, I'd be happy to make some progress here.

[daveaglick]

@pauldotknopf Good call - most of the challenging stuff is going to be in Wyam.Configuration. If you can get some of the other libraries moved over to netstandard 1.5 now that'll reduce the lead time in any case. And it shouldn't be too disruptive since I just updated everything to Framework 4.6.2 which can consume netstandard 1.5 libraries.

[philippgille]

@pauldotknopf @daveaglick Hi, is there any progress regarding Wyam on .NET Core? I didn't try it out, but the docs still say that Wyam runs on .NET Framework 4.6.2

[daveaglick]

@philippgille That's correct, it's still full-framework only. Work is progressing on .NET support but it'll still be another couple months.

We need to convert all the extension libraries to netstandard first, then can convert the core Wyam libraries. netstandard 2.0 is a blocker for some of the core libraries given the way assemblies are dynamically loaded.

Rest assured that .NET Core support is definitely high on the priority list.

[philippgille]

The blocking feature is probably System.AppDomain? This document mentions that AssemblyLoadContext is recommended now, although it doesn't say if it's recommended over AppDomain in general or only until netstandard 2.0 supports it.

Quote:

AppDomains can be used for different purposes on the .NET Framework. For code isolation, we recommend separate processes and/or containers as an alternative. For dynamic loading of assemblies, we recommend the new AssemblyLoadContext class.

I didn't work with any of the two namespaces yet, so I don't know if it's a viable alternative for Wyam (or its dependencies).

[MaximRouiller]

@daveaglick so.............. ? You waiting for the next major release buddy? 😉 😛

/poking

[daveaglick]

@MaximRouiller Waiting on an enterprising developer to drop a full porting PR in my lap. Hasn't happened yet, but doesn't mean it won't :wink: :wink:

[MaximRouiller]

LOLLL

So... apparently, I'm going to be taking vacations soon. How long do you think this is going to take?

[MaximRouiller]

Or rather, what are the "steps" that are necessary to do and how many can I do in a weekend?

[daveaglick]

To be honest, I'm waiting on netstandard 2.0 to drop. There's a lot going on under the hood with regard to assemblies, dynamic compilation, and binding and 2.0 should have the reflection parts to make that much easier to port. Don't want to spend a ton of time on workarounds now if it'll mostly work with minimal changes soon.

[MaximRouiller]

So... it's officially "waiting on 2.0" status.

[daveaglick]

Yeah. I just don't have the time to go chasing the bleeding edge.

[MaximRouiller]

Yeah, no point in stabbing ourselves in the face with that edge. 😛

[williamwgant]

According to this:

https://github.com/dotnet/core/blob/master/roadmap.md

They drop a preview in Q2, with the thing going live in Q3. Assuming less code churn in v2 than in the initial alpha/beta/rc run for core, that means we could be hitting this fairly soon.

[pauldotknopf]

Ya, I don't see any reason why this can't be started now.

I have been watching the changes in corefx/coreclr. The work has been really adding back old api's that don't exist for coreclr yet.

[pauldotknopf]

Many 3rd party libraries are now supporting .NET Core. The water feels good. Come in.

[williamwgant]

What would be the process to do that with this codebase? I've coded in core, but haven't ported anything to it. Do we just pull down the solution and start switching project properties to use netstandard? My impression was that project structures changed significantly going to core and that Microsoft (probably) didn't provide any kind of migration tool. Has that changed?

[pauldotknopf]

I have tried the migration stuff in the cli, but it is garbage. It adds a bunch of needless stuff. What you want is a pure/core storing point.

Create a new Wyam.Common project, and copy the csproj into the existing Wyam.Common directory. The code should compile, but with errors. From then, just add references to the packages required, and massage and code that needs to be updated (should be minimal).

Once you have the Wyam.Common project done, you can still reference that csproj file (even though it is .NET Core) from all the existing non-.NET Core projects (add ons, Wyam.Configuration, etc).

It should be an easy and gradual process, with each step being compilable and runnable.

We probably need to identify the order at which the projects should be done.

• Wyam.Common
• Wyam.Core
• TBD

I'd recommend leaving the Wyam.Configuration until later, as you'd probably need the new AppDomain stuff coming. What that means is that initially, the .NET Core version would only support usage directly from the API. No "recipes", etc.

[pauldotknopf]

Rough plan for the order of attack.

• Wyam.Common
• Wyam.Core
• Wyam.Hosting
• Any plugin that has dependencies that can be met for .NET Core.

[pauldotknopf]

We should probably created a matrix of Wyam extensions and dependencies, and their support for .NET Core. @daveaglick Maybe create a Wiki that we can work on?

[daveaglick]

Yeah, I wouldn't touch the automatic migration. It's going to be multi-stage (I've been thinking about the process in preparation for a while). @pauldotknopf has the right idea. Wyam.Common has to go first since it's consumed by everything. The trick is that ported libraries will be able to be used by unported ones but not the other way around.

Here's my thinking in terms of order:

• Wyam.Common (shouldn't be too bad, mostly interfaces and abstractions)
• All module and extension projects that have only netstandard dependencies
• Recipe projects (these can only be done after their dependent module libraries)
• Wyam.Hosting, Wyam.Testing, Wyam.Configuration (that last one may be rough)
• Wyam.Core (do this last since nothing directly consumes it)
• Console app (not sure if I want to multi-target, do a dotnet "tool" alias, or what yet)

A matrix is spot on. I suspect we'll probably end up with at least a couple modules that end up having to stay on full framework, at least initially. That's okay - the whole NuGet/addin system in Wyam should deal with that no problem, but we'll at least want to document it. I don't want to turn the GH wiki on since it might cause confusion with regard to documentation, but I'll add a Markdown file to the root everyone can PR against.

[daveaglick]

Also note that as part of this process I'm probably going to remove Wyam.Windows. I'd prefer to get a more integrated NuGet-powered cross-platform upgrading system in place. That's for another issue though.

[daveaglick]

https://github.com/Wyamio/Wyam/blob/develop/CorePort.md

[MaximRouiller]

Look at me rolling the ball on things I haven't even git clone yet. 😜

[MaximRouiller]

Thanks @daveaglick for the matrix! 😃

[daveaglick]

This may be relevant when it comes time to tackle clients: https://nblumhardt.com/2017/04/netcore-msi/

[daveaglick]

It looks like based on https://github.com/aspnet/Home/issues/2022#issuecomment-299548928 that Wyam.Hosting may need to target netcoreapp2.0 because of Kestrel. That'll cascade to consuming apps/libs, so the Wyam client will also need to target netcoreapp2.0. That probably forces our hand for the kinds of clients we'll be able to build - it's .NET Core or go home (I.e., no multi-targeting to .NET Framework for distribution).

Thankfully it sounds like Razor will at least continue to target netstandard so the fallout should be limited to the preview server and client. Once all the ASP.NET Core changes are made we'll need to evaluate any other utility libraries we're bringing in from ASP.NET Core for other extensions and transition away from the ASP.NET Core libraries if they target netcoreapp2.0.

I can probably live with the preview server and client targeting netcoreapp2.0 only, but I would really like to continue targeting netstandard for extensions to support the broadest range of embedded scenarios.

[daveaglick]

A few updates:

• Got word directly from Scott Hunter yesterday that the intent is for Kestrel to target netstandard after all. That still leaves the executable story TBD (should it be a .NET Core app, should we also do a .NET Framework release, etc.) but at least the supporting library story is back to normal.
• There will be some issues with native libraries. Those get placed in a runtimes folder in NuGet packages and are typically copied to the app folder based on the RID (see here and here). That all happens at package installation so it's currently unclear how we will support that inside Wyam.Configuration for dynamically installed packages. Still thinking about this.

[daveaglick]

Follow-up to unmanaged assembly loading - it looks like it can be done by implementing a small special-purpose AssemblyLoadContext. Also see here for lots of information about the process in general.

So if I've got this right, we need to:

• Scan installed NuGet packages (including their dependencies down the heirarchy) for a runtimes folder
• If one is found, try to find a valid RID (hopefully some APIs for RID matching)
• When we find the right runtimes RID subfolder, load all .dll files as unmanaged directly from their paths (no copying needed)

[Daniel15]

In the meantime, what's the story with Mono? Does Wyam run in Mono or are there known issues? A lot of .NET Framework apps run fine on Mono, but some apps have issues (particularly if they mess with native code)

It looks like based on https://github.com/aspnet/Home/issues/2022#issuecomment-299548928 that Wyam.Hosting may need to target netcoreapp2.0 because of Kestrel. That'll cascade to consuming apps/libs, so the Wyam client will also need to target netcoreapp2.0. That probably forces our hand for the kinds of clients we'll be able to build - it's .NET Core or go home (I.e., no multi-targeting to .NET Framework for distribution).

You can target multiple platforms with one csproj. With ReactJS.NET and JSPool I'm targeting both net40 and netstandard1.6:

<TargetFrameworks>net40;netstandard1.6</TargetFrameworks>

You can just as easily target both netstandard1.6 and netcoreapp2.0 if needed. You can use preprocessor directives like #if to handle any differences in code - in ReactJS.NET I'm using preprocessor directives to help with sharing code across ASP.NET MVC 5 and ASP.NET Core MVC 1.0.

[daveaglick]

@Daniel15 The short story is that Mono doesn't work right now. There's a hard-coded check to exit with an error message if Wyam detects it's being run on Mono since I know it won't work and that reduces support issues.

The long story is that Wyam does some stuff with assembly loading and target framework detection that Mono doesn't seem to support the correctly. I've gotten part of the way towards a resolution but haven't had time to get it fully resolved. I'd like to though - it would be nice to have Mono support in place when I flip the switch to 1.0. I'll probably take one more stab at it.

[daveaglick]

@Daniel15 With regard to Kestrel and target frameworks, we thankfully won't need to worry about multi-targeting. The ASP.NET team has walked back their whole "Kestrel is netcoreapp2.0 only" stance and made it clear that it was only netcoreapp for the betas and would target netstandard by the time 2.0 is final. That's great news for Wyam because it means all the supporting libraries except the actual "app(s)" can target netstandard.

[pauldotknopf]

I said it before and I will say it again.

For static code html generation, this dynamic loading/nuget/appdomain/cake stuff is a bit overkill. Give people a cross-plat API, and let edge projects handle platform glue.

The concept of a static code generation is really simple, yet it currently seems very complicated.

[daveaglick]

@pauldotknopf

For static code generation, this dynamic loading/nuget/appdomain/cake stuff is a bit overkill

Not sure I follow. Makes sense for cases where you're doing code-gen to include artifacts in some other build process (such as T4), but in this case the code gen is to support a DSL that gets compiled and evaluated. Without support for dynamic loading, NuGet declarations, and the like there would be no way to say "this particular Wyam project needs to use this special package that isn't part of the base distribution" without managing the package dependency by hand. And we'd still have to support adding it as a dynamic reference even then so it could be used from the config file.

Wondering if your thought process is that perhaps there shouldn't even be a Wyam config file and configuring/controlling Wyam execution should/could be handled directly via an API and it's up to the user to determine if they want to use a script, LINQPad, a full project, etc. to call that API?

[pauldotknopf]

Not sure I follow.

Sorry, when I said "code-gen", I meant "html-gen". Converting markdown files to html is a very simple process.

perhaps there shouldn't even be a Wyam config file and configuring/controlling Wyam execution should/could be handled directly via an API and it's up to the user to determine if they want to use a script, LINQPad, a full project, etc. to call that API?

Exactly. I really need Wyam to run on .NET Core, and it is crazy to me that it can't be done because of this platform glue. The power of Wyam is it's (genius) data model (pipeline/streams) and plugin/extension support. We are needlessly stuck on certain platforms because of glue code. This glue code should exist in other repositories/projects, where the weight of the glue doesn't effect the portability of the core API.

[pauldotknopf]

Sorry. I know I have hammered this point before, but @Daniel15 triggered me.

[Silvenga]

@daveaglick before the move to .NetCore/.NetStandard, Wyam should probably upgrade to the new project SDK - which can be done now (while keeping the .NetFull target). Then the different projects can be slowly migrated to .NetStandard one at a time without impact (should be a drop in replacement in the most cases). Finally the executable projects can move to a multi-target of .NetFull + .NetCore.

.NetStandard 2.0 significantly simplifies this migration with the .NetFull shim.

I've been trying to get a couple of projects upgraded, e.g. https://github.com/raspberry-sharp/raspberry-sharp-io/pull/92

I propose a three stage migration.

• Upgrade library projects to new SDK structure - still targeting dotnet 4.0.
• Move library projects to target dotnet standard 1.6 (or a different level).
• Move the rest of the projects (tests, etc.) to new SDK structure, keep same targets.

[MaximRouiller]

@daveaglick I've seen that done without any major problems on this project:

https://github.com/terrajobst/nquery-vnext/commits/master

/ping @terrajobst

[bbqchickenrobot]

Any updates? Is there a branch that is in progress re: the netcore update? I wouldn't mind helping out =)

[daveaglick]

@bbqchickenrobot Making some pretty big progress lately. The last couple weeks leading up to the 1.1.0 release earlier this week have been about switching every project (>40 of them) over to the new SDK MSBuild project system. That paves the way as a first step towards converting to .NET Standard/.NET Core since those targets require the new project format.

Next step is to port Wyam.Common to .NET Standard and then all the other extension libraries can be ported.

[MaximRouiller]

@daveaglick I ran out of positive emoji.

Here. Have a pizza. 🍕

[Silvenga]

At some point it might be prudent to remove the transitive nuget references in the .csproj's.

For example:

Wyam.Razor needs Microsoft.AspNetCore.Mvc.Razor Microsoft.AspNetCore.Mvc.Razor needs Microsoft.CodeAnalysis.CSharp Microsoft.CodeAnalysis.CSharp needs Microsoft.CodeAnalysis.Common

Because the new SDK automatically handles these transitive dependencies automatically we no longer need to reference Microsoft.CodeAnalysis.Common directly.

This will improve future maintainability and prevent bugs in the future.