WASI support tracking - Githubissues

SteveSandersonMS commented 2 years ago

Description

This issue is to track known issues in the early WASI-enabled runtime builds. These need to be resolved in order to have a proper supportable WASI-ready release.

[x] Add wasi-wasm RID
- https://github.com/dotnet/runtime/issues/77780
- https://github.com/dotnet/runtime/issues/78389 API change
[x] Build System.Native using WASI SDK instead of using the Emscripten-built binary.
- MSbuild files for it and generate wasm_m2n_invoke.g.h
- Enable p/invokes to all APIs exposed by System.Native. Currently there's a small hardcoded list.
- https://github.com/dotnet/runtime/pull/79046
[x] Create nuget package for WASI-SDK (dotnet/wasi-sdk)
- [ ] #95086
[x] Add WASI runtime as XHarness target
- https://github.com/dotnet/xharness/pull/977
[x] Enable GC finalizers. Currently the finalizer thread isn't started (see mono_gc_finalize_notify in gc.c).
- fix stack range https://github.com/dotnet/runtime/pull/80456
[ ] Support .NET threads on threading-enabled wasm runtimes. This includes updating the synthetic-pthread.c code to support having separate pthread keys for each thread instead of a single global list.
[x] #95137
[x] create ICU transport package and use it
https://github.com/dotnet/icu/pull/307
https://github.com/dotnet/icu/pull/309
https://github.com/dotnet/runtime/pull/81345
[ ] make it work on wasmedge https://github.com/dotnet/dotnet-wasi-sdk/issues/5 https://github.com/dotnet/runtime/issues/85429
[ ] make it work on wasmer https://github.com/dotnet/dotnet-wasi-sdk/issues/6
[x] #86984
[x] #86985
[x] do we need vitual file system ? https://github.com/dotnet/runtime/issues/81418
[ ] https://github.com/dotnet/runtime/issues/94803
[ ] Mono AOT Support
- [x] https://github.com/dotnet/runtime/pull/95051
- [x] https://github.com/dotnet/runtime/pull/95074
- [ ] #95136

Build

[x] Have a real CI build (including some test) ~and a public distribution mechanism.~
- [x] https://github.com/dotnet/runtime/issues/72641
- [x] https://github.com/dotnet/runtime/pull/81052
[x] Produce binaries into experimental workload , in similar way as we do for multi-threading WASM (https://github.com/dotnet/runtime/pull/81849)
[x] wasi templates (https://github.com/dotnet/runtime/pull/81849)
[x] WasmAppHost support (https://github.com/dotnet/runtime/pull/81849)
[x] wasi.build.tests (https://github.com/dotnet/runtime/pull/81849)
[x] App builder, merged targets etc (https://github.com/dotnet/runtime/pull/81849)

ghost commented 2 years ago

Tagging subscribers to 'arch-wasm': @lewing See info in area-owners.md if you want to be subscribed.

Issue Details

### Description This issue is to track known issues in the early WASI-enabled runtime builds. These need to be resolved in order to have a proper supportable WASI-ready release. - [ ] Support .NET threads on threading-enabled wasm runtimes. This includes updating the `synthetic-pthread.c` code to support having separate pthread keys for each thread instead of a single global list. - [ ] Enable GC finalizers. Currently the finalizer thread isn't started (see `mono_gc_finalize_notify` in `gc.c`). - [ ] Enable p/invokes to all APIs exposed by `System.Native`. Currently there's a small hardcoded list. - [ ] Build `System.Native` using WASI SDK instead of using the Emscripten-built binary. - [ ] Fix ALC support, which in turn probably means fixing something about stack walking. See the WASI comment in `ves_icall_System_Reflection_Assembly_InternalLoad`. ### Reproduction Steps - ### Expected behavior - ### Actual behavior - ### Regression? _No response_ ### Known Workarounds _No response_ ### Configuration _No response_ ### Other information _No response_

Author:	SteveSandersonMS
Assignees:	-
Labels:	`arch-wasm`, `area-Build-mono`
Milestone:	-

mhmd-azeez commented 1 year ago

Is exporting functions to wasi tracked here too?

pavelsavara commented 1 year ago

Is exporting functions to wasi tracked here too?

Added

mhmd-azeez commented 1 year ago

@pavelsavara Thanks! We also need the opposite: Exporting functions so that you can create a .NET library and compile it to wasm and consume it from the other languages. It's already possible from Go, C, C++, and Rust. See this sample: https://github.com/extism/csharp-pdk/blob/master/samples/SamplePlugin/Functions.cs#L9

Currently you have to write the C glue code manually: https://github.com/extism/csharp-pdk/blob/master/samples/SamplePlugin/native.c#L11

pavelsavara commented 1 year ago

We also need the opposite: Exporting functions

That would be [UnmanagedCallersOnly] and I think both will need further discussion on design. Note, I don't know how large portion of above would be in Net8.

Also there trouble is sharing memory between caller and called code. We will implement this assuming shared memory (with the host engine) and passing pointers.

And the "WASI component model" with isolated memory, lowering/lifting etc will need different attributes in the future future.

SteveSandersonMS commented 1 year ago

Some issues I've noticed when trying out a nightly build:

[ ] Doesn't work on most non-wasmtime runtimes because it relies on sock_accept by default. The previous prototype SDK avoided this by not compiling in the sock_accept dependency except when a "start listening for debugger connections" flag was given at build time.
[ ] The default experience of not requiring the WASI SDK and producing a dotnet.wasm file instead of YourApp.wasm is counterproductive in my opinion. Producing a dotnet.wasm file plus YourApp.dll doesn't match anyone's default use case that I know of, so I don't know why we'd do that - it's just confusing. I think it's hugely more valuable for the default experience to be a single-file YourApp.wasm, which entails requiring the WASI SDK, so we should just put that in the standard WASI workload.
[ ] If you build with WasmSingleFileBundle=true, then do dotnet run, it fails and logs System.Text.Json.JsonException: 'F' is invalid after a single JSON value
[ ] When using WasmSingleFileBundle=true, if you haven't set WASI_SDK_PATH, it suggests you could get the WASI SDK using a workload but I don't think that's implemented yet, right?
[ ] Is there a convenient way to run the trimmer on a regular build without doing a publish? I ask because for a lot of WASI scenarios people will strongly value producing a smaller binary even for regular dev builds, and forcing people to go through a publish flow on the inner loop is not ideal. The trimmer is easily fast enough to be part of most inner loop workflows on small apps, so I think there should at least be the option of enabling trimming on build as an MSBuild property. (The previous prototype SDK did this.)
[ ] The build layout bin\Debug\net8.0\wasi-wasm\AppBundle\YourApp.wasm is unnecessarily complicated. Can we reduce this to bin\Debug\net8.0\wasi-wasm\YourApp.wasm? Given that the primary case is single-file deployment, we don't need to create a separate directory structure for the one file, and doing so does not feel idiomatic for .NET apps.
[ ] dotnet run produces a bunch of extra output (e.g., it logs the wasmtime command it used and the working directory) which should be removed before we ship

SteveSandersonMS commented 1 year ago

I've been trying out the pinvoke generation and found it can't yet generate real WebAssembly imports. There are two main reasons:

It ignores pinvokes unless they correspond to an entry on _WasmNativeFileForLinking (even though when you want to generate an import, you won't be linking with any module for it)
When generating pinvoke-table.h, it never adds any import_module/import_name clang attributes, which means:
- The module is always compiled as env (so it's impossible to reference most real-world libraries)
- The symbol name is mangled

I was able to work around it via some MSBuild hackery, so can be fairly confident what would be the steps to solve this:

Change the pinvoke generator so that when a DllImport's module name does not correspond to an entry in _WasmNativeFileForLinking, it does still generate the pinvoke-table.h entry
... but those pinvoke-table.h entries must also be annotated with import_module/import_name

For example, given these two DllImports:

[LibraryImport("libSystem.Native")]
private static partial int Method1(int a);

[LibraryImport("mylib", EntryPoint = "this:should:not:be:mangled")]
private static partial int Method2(int a);

... the pinvoke-table.h definition should look like:

int Method1 (int);

__attribute__((import_module("mylib"), import_name("this:should:not:be:mangled"))) int Method2 (int);

The first one does not need any __attribute__ because libSystem.Native does match up with one of the linker inputs (i.e., _WasmNativeFileForLinking entries). The existing codegen is correct.

The second one does need the clang import annotations because it doesn't match a _WasmNativeFileForLinking entry, so it has to become a real wasm import, and it's important to retain the real module name (mylib) and the real import name (this:should:not:be:mangled) even though it's not a legal C function name.

davidfowl commented 1 year ago

Excellent @SteveSandersonMS! Are these going to be addressed? I’m looking forward to this WASM story out of the browser being easy and solid. Great progress folks!

SteveSandersonMS commented 1 year ago

Oh one other thing: just noticed that having <RuntimeIdentifier>wasi-wasm</RuntimeIdentifier> causes VS to think it's an ASP.NET Core project, which results in:

It generates Properties\launchSettings.json, including some settings about URLs
When you run the project (e.g., Ctrl+F5), it auto-minimizes the console window as if it's a web server, stopping you from easily seeing the output

I don't think either of these should happen - it should be treated like a Console or Class Library project unless you reference a web SDK.

lewing commented 1 year ago

Excellent @SteveSandersonMS! Are these going to be addressed? I’m looking forward to this WASM story out of the browser being easy and solid. Great progress folks!

Yes, this is the next step of the Wasi work

lewing commented 1 year ago

Oh one other thing: just noticed that having <RuntimeIdentifier>wasi-wasm</RuntimeIdentifier> causes VS to think it's an ASP.NET Core project, which results in:

It generates Properties\launchSettings.json, including some settings about URLs

When you run the project (e.g., Ctrl+F5), it auto-minimizes the console window as if it's a web server, stopping you from easily seeing the output

I don't think either of these should happen - it should be treated like a Console or Class Library project unless you reference a web SDK.

@maraf please take a look

bobby commented 1 year ago

@SteveSandersonMS Are there any plans to support the Wasm Component Model in .NET 8? Supporting this standard would allow Wasm components written in .NET languages to interoperate with components written in other languages.

The docs and discussion around the WIT bingen project (an effort to make it easy to write/build guest components in various languages) briefly describes some of the issues involved with supporting a dynamic language runtime (vs. the statically compiled langs already supporting building Wasm/WASI components like C/C++, Rust, Go, etc.).

But perhaps there is another path related to the AOT/Native compilation work featured in .NET 8? Perhaps .NET's WASI/component support could compile AOT/Native or with a minimal runtime/GC (similar to TinyGo) to produce small, interoperable Wasm components with fast startup?

lambdageek commented 1 year ago

@SteveSandersonMS Are there any plans to support the Wasm Component Model in .NET 8?

@bobby The new .NET 8 features are more or less locked down at this point. But we are looking at WASM Interface Types and the Component Model for future .NET releases

MichaelPeter commented 1 year ago

Since this was not mentioned in any preview posts and we are approaching the Release Candidates, what will be the status of wasi in .NET 8?

Will it be "just" part of an experimental workload or will it be considered part of .NET 8 or part of the .NET 8 SDK?

Meai commented 1 year ago

Is there a way to use c# to produce .wasm files that are small enough to run on the edge (like e.g cloudflare workers)? Because right now a c# hello world .wasm with WasmSingleFileBundle=true is 23mb, which is far too big. In comparison the competitor golang has ~1.5mb output for a hello world by default and they have an alternative runtime called tinygo that has even tinier outputs, not sure about ecosystem compatibility of tinygo but still: https://www.reddit.com/r/golang/comments/15yi9ub/comment/jxdylsc/ The cloudflare worker size limit is 10mb and each megabyte adds to the startup time.

danmoseley commented 1 year ago

@lewing can you help answer the q above about experimental status?

maraf commented 1 year ago

Is there a way to use c# to produce .wasm files that are small enough to run on the edge (like e.g cloudflare workers)? Because right now a c# hello world .wasm with WasmSingleFileBundle=true is 23mb, which is far too big.

These trimming settings can get you to ~ 2.7MB for wasiconsole template

<WasmSingleFileBundle>true</WasmSingleFileBundle>
<InvariantGlobalization>true</InvariantGlobalization>
<TrimMode>full</TrimMode>
<DebuggerSupport>false</DebuggerSupport>
<EventSourceSupport>false</EventSourceSupport>
<StackTraceSupport>false</StackTraceSupport>
<UseSystemResourceKeys>true</UseSystemResourceKeys>
<NativeDebugSymbols>false</NativeDebugSymbols>

Documentation is here https://learn.microsoft.com/en-us/dotnet/core/deploying/trimming/trimming-options#trimming-framework-library-features

mhmd-azeez commented 1 year ago

@maraf I tried out the configurations you provided, but I still get a ~23mb .wasm file:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <RuntimeIdentifier>wasi-wasm</RuntimeIdentifier>
    <OutputType>Exe</OutputType>
    <PublishTrimmed>true</PublishTrimmed>

    <WasmSingleFileBundle>true</WasmSingleFileBundle>
    <InvariantGlobalization>true</InvariantGlobalization>
    <TrimMode>full</TrimMode>
    <DebuggerSupport>false</DebuggerSupport>
    <EventSourceSupport>false</EventSourceSupport>
    <StackTraceSupport>false</StackTraceSupport>
    <UseSystemResourceKeys>true</UseSystemResourceKeys>
    <NativeDebugSymbols>false</NativeDebugSymbols>
  </PropertyGroup>
</Project>

I have tried all of these:

dotnet build
dotnet build -c Release
dotnet publish -c Release
dotnet publish -c Release -r wasi-wasm

Note: dotnet publish doesn't create a .wasm file at all, only a dotnet.wasm file

.NET SDK version: 8.0.100-rc.2.23502.2

Meai commented 1 year ago

true @mhmd-azeez, I tried it as well. It still produces a 23mb file for me.

maraf commented 1 year ago

That should work. Can you please open a new issue (so that we don't polute this one tracking general features) and share outputs for dotnet workload list and dotnet --info and version of WASI SDK you're using? Thanks

EDIT: The build/publish output is in the AppBundle folder (eg. .\bin\Release\net8.0\wasi-wasm\AppBundle)

Note: dotnet publish doesn't create a .wasm file at all, only a dotnet.wasm file

~~The WasmSingleFileBundle=true means that everything is wrapped in the dotnet.wasm~~

EDIT: In case of WasmSingleFileBundle=true the resulting wasm file is named after project (eg. SizeApp.wasm)

mhmd-azeez commented 1 year ago

@maraf done: https://github.com/dotnet/runtime/issues/94286

sec commented 1 year ago

@lewing can you help answer the q above about experimental status?

Waiting for this :)

Meai commented 1 year ago

I really don't understand the logic though of why it would only do this optimization during a publish or why the smaller bundled SizeApp.wasm file doesn't appear in the publish folder even though I just told it where I want to publish into. Then there is another dotnet.wasm file of 10mb in the folder I published

lewing commented 1 year ago

@lewing can you help answer the q above about experimental status?

Waiting for this :)

Sorry for the delay, personal stuff came up and haven't completely caught up with the backlog. wasi-wasm targets are still experimental with no support guarantees in net8.0 but we do plan on doing additional work in net9.0. Information on what your use cases are will help us prioritize this work so please let us know what you're looking for.

cmadh commented 1 year ago

I think it would be cool to have the ability to completely disable Exception-Handling and the GC or other parts of the framework that blow up wasm-size and slow down startup-time. Chrome for example now has WasmGC and other hosts will likely have their own GC-implementations, too. For fire-and-forget-style exectutions where a really small wasm with fast startup-time is needed like cloudflare workers or (I personally think this case is much more important) smart contracts, this could be a great advantage and further diversify the use of C#/.NET.

michaelpbean commented 1 year ago

Is it possible to define import memory with wasi-wasm (with C glue code or otherwise)? And if so, are there any examples on how to access imported memory? I've used attribute((import_module("..."), import_name("..."))) and attribute((export_name("..."))) for importing/exporting functions, but haven't seen any documentation or examples on importing and exporting wasm Memories/Tables/Globals.

Similarly, is it possible to adjust the size of the exported memory? By default it appears to export 800 pages, but is there a setting to customize this?

RReverser commented 1 year ago

Couldn't find a good place to ask this and not sure if it's worth raising an issue, so for now I'll just ask here: why does wasi-experimental workload pull in a bunch of seemingly unrelated targets upon installation?

> dotnet workload install wasi-experimental

Downloading microsoft.net.sdk.android.manifest-8.0.100.msi.x64 (34.0.43)
Installing microsoft.net.sdk.android.manifest-8.0.100.msi.x64 .... Done
Downloading microsoft.net.sdk.ios.manifest-8.0.100.msi.x64 (17.0.8478)
Installing microsoft.net.sdk.ios.manifest-8.0.100.msi.x64 .... Done
Downloading microsoft.net.sdk.maccatalyst.manifest-8.0.100.msi.x64 (17.0.8478)
Installing microsoft.net.sdk.maccatalyst.manifest-8.0.100.msi.x64 .... Done
Downloading microsoft.net.sdk.macos.manifest-8.0.100.msi.x64 (14.0.8478)
Installing microsoft.net.sdk.macos.manifest-8.0.100.msi.x64 .... Done
Downloading microsoft.net.sdk.maui.manifest-8.0.100.msi.x64 (8.0.3)
Installing microsoft.net.sdk.maui.manifest-8.0.100.msi.x64 .... Done
Downloading microsoft.net.sdk.tvos.manifest-8.0.100.msi.x64 (17.0.8478)
Installing microsoft.net.sdk.tvos.manifest-8.0.100.msi.x64 .... Done
Downloading microsoft.net.sdk.aspire.manifest-8.0.100.msi.x64 (8.0.0-preview.1.23557.2)
Installing microsoft.net.sdk.aspire.manifest-8.0.100.msi.x64 .... Done
Downloading Microsoft.NET.Runtime.WebAssembly.Wasi.Sdk.Msi.x64 (8.0.0)
Installing Microsoft.NET.Runtime.WebAssembly.Wasi.Sdk.Msi.x64 ..... Done
Downloading Microsoft.NETCore.App.Runtime.Mono.wasi-wasm.Msi.x64 (8.0.0)
Installing Microsoft.NETCore.App.Runtime.Mono.wasi-wasm.Msi.x64 ....... Done
Downloading Microsoft.NET.Runtime.WebAssembly.Templates.Msi.x64 (8.0.0)
Installing Microsoft.NET.Runtime.WebAssembly.Templates.Msi.x64 .... Done
Downloading Microsoft.NET.Runtime.MonoAOTCompiler.Task.Msi.x64 (8.0.0)
Downloading Microsoft.NET.Runtime.MonoTargets.Sdk.Msi.x64 (8.0.0)

Successfully installed workload(s) wasi-experimental.

Android, iOS, tvOS etc have no relation to WASI or Wasm, so... are they pulled because they're implicit dependencies of Mono itself? Or some other reason?

lewing commented 1 year ago

dotnet workload install will by default update all the manifests which is what you are seeing. This behavior is expected and follows the current design but will likely change in important ways soon (8.0.xx) when workload sets are released.

MichaelPeter commented 1 year ago

@lewing can you help answer the q above about experimental status?

Waiting for this :)

Information on what your use cases are will help us prioritize this work so please let us know what you're looking for.

Here are the scenarios how we would like to use Wasi:

Service-Mesh without Kubernetes: Cloud: Since Databases/Message are Native Azure Resources, all our Microservices could be written in WASI without having to have a kubernetes.

Edge/Customers with self hosted environmenments where kubernetes is problematic: We primarily have cloud offerings, but some customers cannot have their data in the cloud and have their own self hosted environments: That would be for example Hospitals in some countries which do not want their patient data in the cloud, or for example Patent lawyers, who have unreleased Patents which could be of Interest to Microsoft/Amazon/Google and customers of these Patent lawers require in contracts that unreleased Patents not to be uploaded in any Cloud Service.

Some of these Customers do just have 2-3 IT Personell and don't want to service their own kubernetes cluster, here a service mesh based on wasi/Wasmtime would be very interesting. Even for us automating the setup of a self hosted kuberentes cluster is not easy even with modern Kubernetes Solutions

Also is it not always possible to have Linux VMs with a Kuberentes in an Windows Only Environment. Be it for compatibility with customer required mandatory virus scanners, lack of knowlege of Linux Administration, or Licencing Situations which only allow Windows VMs and not Linux VMs (Windows Server, VMWare).

Serverless Scale down to (near)Zero "Azure Functions" with wasi. Since wasi starts so fast, this should be possible. Alternativly Serverless KEDA Wasi Functions inside Kubernetes to be multicloud capable

davidfowl commented 1 year ago

Cloud: Since Databases/Message are Native Azure Resources, all our Microservices could be written in WASI without having to have a kubernetes.

@MichaelPeter Can you help me understand how WASI helps in the 1st case over just running dotnet on the hardware today? I see the benefit for platform owners that want to run anything, but I'm having a hard time understanding how it helps developers? What does it mean to build and deploy a WASI service in this situation? Where are you deploying? What orchestrates your services? Without k8s what does networking?

Also is it not always possible to have Linux VMs with a Kuberentes in an Windows Only Environment. Be it for compatibility with customer required mandatory virus scanners, lack of knowlege of Linux Administration, or Licencing Situations which only allow Windows VMs and not Linux VMs (Windows Server, VMWare).

Are we talking about some new device that has a native WASM runtime? What is this exactly?

Serverless Scale down to (near)Zero "Azure Functions" with wasi. Since wasi starts so fast, this should be possible.

Lots of ideas here but WASI doesn't solve this. There are many other architectural problems to tackle with Azure functions so that this benefit can be observed.

MichaelPeter commented 12 months ago

Can you help me understand how WASI helps in the 1st case over just running dotnet on the hardware today? I see the benefit for platform owners that want to run anything, but I'm having a hard time understanding how it helps developers? What does it mean to build and deploy a WASI service in this situation? Where are you deploying? What orchestrates your services? Without k8s what does networking?

These are very good questions and I see too: what benefit has WASI that I can not already do with Native Applications?

I see Wasmtime or different Wasm runners here as Platform that provides Networking, at least for now on a single maschine for Platform Owners. But here the WASI Commuity Group is not yet far enough with Networking and many other things... Maybe there will be a more lightweight platform indipendent Kubernetes just for WASI Containers.

Alternativly frameworks like Aspire could abstract away the underlying Networking for WASI Runner-Platform and/or Kubernetes Platform. But here again, I can do that already on windows/linux too.

Where do I deploy? Everywhere, where I can start a Wasi Runner (Windows Server, Developer/Sales Laptop, Custom Hardware like Production Lines or devices which are not powerfull enough for a whole Kubernetes Clusters).

What are the Advantages for the Developer: For Testing he doesn't need a whole Kubernetes Cluster on his Developer PC, only a WASI runner.

But then still Who does Updating? there are WASI/WASM Container registries, but no Gitops Platforms yet like ArgoCD/Flux to automatically update...

But as you said these are just Ideas and no concrete use cases I am quite unhappy with my answer too and see it is not yet well thought out yet.

davidfowl commented 12 months ago

I see Wasmtime or different Wasm runners here as Platform that provides Networking, at least for now on a single maschine for Platform Owners. But here the WASI Commuity Group is not yet far enough with Networking and many other things...

I meant for distributed systems, not networking APIs that WASI may describe.

Maybe there will be a more lightweight platform indipendent Kubernetes just for WASI Containers.

Sure but that has yet to manifest. I'm also not convinced it will be that different to Kubernetes at the core.

What are the Advantages for the Developer: For Testing he doesn't need a whole Kubernetes Cluster on his Developer PC, only a WASI runner.

You can run your apps today locally on your PC without Kubernetes. This isn't a feature unique to WASI.

I'm very supportive of the work to make .NET work with WASM on the server side, but the ecosystem barely exists so the benefits called out are hypothetical and in the early idea stage. There are a few platforms and tools that can run WASM binaries but from a .NET developer point of view, it seems like an implementation detail.

SteveSandersonMS commented 12 months ago

This is a great conversation and helps to flush out a lot of the ambiguity that surrounds WASI today. It's important to know that WASI (i.e., a combination of API interface definitions for things like HTTP requests, plus tooling like wasmtime.exe for executing wasm modules with support for those APIs) does not itself replace or even compete with Kubernetes as a multi-process/machine/container app orchestrator. WASI/wasmtime has no concept of deployments, scaling, service discovery, or pretty much anything except what native code could do if you deployed it to a super minimal OS like on an embedded device. It does have the benefits of hardware abstraction, better sandboxing, and faster startup compared with a traditional container.

Long term it's completely possible that K8s devs will commonly run WASI components in preference over traditional containers within their Kubernetes clusters. Or maybe that some WASI-first K8s alternative will emerge, though as @davidfowl says it would probably look the same as Kubernetes from the point of view of the person deploying and maintaining the cluster itself. However, neither of these things are definite, and the WASI ecosystem remains at an early stage.

sec commented 12 months ago

Having possibility to run ASP.NET WebApi project on system that don't have .net supported (ex. *BSD and/or exotic arch) would be +1 for me. It was possible using https://github.com/SteveSandersonMS/dotnet-wasi-sdk and implementing http host under wasmrunner to interop with .net or using the one with WASI tcp-level where available - is/will that be planned/supported in future?

davidfowl commented 12 months ago

It does have the benefits of hardware abstraction, better sandboxing, and faster startup compared with a traditional container.

Right! I was hoping somebody would have a scenario around that.

Having possibility to run ASP.NET WebApi project on system that don't have .net supported (ex. *BSD and/or exotic arch) would be +1 for me. It was possible using https://github.com/SteveSandersonMS/dotnet-wasi-sdk and implementing http host under wasmrunner to interop with .net or using the one with WASI tcp-level where available - is/will that be planned/supported in future?

Are you expecting a very small binary with a tiny performance hit? WASI is experimental and we don't be shipping anything non-experimental on top of it until it goes stable. Our path for WASM on the server should be based on the work we've been doing to make ASP.NET Core NativeAOT compatible (and this is a long journey 😄).

RReverser commented 12 months ago

Our path for WASM on the server should be based on the work we've been doing to make ASP.NET Core NativeAOT compatible (and this is a long journey 😄).

FWIW we're currently experimenting with Wasm support via NativeAOT-LLVM experiment (https://github.com/dotnet/runtimelab/tree/feature/NativeAOT-LLVM) and had a very good experience performance-wise. The only blockers are that it's experimental and that it currently builds only on Windows. Other than that, it's pretty competitive with Wasm compiled from Rust (except for binary size, but that's not very important to us anyway), especially compared to the Mono interpreted version.

Hoping that the cross-platform variant of NativeAOT will get to similar level feature- and performance-wise someday, or that NativeAOT-LLVM experiment will make it into mainstream.

That said, in our project we're not interested in WASI but in pure Wasm as we provide custom host functions. WASI target is just means to an end because in .NET there is currently no option for wasm32-unknown-unknown targets like there is in other compiles (C / C++ / Rust / Zig), so using WASI, shimming out WASI imports by linking with custom C file, and hooking up own imports / exports is the only path for now.

sec commented 12 months ago

Right, I was just hoping to use wasi-wasm with AspNetCore more easily or at least somehow-supported. The way you wrote, is what basically @SteveSandersonMS did in his repo - would be nice to have it under .net 8 (which I was thinking this issue is also about). more docs/examples for newbie on how to get into hacking this would be 🚀

@davidfowl As for performance and binary size - that's something I wouldn't care for at the start, as having possibility to run .net code on new platform/os without "native" runtime would be more important IMO (sure NativeAOT would be way better, but as you said, it's a journey :)

davidfowl commented 12 months ago

@sec Then the current support is likely fine for you without native aot 😄

MichaelPeter commented 10 months ago

Another usage scenario are server side plugins, @SteveSandersonMS showed it already in his demos, but wanted to mention it again.

On self hosted environments by customers, computation is not that expensive and our server side applications can load Plugin DLLs (from us or third party) and run them into the server process. There is always the risk of memory leaks, infite loops and reduced performance but it can be tolerated.

On Cloud environments this is not tolerable anymore, since one plugin DLL could bring down all customers, also there is the risk of data of other customers can be accessed. So third party Plugin Developers need to host REST/GRPC Services for their plugins which then need to be integrated. This is much more complex and much slower than a DLL. Or there need to be a whole infrastructure for hosting third party docker containers.

With WASI, we could still load the WASI-File-Plugins in Our Application, data of other customers could not be accessed (or at least an acceptable risk if no error on our side), we could limit the amount of CPU resources to disable slow plugins and what would be importaint - if the used CPU time would be measurable - we could even charge third parties for the resources their plugins take, so they have a motivation to make them actually more efficent. Not to mention the performance gain from WASI call vs REST call.

It also would be importaint that it can be configured that only certain Rest/Socket resources can be accessed, or for example only external and no Lan-Endpoints. (be it by IP-Ranges or other means)

davidfowl commented 10 months ago

With WASI, we could still load the WASI-File-Plugins in Our Application, data of other customers could not be accessed (or at least an acceptable risk if no error on our side), we could limit the amount of CPU resources to disable slow plugins and what would be importaint - if the used CPU time would be measurable - we could even charge third parties for the resources their plugins take, so they have a motivation to make them actually more efficent. Not to mention the performance gain from WASI call vs REST call.

AFAIK WASM runtimes (the ones I've seen) don't let you limit CPU usage of running code (you can limit the memory)

.NET process host -> Some WASM runtimes-> .NET code compiled to WASM running inside of those runtimes.

Of course, these systems are not "just secure" as you need to audit the host code and WASM runtime code to make sure bugs don't lead to leaking customer data across...

I think we will see plugins using WASM more and more so that's a reasonable scenario.

mhmd-azeez commented 10 months ago

AFAIK WASM runtimes (the ones I've seen) don't let you limit CPU usage of running code

Wasmtime has the concept of "fuel": https://docs.rs/wasmtime/latest/wasmtime/struct.Config.html#method.consume_fuel

But, it's different from docker's cpu limits, it's more about preventing infinite loops

MichaelPeter commented 10 months ago

Another point - is debuggability of production environments vs. chiseled-docker

In one project we switched to chinseled dotnet containers a few days ago and now we have the problem that debugging is much more limited of production environments since there is no shell. Logging into the container and creating a memory dump? Not possible anymore without a shell, Update a file? Also not possible. Executing a curl/wget from within the container? Also not possible.

Now one could argue well then keep the shell installed for production, well then the container is bigger and its a potential security risk. A debugging sidecar? Not within the container, except written in the same language.

Wasi would allow a "sidecar"-wasi written for example in rust for .NET, which could be included in the "wasi-container" Memory dumps?, Viewing Thread Stacks? Sure. (https://docs.wasmtime.dev/examples-debugging-core-dumps.html) Debugging within the Runtime? Also possible.

But yes, most of this tooling doesn't exist yet. And you can with "drawbacks" also do that with docker.

RReverser commented 2 months ago

Is there any way to produce raw Wasm MVP files from .NET 9 after https://github.com/dotnet/runtime/pull/104683 has landed?

We've been "piggy-backing" on .NET's Wasi target to produce Wasm files targeted for custom hosts, as, unlike C/C++/Rust/etc .NET doesn't have support for raw ("hostless") Wasm output and WASI is the closest target where we could simply stub out all the imports during compilation and rely on UnmanagedCallersOnly and similar attributes to add custom imports/exports instead.

This has been possible starting from the original @SteveSandersonMS' Wasi.Sdk and up until and including .NET 8.

I now wanted to try and test how hard it would be to migrate to .NET 9 preview as it has some exciting Wasm features - in particular, AOT and [WasmImportLinkage] - but it appears that .NET 9 did a hard switch to component-based output that is 1) tied a lot more tightly with the WASI imports, making them harder to stub out and 2) means that we can no longer target engines without Wasm components support, which is still the majority of non-WASI engines.

Which brings me to the crux of my question - is there any way path to opt-out from Wasm component output or, even better, an official example that shows how to produce a raw "hostless" Wasm in plain old MVP format, going to be provided in .NET 9 or does this mean we're stuck with .NET 8 with its slow interpreter mode for IL bytecode for the foreseeable future?

UPD: I should probably add that, by decomposing components into individual Wasm files using wasm-tools component unbundle, I can see that one of those files has basically the same shape as what we used to get before, but I wouldn't want to rely on essentially implementation details and extract it from the bundled output as part of the build process. Ideally, there would be an MSBuild option instead that I could use to request statically linked Wasm MVP output.

pavelsavara commented 2 months ago

@RReverser

We switched to WASI preview 2 and component model in Net9 Preview 7. We don't plan to support preview 1 or WASIX.

We now depend on wasi:http component import directly.

We don't have dev bandwidth to support multiple build configurations, I'm sorry. WASI support is still experimental and moving fast.

RReverser commented 2 months ago

I imagine that's going to break quite a few users in similar positions to us.

Out of curiosity, what was the driver behind that decision? Wasm components are still in rather raw stages, and the engine support is pretty poor compared to regular Wasm. While it's interesting to watch its development, it seems rather too early to switch to it as the primary output format for stable .NET version, cutting off majority of engines other than I guess Wasmtime.

Perhaps there is some wasm-ld have some flag to request Wasm MVP output, where those component imports would become plain C imports instead? As mentioned above, I can see that the main raw Wasm inside the component bundle looks exactly like what we'd need - here are its imports:

Import[32]:
 - func[0] sig=3 <__imported_wasi_snapshot_preview1_environ_get> <- wasi_snapshot_preview1.environ_get
 - func[1] sig=3 <__imported_wasi_snapshot_preview1_environ_sizes_get> <- wasi_snapshot_preview1.environ_sizes_get
 - func[2] sig=40 <__imported_wasi_snapshot_preview1_clock_time_get> <- wasi_snapshot_preview1.clock_time_get
 - func[3] sig=22 <__imported_wasi_snapshot_preview1_fd_advise> <- wasi_snapshot_preview1.fd_advise
 - func[4] sig=1 <__imported_wasi_snapshot_preview1_fd_close> <- wasi_snapshot_preview1.fd_close
 - func[5] sig=3 <__imported_wasi_snapshot_preview1_fd_fdstat_get> <- wasi_snapshot_preview1.fd_fdstat_get
 - func[6] sig=3 <__imported_wasi_snapshot_preview1_fd_fdstat_set_flags> <- wasi_snapshot_preview1.fd_fdstat_set_flags
 - func[7] sig=3 <__imported_wasi_snapshot_preview1_fd_filestat_get> <- wasi_snapshot_preview1.fd_filestat_get
 - func[8] sig=24 <__imported_wasi_snapshot_preview1_fd_filestat_set_size> <- wasi_snapshot_preview1.fd_filestat_set_size
 - func[9] sig=41 <__imported_wasi_snapshot_preview1_fd_pread> <- wasi_snapshot_preview1.fd_pread
 - func[10] sig=3 <__imported_wasi_snapshot_preview1_fd_prestat_get> <- wasi_snapshot_preview1.fd_prestat_get
 - func[11] sig=2 <__imported_wasi_snapshot_preview1_fd_prestat_dir_name> <- wasi_snapshot_preview1.fd_prestat_dir_name
 - func[12] sig=6 <__imported_wasi_snapshot_preview1_fd_read> <- wasi_snapshot_preview1.fd_read
 - func[13] sig=41 <__imported_wasi_snapshot_preview1_fd_readdir> <- wasi_snapshot_preview1.fd_readdir
 - func[14] sig=42 <__imported_wasi_snapshot_preview1_fd_seek> <- wasi_snapshot_preview1.fd_seek
 - func[15] sig=6 <__imported_wasi_snapshot_preview1_fd_write> <- wasi_snapshot_preview1.fd_write
 - func[16] sig=16 <__imported_wasi_snapshot_preview1_path_filestat_get> <- wasi_snapshot_preview1.path_filestat_get
 - func[17] sig=43 <__imported_wasi_snapshot_preview1_path_open> <- wasi_snapshot_preview1.path_open
 - func[18] sig=17 <__imported_wasi_snapshot_preview1_path_readlink> <- wasi_snapshot_preview1.path_readlink
 - func[19] sig=2 <__imported_wasi_snapshot_preview1_path_unlink_file> <- wasi_snapshot_preview1.path_unlink_file
 - func[20] sig=6 <__imported_wasi_snapshot_preview1_poll_oneoff> <- wasi_snapshot_preview1.poll_oneoff
 - func[21] sig=0 <__imported_wasi_snapshot_preview1_proc_exit> <- wasi_snapshot_preview1.proc_exit
 - func[22] sig=10 <__imported_wasi_snapshot_preview1_sched_yield> <- wasi_snapshot_preview1.sched_yield
 - func[23] sig=3 <__imported_wasi_snapshot_preview1_random_get> <- wasi_snapshot_preview1.random_get
 - func[24] sig=1 <__wasi_preview1_adapter_close_badfd> <- wasi_snapshot_preview1.adapter_close_badfd
 - func[25] sig=0 <__wasm_import_poll_pollable_drop> <- wasi:io/poll@0.2.0.[resource-drop]pollable
 - func[26] sig=0 <__wasm_import_streams_input_stream_drop> <- wasi:io/streams@0.2.0.[resource-drop]input-stream
 - func[27] sig=0 <__wasm_import_streams_output_stream_drop> <- wasi:io/streams@0.2.0.[resource-drop]output-stream
 - func[28] sig=0 <__wasm_import_udp_udp_socket_drop> <- wasi:sockets/udp@0.2.0.[resource-drop]udp-socket
 - func[29] sig=0 <__wasm_import_udp_incoming_datagram_stream_drop> <- wasi:sockets/udp@0.2.0.[resource-drop]incoming-datagram-stream
 - func[30] sig=0 <__wasm_import_udp_outgoing_datagram_stream_drop> <- wasi:sockets/udp@0.2.0.[resource-drop]outgoing-datagram-stream
 - func[31] sig=0 <__wasm_import_tcp_tcp_socket_drop> <- wasi:sockets/tcp@0.2.0.[resource-drop]tcp-socket

So, while I totally understand lack of bandwidth for actual development & support, perhaps there is a way to just provide a different link-stage flag so that .NET would produce just this one single file, which already has the required shape? We can do the rest of the stubbing ourselves.

pavelsavara commented 2 months ago

@RReverser

You could try disable wasm-component-ld and return back to wasm-ld. Possibly passing -fuse-ld=lld could do the trick for you. See relevant LLVM change of the default linker.

We are also passing .wit files to the linker, which wasm-ld would not understand. You may need to hack around that. We can't support targeting wasip1 libc, that would be too much of conditional compilation (and mental overhead).

Here is WASM & WASI ABI, I'm not sure which of them would apply.

Out of curiosity, what was the driver behind that decision?

"Skate to where the puck is going, not where it has been."

RReverser commented 2 months ago

Possibly passing -fuse-ld=lld could do the trick for you. See relevant LLVM change of the default linker.

Thanks, this is helpful, going to take a look.

We are also passing .wit files to the linker, which wasm-ld would not understand. You may need to hack around that.

Yeah I suspect this is going to be a problem, too. I looked at it briefly before, and I think I could hack around it with MSBuild's Remove=... as those WIT linker args are passed as separate ItemGroup items, but that's going to be very fragile as it would need to match a linker arg by wildcard pattern... Would you be open to accepting a PR that adds an unofficial conditional MSBuild flag instead that skips the WIT files when set?

We can't support targeting wasip1 libc, that would be too much of conditional compilation (and mental overhead).

That is fine, since we are going to stub them out anyway, it doesn't matter which version is used.

On the C ABI side it still behaves like a regular C import, after all, so declaring stubs isn't very different from the original preview. It's really just the components + WIT that cause issues, which is why I'm hopeful that what we want is achievable with purely a link-time change.

"Skate to where the puck is going, not where it has been."

Heh, I agree with the sentiment, just saying it feels... risky / early for a switch, given the very limited native engine support. I liked WASI p1 for its simplicity and compatibility with any regular Wasm, and was one of probably dozens of people who made their own in-browser implementations at the time because it was just a matter of loading same Wasm file but with custom imports. Oh well, either way that's beside the point for this particular usecase.

pavelsavara commented 2 months ago

Would you be open to accepting a PR that adds an unofficial conditional MSBuild flag instead that skips the WIT files when set?

For Net10 possibly yes. But I'm afraid we will break it again ... let's give it a try.

enghch commented 2 months ago

@RReverser Did you ever get this workaround working? I'm in the same boat where I'm trying to code to an existing host that only supports modules, not components. .NET 9 preview 6 was working for me, but for some reason I couldn't get it to include native assets.

nilslice commented 2 months ago

This definitely seems like a misstep. There is practically no use of WASI Preview 2[0] in the ecosystem, and very few runtime maintainers are interested in supporting this moving target. WASI Preview 1 is still in significant demand, and it would be a shame to have to drop .NET support in our projects because of this decision.

"Skate to where the puck is going, not where it has been"

The puck definitely does not seem to be moving in the direction you're alluding to with this change. Based on the component model implementation across the runtime landscape, this severley limits the utility of .NET in the Wasm ecosystem. If that is the intention of the team here, then I suppose you are making the right call.

[0]: from https://crates.io/crates/wasi

Note the actual downloads are WASI Preview 1, with barely any activity for the Preview 2 versions.

dotnet / runtime

WASI support tracking #65895

Description

Build