Closed steveharter closed 1 year ago
Tagging subscribers to this area: @dotnet/area-system-reflection See info in area-owners.md if you want to be subscribed.
Author: | steveharter |
---|---|
Assignees: | steveharter |
Labels: | `area-System.Reflection`, `tenet-compatibility`, `User Story`, `Priority:2`, `Cost:L`, `Bottom Up Work`, `Team:Libraries` |
Milestone: | 7.0.0 |
Would that make LambdaExpression.CompileToMethod
(or smth along those lines) more likely? :) I know System.Linq.Expressions
is archived, but apparently things are going to happen.
@steveharter I have not fully investigated MetadataBuilder at this time, but is there a 'cheap' way to get all the IL needed and just pipe it into MetadataBuilder or do we need to do this IL-statement by IL-statement?
Also, thanks @danmoseley for mentioning this API. I've been looking for this forever.
I have not fully investigated MetadataBuilder at this time, but is there a 'cheap' way to get all the IL needed and just pipe it into MetadataBuilder or do we need to do this IL-statement by IL-statement?
It is done currently statement-by-statement, you need to create InstructionEncoder and emit statements into it, then pass it to MethodBodyStreamEncoder.AddMethodBody. I suppose, we could add new AddMethodBody overload that takes array of IL bytes and emits it as a whole if it would be found useful. But the important thing about that proposed adapter is that it should not just mechanically copy the contents of in-memory assembly into MetadataBuilder, it needs to do some changes - replace every reference to System.Private.CoreLib to System.Runtime, for example. So the ability to emit the whole IL array might not simplify the task significantly.
@MSDN-WhiteKnight Thanks for the details. So, it's definitely not "cheap" as in "copy all IL bytes inside the assembly from A to B" but requires a more structured approach. Good to know because for me this means it's best to wait for the official API to catch up instead of doing it in my own code and then switch later on once the official API also supports it (aside from the question if/when/how best to contribute to this).
Moving to 8.0. There is currently active prototyping work for this.
Can we also save assembly to bytes so I can do something with them without saving to file first which in some situations like Blazor webassembly there is no file system
Can we also save assembly to bytes so I can do something with them without saving to file first which in some situations like Blazor webassembly there is no file system
Do you mean like byte[] ToBytes()
? If so, I guess that would write to a MemoryStream, not a FileStream which at that point I guess we could just add void ToStream(Stream)
?
Do you mean like byte[] ToBytes()? If so, I guess that would write to a MemoryStream, not a FileStream which at that point I guess we could just add void ToStream(Stream)?
Yea ok, I can use memory stream to write to files or I can do whatever I want with it when I don't want to write to files
While we are at it, might as well let us be able to rewrite the assembly while saving it to contain certain types
or static variables
, then output it as bytes - - - We can save one types, ten types, not just assembly but maybe only a function or a string or something else into bytes
public static class Program {
public class Fruit {}
public class Apple {}
public static (int,string) [] OhMy = [(0, "Hello world !")];
public static void Main () {
Assembly.Save<Apple>();//Returns a stream
Assembly.Capture<Apple>().Capture(OhMy).Save();//Returns a stream
}
}
Something like this
The return stream can be put to Asssembly.Load(bytes)
to get an assembly containing only pieces of the original assembly such as Fruit or Apple + Fruit or OhMy + Appple + Fruit
Not mentioned anywhere: PDB generation. That was an important part of refemit; is it going to be in 8.0?
I watched the code review video on youtube, and I hear people wondering about safety & the utility of the feature.
Let's start by saying that I personally don't like features being censored out of safety concerns, just like making BinaryFormatter obsolete on ASP.NET, I don't believe it's a great idea to prevent people doing things, because people are adults they can make their choices.
I heard people saying this feature is being used for diagnostic, I guess people are different, I'm using this to exchange Assembly between applications and enable features / capabilities on applications letting them do things they couldn't do before. A knife is a kitchen tool but it can be used for killing people too, a feature like this allows me to negotiate security protocol for my distributed application sharing components on the fly dispatching patches autonomously, while it can be used by hackers who hacked my origin server or who is trying to pretend to be my application to inject malicious behaviors, but we are not going to talk about the security concern, because that's my business none of yours, I am hoping I am expressing opinions I have here today, of course I shouldn't infringe on your right to freely choose whether to implement this feature. If I can get this mechanism that I can save .NET assembly to bytes on the fly it would makes me happy, I use .NET it's a close partner for me as a software platform, otherwise I'll implement myself by embedding a virtual machine in my program no big deal it's all lambda calculus in the end.
About Binary serialization I want to talk a bit more about it, I guess it had concerns that so many people using ASP.NET and the web is such a big attack surface it made sense to block certain misconducts, I understand but think about the idea of this country and the advantage we have over china for a sec, now, in this country freedom is a very important element that we had inherited from the people who're living before us on this land, here in this country we have a free market whether people can decide what's good for them. There're numerous methods for communicating with the server aside from using BinaryFormatter
, for example Protobuf
, or Json
, some use binary serialization some doesn't, you are reasonable for considering BinaryFormatter for something that's like a hack that should belong in prototypes program or for academic research only and never be allowed to run in production, meawhile we have all these serialization methods, some are binary Serialization some are not, so we still end up using binary serialization sometimes, Meanwhile in .NET we can MemoryMarshal a struct to a bunch of bytes too, and share pointers between process or even machines and let memory itself be accessible from the network using memory mapped files, all that's possible and sometimes introduce more serious security risks than the BinaryFormatter itself and banning BinaryFormatter on ASP.NET will never be enough to make a system secure, on the other hand there are these way to do things, and they can become a valid option instead of a security risk, for example the microservice that's running on 10k container that lives for a couple minutes when dynamically scaled probably doesn't care if it's hacked, so what's important was to let people making their own choices, instead of banning all the bad fruits on the market and only allow people to pump good gasoline into their cars.
I would be interested in hearing the philosophies from you guys and it's always been a pleasure using dotnet
Another usage of the binary serialization of assembly, is today we can already turn string into compiled program in .NET, so tomorrow we can dynamically generate part of the program in the cloud by being able to precieve the needs of the application and prompting AI to generate the code for the requirement via their api (for example, there should now be a button in the application that calls my method when clicked) then compiling the code on a powerful enough server or locally then send the compiled code as serialized assembly if we need to
@Xyncgas BinaryFormatter
is too easy to misuse for such common operation (transfer some data) and it doesn't work by magic, it adds maintenance burden (821 references to StreamingContext, 844 references to SerializationInfo in dotnet/runtime, and numerous mentions like this one)
https://github.com/dotnet/runtime/blob/91a572de92d347a30ea41265a9a3cf86fbcafeb9/src/libraries/System.Net.Primitives/src/System/Net/CookieCollection.cs#L27-L29
10k container ... doesn't care if it's hacked
If someone hacked 1 container it means you he can hack all 10k replicas of it. It means that now someone have 10k botnet ready to bring down any server in your private network. You failed to understand that security is important and it's okay (not really), .NET team covered you.
turn string into compiled program
use appropriate tool (Roslyn API)
@OwnageIsMagic
use appropriate tool (Roslyn API)
Not even close. Roslyn is the appropriate tool for turning C# code into a compiled assembly. It is not the appropriate tool for the general use-case of generating assemblies, unless your intention is to completely throw the most fundamental promise of a Common Language Runtime out the window.
@masonwheeler I'm not trying to say AssemblyBuilder.Save
is not useful, I said for case described by @Xyncgas Roslyn is more appropriate tool (LLMs generate code in high level languages, so it's about string to assembly, not general use-case of generating assemblies).
Can you elaborate more on the most fundamental promise
? I know for Java it was Write once, run anywhere
, what about .NET?
CLR is actually Microsoft implementation of CLI (ECMA-335). V.5.5 Reflection library
of ECMA-335 states that Reflection
library is optional, library specification doesn't mention Reflection.Emit
namespace at all.
So Reflection.Emit
is just another regular library, you are free to use Mono.Cecil
or any other.
@OwnageIsMagic The most fundamental promise of a common language runtime is right there in the name: it's one runtime that can be shared in common by multiple languages. What I was saying is that trying to narrow its scope to only C# — as far too many people have actually done, even if you aren't one of them — is a betrayal of that promise.
Update: We have added partial support for AssemblyBuilder.Save in .NET 8:
We have completed our goal to add MVP (Minimum Viable Product) support in .NET 8 that allows us to save *.tlb
files produced by TlbImp tool. Now we can port the tool into .NET Core using the new AssemblyBuilder.Save implementation.
The implementation support following:
The implementation doesn't support other members (a method with body (IL not supported), Properties, Constructors, Events). Because we have only partial implementation we did not exposed the approved new APIs publicly. Those APIs are internal, if you want to give it a try you could use reflection, example:
using System.Reflection.Emit;
using System.Reflection;
namespace TestApp
{
internal class Program
{
static void Main(string[] args)
{
AssemblyName aName = new AssemblyName("MyAssembly");
AssemblyBuilder ab= PopulateAssemblyBuilderAndSaveMethod(aName, null, out MethodInfo saveMethod);
ModuleBuilder moduleBuilder = ab.DefineDynamicModule("ModuleName"); // Only one module supported
moduleBuilder.DefineType("TestInterface", TypeAttributes.Interface | TypeAttributes.Abstract);
// Add other classes, interfaces, and structs. Define method, field, custom attributes for them
// Note that non abstract method not supported yet as writing IL is not supporeted yet
// defining Constructors, Properties and Events are also not supported yet
// Then save the assembly into a file (or Stream)
saveMethod.Invoke(ab, new object[] { "MyAssembly.dll" });
}
private static AssemblyBuilder PopulateAssemblyBuilderAndSaveMethod(AssemblyName assemblyName,
List<CustomAttributeBuilder>? assemblyAttributes, out MethodInfo saveMethod)
{
Type abType= Type.GetType("System.Reflection.Emit.AssemblyBuilderImpl, System.Reflection.Emit", throwOnError: true)!;
saveMethod = abType.GetMethod("Save", BindingFlags.NonPublic | BindingFlags.Instance,
new Type[] { typeof(string) /* or use typeof(Stream) */ });
MethodInfo defineDynamicAssembly = abType.GetMethod("DefinePersistedAssembly", BindingFlags.NonPublic | BindingFlags.Static,
new Type[] { typeof(AssemblyName), typeof(Assembly), typeof(List<CustomAttributeBuilder>) });
return (AssemblyBuilder)defineDynamicAssembly.Invoke(null, new object[] { assemblyName, typeof(object).Assembly, assemblyAttributes });
}
}
}
IL not supported
So... a whole lot of work to produce a resulting product that's entirely worthless. If you can't create method bodies, is there any point at all to any of it?
IL not supported
So... a whole lot of work to produce a resulting product that's entirely worthless. If you can't create method bodies, is there any point at all to any of it?
I think this response is unnecessarily mean and ignorant.
Yes, there is a lot of value in proving out the MVP and ensuring that the design can satisfy the needs of a real scenario. As @buyaa-n mentioned the TlbImp
scenario can be completed for the most common scenarios. There are limitations, but for right now the private implementation has helped us to understand how to architect a high quality solution.
@masonwheeler
As Aaron said, your comment is unnecessarily hostile and violates our Code of Conduct. I've called you out for your commenting style before. This is the last warning. If we see further behavior like this, we'll block your account.
@buyaa-n Thank you for the update!
Since IL is not supported in the first drop and you mentioned testing out the APIs via Reflection: My scenario is to use runtime code generation to build subclass proxies and then persisting the generated assemblies so we don't pay for it during each application start. Right now, the Save-code is enabled only for the .NET Framework targeted assembly of our library.
If I read the current state of the implementation correctly, there is no helpful information to gain if I try the current state via reflection. Since the complete feature is still marked for the .NET 8 release, I'll happily wait for the drop with IL support and test once my scenario becomes available. If there is anything I should test (and provide feedback on) before that point, please let me know.
For completeness sake: right now I'm expecting that I can just call the same APIs again as I do in .NET Framework once the feature is production ready (AssemblyBuilder.Save()
and ILGenerator.MarkSequencePoint
and the ISymbolDocumentWriter
interface's API)
Thanks, Michael
OK, sorry about that. Let's try this again, a bit less confrontationally.
I was under the impression that the main reason AssemblyBuilder.Save
was not originally included in .NET Core was a technical one: generating cross-platform PE files and PDB files is not as simple as generating Windows-only binaries. Therefore, I was kind of expecting that the Core implementation would keep about 90% of the existing, tried-and-tested Save
code from .NET Framework and focus on that specific challenge.
The not-supported features described above all fall under the category of "generate IL and metadata," which doesn't feel like something that needs to be reimplemented in a cross-platform manner. (There is no platform-dependent code in System.Reflection.Metadata.IL.MethodBodyBlock
, for example.) But apparently this is not the case. So what am I missing?
For completeness sake: right now I'm expecting that I can just call the same APIs again as I do in .NET Framework once the feature is production ready
This expectation can't be met. It is already decided that API surface is different, the new API takes additional parameter for core assembly, while .NET Framework API always implied the current implementation's core assembly.
Therefore, I was kind of expecting that the Core implementation would keep about 90% of the existing, tried-and-tested Save code from .NET Framework and focus on that specific challenge.
Same goes for this expectation, too. It was already discussed at length, maintainers stated that existing code is native and tied to Windows too much, so it's too hard to bring it cross platform. So the current work entirely reimplements everything as managed implementation on top of System.Reflection.Metadata.
@masonwheeler
Therefore, I was kind of expecting that the Core implementation would keep about 90% of the existing, tried-and-tested Save code from .NET Framework and focus on that specific challenge.
Let me paraphrase: AssemblyBuilder.Save()
in .NET 8 is already able to persist a complete .NET assembly (the DLL file, personally, I don't care about doing it with EXE files) to disk after the types (including method bodies) have been created at runtime in memory?
The not-supported features described above all fall under the category of "generate IL and metadata," which doesn't feel like something that needs to be reimplemented in a cross-platform manner. (There is no platform-dependent code in System.Reflection.Metadata.IL.MethodBodyBlock, for example.)
This sentence actually confuses me a bit right now:
The not-supported features described above
Do you mean the features I mentioned or the features discussed in the announcement post ? In the announcement post, method bodies are listed as not (yet) supported.
Thanks, Michael
generating cross-platform PE files and PDB files is not as simple as generating Windows-only binaries
No, this is easily handled by Roslyn and the runtime can handle this as well. Windows PDBs (non-portable) are hard, but that is why Portable PDBs exist.
But apparently this is not the case. So what am I missing?
The original was deep within the runtime with a lot of unmanaged code. This is a complete rewrite in pure managed so that it can be independent of the runtime implementation and shared with other runtimes (i.e., NativeAOT mono etc).
Also note that in .NET 8 the goal was for an MVP see the original announcement - https://github.com/dotnet/runtime/issues/62956#issuecomment-1644570818.
We have completed our goal to add MVP (Minimum Viable Product) support in .NET 8
This is not ready for everyone to start using as if it is full support in .NET 8. It was meant as a friendly update on status with a brief outline of current support. The goal is to have the ability to generate arbitrary .NET Assemblies in a similar way in .NET Core, but that isn't the current level as it is a full rewrite and reconciliation of how an assembly is generated.
There will be slight API changes for all the reasons mentioned in https://github.com/dotnet/runtime/issues/62956#issuecomment-1645609191.
@MSDN-WhiteKnight
This expectation can't be met. It is already decided that API surface is different, the new API takes additional parameter for core assembly, while .NET Framework API always implied the current implementation's core assembly.
Okay, that's not a problem. Probably should have been a bit more highlevel with "expect". What I intended to express was, that there's a way similiar to AssemblyBuilder.Save(...)
that allows persisting the runtime generated assembly including methods with their bodies. And for added value, it would also be great to support the generation Debug symbols.
@MichaelKetting
Do you mean the features I mentioned or the features discussed in the announcement post?
The latter.
@AaronRobinsonMSFT
The original was deep within the runtime with a lot of unmanaged code.
OK, that would do it. Thanks for clarifying!
@AaronRobinsonMSFT and now I'm coming full circle and I'm terribly sorry for repeating the same question. I'll try super simple / unabiguous ones:
Thanks, Michael
If the answer to 1) is "no", is it planned for RC of .NET 8 and is there something I should try as a preview via reflection to give feedback?
I'll take this one step further. I have relevant experience in implementing stuff like this. Is there any way I can contribute to get this into .NET 8? The community's been waiting 7 years for this feature already. If I can keep from adding another whole year onto that total, I'd like to.
- Can we use AssemblyBuilder right now to persist runtime generated types, including their method bodies? (from reading https://github.com/dotnet/runtime/issues/62956#issuecomment-1644570818, the answer would be "no")
No.
- If the answer to 1) is "no", is it planned for RC of .NET 8 and is there something I should try as a preview via reflection to give feedback?
Sadly no. It took a substantial amount of time to detangle the current implementation and we are not up against the code complete time frame.
- If the answer to 2) is "no", do you have a milestone for it that are are allowed to communicate?
This is likely a .NET 9 deliverable at the moment. There just hasn't been enough time for the team to make progress on it and handle the Reflection related work in flight.
I'll take this one step further. I have relevant experience in implementing stuff like this. Is there any way I can contribute to get this into .NET 8?
@masonwheeler I don't think we have the time this release, but I'll let @buyaa-n @steveharter or @jkotas comment specifically on whether help could change that calculus.
@AaronRobinsonMSFT Thank you for the answers! To me, the details on the updated milestones are very helpful :)
Sorry seems my update was not clear about remaining work will not happen in .NET 8. I had mentioned it in the description: Remaining work that out of .NET 8.0:
Remaining work that out of .NET 8.0:
- [ ] Implement remaining parts needed for
AssemblyBuilder.Save(string/stream)
(mostly will rely on community contributions)
- [ ] ILGenerator implementation
- [ ] Add support for remaining members
- [ ] Add ConstructorBuilderImpl and save it to file/stream (this needs minimal IL support)
- [ ] Add PropertyBuilderImpl and save it to file/stream (this also needs minimal IL support)
- [ ] Add EventBuildterImpl and save it to file/stream
- [ ] Add implementation for remaining APIs for a MethodBuilder/TypeBuilder/FieldBuilder (like
SignatureCallingConvention
,*RequiredCustomModifiers
,*OptionalCustomModifiers
)- [ ] Have an API for creating PDBs and implement it
- [ ] Entry point support
The above will be planned for .NET 9
I'll take this one step further. I have relevant experience in implementing stuff like this. Is there any way I can contribute to get this into .NET 8? The community's been waiting 7 years for this feature already. If I can keep from adding another whole year onto that total, I'd like to.
Thanks @masonwheeler, bare estimate of remaining work is listed above. You are welcome to contribute to any of those, though most of these needs IL support. Now we don't have a resource to put on this effort within .NET 8 timeframe. Therefore, I don't believe all of these could finish within .NET 8 timeframe with community contributions. Though your contribution could help the feature complete successfully in .NET 9
@buyaa-n Where in the repo would I find the relevant work-in-progress?
Also, are you guys aware of ILPack, which implements most of this already? Have you looked at it and found any good reasons why its code couldn't be pulled into the Core codebase? (I've tried using it and found it to be an imperfect replacement for AssemblyBuilder.Save
. It doesn't do everything, but it gets you about 85% of the way there.)
@buyaa-n Where in the repo would I find the relevant work-in-progress?
This issue is for tracking all relevant work and you can see relevant PRs linked to this issue. The last PR was https://github.com/dotnet/runtime/pull/88503
We are aware of ILPack and it is saves Assembly
not AssemblyBuilder
, Module
not ModuleBuilder
etc there is some similarity but not directly usable.
@buyaa-n thanks for the contribution.
I think most people here are just interested in dumping runtime generated code. This does not mean the code has to to run when reloaded. We have the tools to inspect the IL. While falling back to .NET desktop is generally good enough, there are cases, it makes it impossible to debug codegen on .NET core.
So, is it possible to dump some (possible malformed) IL to a stream for analysis?
So, is it possible to dump some (possible malformed) IL to a stream for analysis?
From within the process that generated it, or outside? If the latter, ClrMD could be used to inspect dynamic modules from external process, it represents them as ClrModule with IsDynamic property set to true. I have my project CIL View that attempts to implement viewing dynamic assemblies from external process, but it does not handle everything and it is not tested with newer CLR versions (only tested with .NET Core 3.1 as of now).
So, is it possible to dump some (possible malformed) IL to a stream for analysis?
We could consider adding a bare minimum (or empty) IL support within .NET 8 RC1 period (by August 14th). If a community willing to contribute, else currently we don't have a resource to work on it.
Closing the issue, we will open a new issue(s) for remaining work for .NET 9 planning.
turn string into compiled program
use appropriate tool (Roslyn API)
How does the Roslyn API help with IronPython? (See https://github.com/IronLanguages/ironpython2/issues/351)
As far as I know, the Rosyln API is rather centric to C#, and does not support the different interpretations of the DLR needed by Python.
Closing the issue, we will open a new issue(s) for remaining work for .NET 9 planning.
Could you link those new issues here, so we can follow them? Thanks!
Could you link those new issues here, so we can follow them? Thanks!
I will, when the issue(s) created.
My dear friends, I am extremely happy that this issue gets some traction, but I feel like it is getting a little bit mishandled.
Perhaps new issues should've been created before we close the root one, the one that got a fair number of upvotes and people actually watching it?
And also, perhaps the issue that asks for a working AssemblyBuilder.Save()
should only be closed after we get a working AssemblyBuilder.Save()
in the public API? Even if we ignore the fact that the current solution doesn't yet cover all the use cases, it is only callable via reflection. While perfectly okay as a temporary measure, it shouldn't be considered as the target here.
Perhaps new issues should've been created before we close the root one, the one that got a fair number of upvotes and people actually watching it?
And also, perhaps the issue that asks for a working AssemblyBuilder.Save() should only be closed after we get a working AssemblyBuilder.Save() in the public API? Even if we ignore the fact that the current solution doesn't yet cover all the use cases, it is only callable via reflection. While perfectly okay as a temporary measure, it shouldn't be considered as the target here.
FYI the root issue is still open: https://github.com/dotnet/runtime/issues/15704, this was only for tracking the work for 8.0
Ah, I'm sorry, my bad then. Thanks for your work!
Could you link those new issues here, so we can follow them? Thanks!
UPDATE: we are making progress here, see the Tracking issue for detail
UPDATE: we are making a good progress:
CreateGlobalFuntions
, DefineGlobalMethod
etc. The work is in progressPlanning to finish the main functionality (excluding entry point and PDB) and make related APIs public in preview 1. Here is the tracking issue for details: https://github.com/dotnet/runtime/issues/92975.
Now it would be great to test the implementation with a real-life scenario. I would really appreciate if you could:
Thank you in advance!
The most common feature request in reflection, with 100+ upvotes, is to add support for equivalent
AssemblyBuilder.Save()
functionality. Since this exists in .NET Desktop, some consider this an adoption blocker.Prototyping done in .NET 7.0 to investigate feasibility of adding the
Save()
via an adapter fromAssemblyBuilder
to the newerMetadataBuilder
which supports writing IL to a Stream\file. Related issues:System.Reflection.Emit.AssemblyBuilder.Save Unable to set EntryPoint on generated assemblies with System.Reflection.Emit
We plan to abstract out Reflection.Emit APIs in order to have two implementations, one that having old code to support all downlevel runtime, and a new implementation that replaces the current Reflection.Emit implementation with
AssemblyBuilder.Save()
support, steps:ParameterBuilder
andAssemblyBuilder.Save(string/stream)
reviewed and approvedAssemblyBuilder.Save(string/stream)
CustomAttributes
for all members like assembly/module (PR)Remaining work that out of .NET 8.0:
AssemblyBuilder.Save(string/stream)
(mostly will rely on community contributions)SignatureCallingConvention
,*RequiredCustomModifiers
,*OptionalCustomModifiers
)