Update ILSpy decompiler engine to a later version

ElektroKill commented 3 years ago

Currently, dnSpyEx uses a heavily modified and patched version of ICSharpCode.Decompiler version 2.4 and NRefactory to decompile C# code. However, with the continuous updates of the C# language and the Roslyn compiler the decompilation output is slowly becoming more and more unreadable and inaccurate. The rather dated 2.4 version of ILSpy has a hard time dealing with newer code for features such as async, yield return, nullable, Span<T>, etc. Moreover, the version 2.4 decompiler engine and codebase are hard to work with, so pushing updates to it is rather hard and painful.

Originally the ILSpy 2.4 engine was based on Mono.Cecil but due to cecil's poor ability to load obfuscated assemblies, it was ported to dnlib by Ki (yck1509). Moreover, the way it prints the decompiled code was heavily modified to match dnSpy together with several other changes.

Possible approaches of taking on this problem:

Port ILSpy 7.1 decompiler engine from SRM to dnlib and apply the necessary changes needed to make it work under dnSpy. Since version 4.0 the ILSpy team switched the decompiler and type system implementation to use System.Reflection.Metadata, a very low-level metadata reader. Porting from SRM to dnlib will definitely be very challenging and time-consuming. The hrdest part being reimplementing the type system to use dnlib.
Port ILSpy 3.2 decompiler engine to dnlib and backport changes from ILSpy 7.1 creating a "hybrid". ILSpy 3.2 was the last version that still used a higher-level metadata reading library(cecil) to power the decompiler. This makes porting it to dnlib significantly easier. Moreover, the engine has not undergone any huge architectural changes since version 3.2 making it quite easy to backport most of the decompiler changes that don't rely on the type system implementation or SRM. The 3.x engine is much better and more flexible than the 2.4 engine but has one thing which makes porting the hardest, the type system.
Backup plan in case the first two flop... We can continue to rely on the dated 2.4 decompiler and attempt to implement support for new features into it even if it will require a lot of pain...

It is worth noting that approaches 1 and 2 also will require a considerable amount of changes to the decompiler and dnSpy to properly function with the dnSpy frontend, debugger, and other dnSpy features on top of the port to dnlib.

Suggestions, discussions, and novel approach ideas are welcome in the comments!

BlackOfWorld commented 3 years ago

Just wanna say good luck to your project, I might create a PR containing it, however no promises 🤗 Good luck and have fun!

ElektroKill commented 3 years ago

Great news! I Have successfully ported the ILSpy 3.2.0 decompiler engine to dnlib. The port passes all the unit tests in the project. It is still far from ideal since there are lots of changes from the future ILSpy version left to pull in. It also hasn't been adapted for usage in dnSpy yet. It is available here https://github.com/dnSpyEx/ILSpy/tree/ilspyv3.

If anyone wants to help with this issue and has free time I provide a fully built release that can be tested for bugs. The decompilation results should be compared to the original 3.2 release uploaded here: ILSpy_binaries_3.2.0.3856.zip My dnlib port: ILSpy-3.2.0-dnlib.zip Errors to ignore: BAML decompilation errors, analyzer errors. The result of the c# code and the IL disassembly should ideally be identical to that of the original 3.2 release.

Sato-Isolated commented 3 years ago

Hello I have this error when I try to open ilspy dnlib

Nuklon commented 3 years ago

Crashes for me as well.

ElektroKill commented 3 years ago

Im super sorry 😅 Here is a working build for testing: net46.zip

Sato-Isolated commented 3 years ago

the only difference I saw was in the IL code on line 105,168,212,239,264

original.txt

dnlib.txt

BlackOfWorld commented 3 years ago

Looking up in Windiff, it's clear to what's happening. It's adding class to all void instances. In theory this shouldn't affect anything. But oh well

Nuklon commented 3 years ago

If you open System.Collections from .NET 5. Then select BitArray > IsReadOnly. This works in official ILSpy. But not in this version, it gives an error:

System.InvalidOperationException: Could not find property definition in NR type system
   at ICSharpCode.Decompiler.CSharp.CSharpDecompiler.Decompile(IList`1 definitions)
   at ICSharpCode.ILSpy.CSharpLanguage.DecompileProperty(PropertyDef property, ITextOutput output, DecompilationOptions options) in D:\Documents\RiderProjects\dnSpy-NewILSpy\ILSpy\Languages\CSharpLanguage.cs:line 200
   at ICSharpCode.ILSpy.TextView.DecompilerTextView.DecompileNodes(DecompilationContext context, ITextOutput textOutput) in D:\Documents\RiderProjects\dnSpy-NewILSpy\ILSpy\TextView\DecompilerTextView.cs:line 530
   at ICSharpCode.ILSpy.TextView.DecompilerTextView.<>c__DisplayClass33_0.<DecompileAsync>b__0() in D:\Documents\RiderProjects\dnSpy-NewILSpy\ILSpy\TextView\DecompilerTextView.cs:line 515

The IL is also different, same as above with class to instance void .ctor. I will check out some others DLLs when I have time.

hymccord commented 3 years ago

I had also looked at option 1 as well since the decompilation was getting so bad on new compiler features. As you said, it's a mind-numbing amount of work upfront but if done correctly we essentially can let the ILSpy team be responsible for the decompiler keeping up with the latest C# features. The original dnSpy was slow to update the decompiler feature because it was so heavily modified.

I use dnSpy nearly every day and enjoy contributing. I'm going to help out where I can.

ElektroKill commented 3 years ago

Update: the ported version is now equivalent to the 4.0-beta1 release of official ILSpy. Still, quite a few versions to go through until we reach 7.1 (current version)...

Nuklon commented 3 years ago

Nice work @ElektroKill! Getting it based on the massive changed https://github.com/icsharpcode/ILSpy/pull/1030/ would be the next thing I guess?

ElektroKill commented 3 years ago

Nice work @ElektroKill! Getting it based on the massive changed https://github.com/icsharpcode/ILSpy/pull/1030/ would be the next thing I guess?

The type system and other architectural changes from that pull request were already implemented together with the 4.0-beta1 merge ;)

ElektroKill commented 2 years ago

Hello everyone!

Over the past few weeks, I've been working hard on ILSpy and I'm happy to announce that the dnlib version of ILSpy is now equivalent to version 4.0. If you want to read the ported code for whatever reason it's under the same branch as before (https://github.com/dnSpyEx/ILSpy/tree/ilspyv3). If you are interested in bug testing the port with the 4.0 changes and the newer type system here is a link to a build: net46.zip As before I only care about the C# decompilation quality as other parts will be kept from the original dnSpy. If you encounter any issues please report them!

Nuklon commented 2 years ago

@ElektroKill Windows Defender blocks this one. Trojan:Script/Wacatac:B!ml.

When disabled it does run, but I'm encountering exceptions with several assemblies (this does not happen with ILSpy 4.0):

System.FormatException: Index (zero based) must be greater than or equal to zero and less than the size of the argument list.
   at System.Text.StringBuilder.AppendFormatHelper(IFormatProvider provider, String format, ParamsArray args)
   at System.String.FormatHelper(IFormatProvider provider, String format, ParamsArray args)
   at System.String.Format(String format, Object[] args)
   at ICSharpCode.Decompiler.Disassembler.DisassemblerHelpers.WriteTo(ITypeDefOrRef type, ITextOutput writer, ILNameSyntax syntax, ThreeState isValueType)
   at ICSharpCode.Decompiler.Disassembler.DisassemblerHelpers.WriteFieldTo(IField field, ITextOutput writer)
   at ICSharpCode.Decompiler.Disassembler.DisassemblerHelpers.WriteOperand(ITextOutput writer, Object operand)
   at ICSharpCode.Decompiler.Disassembler.DisassemblerHelpers.WriteTo(Instruction instruction, ITextOutput writer)
   at ICSharpCode.ILSpy.CSharpILMixedLanguage.MixedMethodBodyDisassembler.WriteInstruction(ITextOutput output, Instruction instruction) in D:\Documents\RiderProjects\dnSpy-NewILSpy\ILSpy\Languages\CSharpILMixedLanguage.cs:line 111
   at ICSharpCode.Decompiler.Disassembler.MethodBodyDisassembler.WriteStructureBody(CilBody body, ILStructure s, HashSet`1 branchTargets, Int32& index, Int32 codeSize)
   at ICSharpCode.Decompiler.Disassembler.MethodBodyDisassembler.WriteStructureBody(CilBody body, ILStructure s, HashSet`1 branchTargets, Int32& index, Int32 codeSize)
   at ICSharpCode.Decompiler.Disassembler.MethodBodyDisassembler.Disassemble(MethodDef method, CilBody body)
   at ICSharpCode.ILSpy.CSharpILMixedLanguage.MixedMethodBodyDisassembler.Disassemble(MethodDef method, CilBody body) in D:\Documents\RiderProjects\dnSpy-NewILSpy\ILSpy\Languages\CSharpILMixedLanguage.cs:line 76
   at ICSharpCode.Decompiler.Disassembler.ReflectionDisassembler.DisassembleMethodBlock(MethodDef method)
   at ICSharpCode.Decompiler.Disassembler.ReflectionDisassembler.DisassembleType(TypeDef type)
   at ICSharpCode.Decompiler.Disassembler.ReflectionDisassembler.DisassembleNamespace(String nameSpace, IEnumerable`1 types)
   at ICSharpCode.ILSpy.TreeNodes.NamespaceTreeNode.Decompile(Language language, ITextOutput output, DecompilationOptions options) in D:\Documents\RiderProjects\dnSpy-NewILSpy\ILSpy\TreeNodes\NamespaceTreeNode.cs:line 62
   at ICSharpCode.ILSpy.TextView.DecompilerTextView.DecompileNodes(DecompilationContext context, ITextOutput textOutput) in D:\Documents\RiderProjects\dnSpy-NewILSpy\ILSpy\TextView\DecompilerTextView.cs:line 530
   at ICSharpCode.ILSpy.TextView.DecompilerTextView.<>c__DisplayClass33_0.<DecompileAsync>b__0() in D:\Documents\RiderProjects\dnSpy-NewILSpy\ILSpy\TextView\DecompilerTextView.cs:line 515

ElektroKill commented 2 years ago

@Nuklon I have no idea why windows defender picks this build up as malware… it must be a false positive :/

as for the error could you upload a assembly that causes this error and provide the type and method name or if it’s a assembly included in .net runtime just provide the name of the file with the type and method names. Thanks for testing :)

Nuklon commented 2 years ago

It only seems to happen at the IL and IL with C# view. C# view works. See attachment for an example. Select Raptorious.SharpMt940Lib" namespace and this exception appears. Raptorious.SharpMt940Lib.txt - rename to .dll

TheLordOfTheKings commented 2 years ago

Hi @ElektroKill dnlib was updated a month ago plus some newly merged PR's. So I guess the development is still in action. Have you merged the latest official changes in your branch?

ElektroKill commented 2 years ago

Hi @ElektroKill

dnlib was updated a month ago plus some newly merged PR's. So I guess the development is still in action.

Have you merged the latest official changes in your branch?

Yes this fork uses the latest available dnlib nuget package. By the way next time please open a new issue instead of asking in one dedicated to something else :)

ElektroKill commented 2 years ago

The ported decompiler is now up to version 6.0 :tada:

ElektroKill commented 2 years ago

Decided I'd post an update here since it's been quite a while since the last one. I officially have a fully working version of ILSpy 7.2 running with dnlib as the backend. The source code for that is available in the following GitHub repository: https://github.com/dnSpyEx/ILSpy/tree/ilspyv3.

As for getting dnSpyEx to nicely integrate with the modern ILSpy decompiler, it seems I underestimated the challenge of achieving that. dnSpyEx is currently able to run and decompile valid code and display output. That's about it for now. Features like the debugger have not been fixed yet as they require adding special annotations to the code produced by the decompiler. The same annotation problem occurs with features such as jump to reference, syntax highlighting, etc.

The current code is available here: https://github.com/dnSpyEx/dnSpy/tree/feature/new-ilspy and the CI uploads artifacts for builds on that branch if anyone is interested in picking up a build. (I want to make this very clear: please don't use these builds for daily use they are very unstable and a lot of dnSpy's features do not work).

Currently, I'm working on making some changes to the ILSpy type system to allow storing the original MemberRef and TypeRef objects instead of just storing the resolved ones. These are necessary to properly annotate the decompiled code for features such as jump to reference.

TLDR: ILSpy 7.2 works with dnlib as the metadata reading backend, but the integration with dnSpyEx requires a lot of work.

rollsch commented 2 years ago

Amazing work

mitchcapper commented 2 years ago

As everyone has said a massive feat. Small note, the CI does not seem to actually upload artifacts unless it is a tag or master (or I am missing it). Not an issue to compile on ones own, but just per the comment.

ElektroKill commented 2 years ago

As everyone has said a massive feat. Small note, the CI does not seem to actually upload artifacts unless it is a tag or master (or I am missing it). Not an issue to compile on ones own, but just per the comment.

@mitchcapper My bad, I forgot to enable that! Latest CI build which is currently running should have the artifacts uploaded!

greenozon commented 2 years ago

ILSpy already bumped to ver 8.x any plans to port it as well with dnSpyEx?

ElektroKill commented 1 year ago

Thought I would give an update on this matter.

I am currently waiting for ILSpy version 8.0 to release. This is because it contains some important fixes and improvements to the decompiler. I have also recently contributed many improvements to help bring the new ILSpy engine's VB.NET support up to the level of the current patched ILSpy 2.4 version. I am waiting for these changes to make it stable and then I will increase the amount of time I dedicate to this issue :P

Hope this is understandable.

rollsch commented 1 year ago

Thanks for doing god's work

funkvps commented 1 year ago

Good news, ILSpy 8.0 has officially released.

lofcz commented 9 months ago

@ElektroKill now with ILSpy 8.1.1 released are there any plans to move this forward? ILSpy 8.2 should target .net 6, the other option would be to wait for v8.3 / v9 targeting .net 8.

greenozon commented 9 months ago

or 8.X to target .NET 9 / 10 /11 :)

ElektroKill commented 9 months ago

The https://github.com/dnSpyEx/dnSpy/tree/feature/new-ilspy is already running on ILSpy 8.1. I'm currently still working on getting the dnSpy sequence point system to work properly when decompiling complex control flow scenarios using the updated decompiler as well as integrating it well with the dnSpy frontend and other features. I will also soon get working on merging in the old decompiler to allow the Post ILSpy update to still include the old decompiler for backward compatibility as well as VB decompilation since the new decompiler does not support VB decompilation at all. There are still a lot of things to do to get all the other dnSpy features working smoothly but decompilation is working smoothly now.

funkvps commented 9 months ago

The https://github.com/dnSpyEx/dnSpy/tree/feature/new-ilspy is already running on ILSpy 8.1. I'm currently still working on getting the dnSpy sequence point system to work properly when decompiling complex control flow scenarios using the updated decompiler as well as integrating it well with the dnSpy frontend and other features. I will also soon get working on merging in the old decompiler to allow the Post ILSpy update to still include the old decompiler for backward compatibility as well as VB decompilation since the new decompiler does not support VB decompilation at all. There are still a lot of things to do to get all the other dnSpy features working smoothly but decompilation is working smoothly now.

Fantastic work!

Maybe the most import feature for dnspy against ilspy or other similar tool is the debugger and editor feature, which would also means lots of work needed on base of ilspy.

BTW: it seems from the action page, Branch other than master will not generate binaries.

MadProbe commented 8 months ago

How much work is left to do to update the engine? Maybe I could also help but i don't know where even to start.

greenozon commented 7 months ago

.NET 8 became GA!

ILSpy already adopted and we have got first ver 9 series build:

ILSpy version 9.0.0.7539-preview1

any news when dnSpyEx will start to catch up...?

ElektroKill commented 7 months ago

The branch running the WIP ported decompiler is running the latest stable version of ILSpy currently, version 8.2. I only move to stable release versions so ILSpy 9 will only be incorporated when it's fully released. As for actually finishing the ported branch, I'll see if I can work on it in the upcoming weeks (I've really been struggling with time the last couple of months (school is taking its toll)).

Symbai commented 7 months ago

Is there any release build we can download and run?

ElektroKill commented 7 months ago

Is there any release build we can download and run?

The branch is far from very stable therefore there are no prebuilt binaries available. You have to compile them yourself!

Symbai commented 7 months ago

The branch is far from very stable

That sounds different from all your posts where you said "only this is missing" "so close" "will see if I can finish this next week" etc. So how far are we away then? Release in 2025?

hymccord commented 7 months ago

The branch is far from very stable

That sounds different from all your posts where you said "only this is missing" "so close" "will see if I can finish this next week" etc. So how far are we away then? Release in 2025?

Download and compile yourself. It’s what I do, if you don’t like the results then use the stable branch.

Elektro has generously dedicated their time to dnSpy. They’re entitled to work at their own pace.

Otherwise, help contribute to the project like I have.

Apfelmoes commented 6 months ago

FYI, CI still seems to be skipping the upload artifact task on the new ilspy branch.

slash-under commented 4 months ago

FYI, CI still seems to be skipping the upload artifact task on the new ilspy branch.

this appears to be intentional: relevant line in build.yml

CoolCoderSuper commented 3 months ago

Just wondering about a status update? Super awesome project and good work though.

mitchcapper commented 3 months ago

Just wondering about a status update? Super awesome project and good work though.

I can promise it has made it to at least the contributions welcome (monetary or otherwise:). You can watch the gh branch and sub to this ticket for updates but I doubt he wants to be pinged for them.

marcussacana commented 2 months ago

the current WIP branch has the deubgger totally broken or is already working at least partially?

ElektroKill commented 2 months ago

the current WIP branch has the deubgger totally broken or is already working at least partially?

The latest commits in the WIP branch contain a debugger which should function in most cases, however, occasionally stepping/breakpoints may not work as expected due to errors in C# -> IL offset mapping. Use with caution if you are analyzing any malicious files. If you encounter any issues, let me know here. (Note: Visual Basic view is broken in the debugger since the new ILSpy decompiler does not contain a VB.NET decompiler, in order to make VB work we will need to use the old decompiler engine as a backup.)

dnSpyEx / dnSpy

Update ILSpy decompiler engine to a later version #5