Open Joe-Sewell-PreEmptive opened 4 years ago
Also, I think this might better be categorized as "Area: Linker", but I don't think I can change that myself.
same.... please fix this asap
I am unable to reproduce with VS 16.7, so hopefully it has been fixed meanwhile. Could you please confirm that it works for you with 16.7 as well?
I think the problem is/was not in the linker itself, but rather msbuild
tool, which crashed when logging the output messages from the linker about the reflection calls.
Locally the build completes and I see:
...
Reflection call 'System.Type System.Type::GetType(System.String)' inside 'System.Type Java.Interop.__TypeRegistrations::Lookup(System.String[],System.String)' was detected with argument which cannot be analyzed (TaskId:243)
Reflection call 'System.Reflection.FieldInfo System.Type::GetField(System.String,System.Reflection.BindingFlags)' inside 'Xamarin.Forms.BindableProperty Xamarin.Forms.VisualElement::Xamarin.Forms.StyleSheets.IStylable.GetProperty(System.String,System.Boolean)' was detected with argument which cannot be analyzed (TaskId:243)
Reflection call 'System.Type System.Type::GetType(System.String)' inside 'System.Void InvalidUTF16.MainPage/<Button_OnClicked>d__1::MoveNext()' was detected with type name `E` which can't be resolved. (TaskId:243)
Reflection call 'System.Type System.Type::GetType(System.String)' inside 'System.Void InvalidUTF16.MainPage/<Button_OnClicked>d__1::MoveNext()' was detected with type name `?` which can't be resolved. (TaskId:243)
Reflection call 'System.Type System.Type::GetType(System.String,System.Boolean)' inside 'System.Object Android.Runtime.AndroidEnvironment::GetHttpMessageHandler()' was detected with argument which cannot be analyzed (TaskId:243)
...
It is not working with Latest VS 16.7.2
Dotfuscator is enabled in release mode Xamarin Forms Android Project
Once the Linker is set to none the issue is gone but we can't go with this workaround as the application size is doubled.
Interesting, I can now replicate it as well. It happens in UI only and does not on the command line.
@joj could you please take a look? is it UI specific or msbuild problem?
@radekdoulik I'm seeing the same error in command-line. The error is not shown (unless you're on a diagnostic build and even then it's difficult to find), but if you go to temp you'll see a new MsBuild_ file with the same error. I have a couple of binlogs (one from VS and one from command line) but they don't look very useful. The only difference between VS and command-line when building is that VS builds each project, whereas command-line builds the solution. This results in slightly different build orders sometimes, but I'm not seeing that here.
@joj for me it passes the problematic output line (https://github.com/xamarin/xamarin-android/issues/4909#issuecomment-688074485) and the build succeeds.
my msbuild version in the terminal:
MSBuild.exe /version
Microsoft (R) Build Engine version 16.7.0+b89cb5fde for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.
16.7.0.37604
how does the IDE handles the build? could I have different msbuild version in the terminal compared to what my IDE uses?
the stacktrace looks like a crash in msbuild, I am not sure though whether it might be originating in UI.
As long as you're using the right command line prompt, you should have the same msbuild. Make sure you're rebuilding, and in release (/p:Configuration=Release /t:Rebuild
). The build will generate artifacts, so a simple build will not repro the error. I'm on 16.7.0.27905 right now, but I really don't think msbuild is the one failing here.
Also, I'm seeing the build succeed... but then I have the new error file and if I generate a binlog for it I can see the same issue VS has.
OTOH, if msbuild is crashing on a log, then it may make sense to add /v:diag
so you can have the full log and see the crash? MSBuild might have changed something or we may be logging incorrectly at some point, too.
I am using the diag verbosity build, but don't see any error and the build succeeds.
I have tried to use the VS command line prompt too (otherwise I use the Windows Terminal) and it still works ok. the same msbuild version as in terminal.
@joj does the crash originate in UI code or msbuild code?
@radical could you take a look too please?
I'm not seeing a crash at all. I'm just seeing an error in temp, which is weird but not super weird. And I'm seeing it in both msbuild running from VS and msbuild running from command line. Contents of the file in temp are:
UNHANDLED EXCEPTIONS FROM PROCESS 22356:
=====================
9/7/2020 8:52:39 AM
System.Text.EncoderFallbackException: Unable to translate Unicode character \uDDD0 at index 173 to specified code page.
at System.Text.EncoderExceptionFallbackBuffer.Fallback(Char charUnknown, Int32 index)
at System.Text.EncoderFallbackBuffer.InternalFallback(Char ch, Char*& chars)
at System.Text.UTF8Encoding.GetByteCount(Char* chars, Int32 count, EncoderNLS baseEncoder)
at System.Text.UTF8Encoding.GetByteCount(String chars)
at System.IO.BinaryWriter.Write(String value)
at Microsoft.Build.Framework.BuildEventArgs.WriteToStream(BinaryWriter writer)
at Microsoft.Build.Framework.LazyFormattedBuildEventArgs.WriteToStream(BinaryWriter writer)
at Microsoft.Build.Framework.BuildMessageEventArgs.WriteToStream(BinaryWriter writer)
at Microsoft.Build.Shared.LogMessagePacketBase.WriteToStream(ITranslator translator)
at Microsoft.Build.Shared.LogMessagePacketBase.Translate(ITranslator translator)
at Microsoft.Build.BackEnd.NodeEndpointOutOfProcBase.RunReadLoop(Stream localReadPipe, Stream localWritePipe, ConcurrentQueue`1 localPacketQueue, AutoResetEvent localPacketAvailable, AutoResetEvent localTerminatePacketPump)
===================
Which is identical to what was reported.
Interestingly, enabling the binlog makes the difference. So it looks like msbuild bug to me.
@radical could you please handle it? or should I transfer the issue to dotnet/msbuild
repo?
The error, in case of using binlog, is a bit longer. Putting it here in case it can provide additional information:
=====================
9/7/2020 4:51:11 PM
System.Text.EncoderFallbackException: Unable to translate Unicode character \uDDD0 at index 147 to specified code page.
at System.Text.EncoderExceptionFallbackBuffer.Fallback(Char charUnknown, Int32 index)
at System.Text.EncoderFallbackBuffer.InternalFallback(Char ch, Char*& chars)
at System.Text.UTF8Encoding.GetByteCount(Char* chars, Int32 count, EncoderNLS baseEncoder)
at System.Text.UTF8Encoding.GetByteCount(String chars)
at System.IO.BinaryWriter.Write(String value)
at Microsoft.Build.Logging.BuildEventArgsWriter.WriteBaseFields(BuildEventArgs e, BuildEventArgsFieldFlags flags)
at Microsoft.Build.Logging.BuildEventArgsWriter.WriteMessageFields(BuildMessageEventArgs e)
at Microsoft.Build.Logging.BuildEventArgsWriter.Write(BuildMessageEventArgs e)
at Microsoft.Build.Logging.BuildEventArgsWriter.Write(BuildEventArgs e)
at Microsoft.Build.Logging.BinaryLogger.Write(BuildEventArgs e)
at Microsoft.Build.Logging.BinaryLogger.EventSource_AnyEventRaised(Object sender, BuildEventArgs e)
at Microsoft.Build.Evaluation.ProjectCollection.ReusableLogger.AnyEventRaisedHandler(Object sender, BuildEventArgs e)
at Microsoft.Build.Framework.AnyEventHandler.Invoke(Object sender, BuildEventArgs e)
at Microsoft.Build.BackEnd.Logging.EventSourceSink.RaiseAnyEvent(Object sender, BuildEventArgs buildEvent)
===================
9/7/2020 4:51:11 PM
Microsoft.Build.Exceptions.InternalLoggerException: The build stopped unexpectedly because of an unexpected logger failure. ---> System.Text.EncoderFallbackException: Unable to translate Unicode character \uDDD0 at index 147 to specified code page.
at System.Text.EncoderExceptionFallbackBuffer.Fallback(Char charUnknown, Int32 index)
at System.Text.EncoderFallbackBuffer.InternalFallback(Char ch, Char*& chars)
at System.Text.UTF8Encoding.GetByteCount(Char* chars, Int32 count, EncoderNLS baseEncoder)
at System.Text.UTF8Encoding.GetByteCount(String chars)
at System.IO.BinaryWriter.Write(String value)
at Microsoft.Build.Logging.BuildEventArgsWriter.WriteBaseFields(BuildEventArgs e, BuildEventArgsFieldFlags flags)
at Microsoft.Build.Logging.BuildEventArgsWriter.WriteMessageFields(BuildMessageEventArgs e)
at Microsoft.Build.Logging.BuildEventArgsWriter.Write(BuildMessageEventArgs e)
at Microsoft.Build.Logging.BuildEventArgsWriter.Write(BuildEventArgs e)
at Microsoft.Build.Logging.BinaryLogger.Write(BuildEventArgs e)
at Microsoft.Build.Logging.BinaryLogger.EventSource_AnyEventRaised(Object sender, BuildEventArgs e)
at Microsoft.Build.Evaluation.ProjectCollection.ReusableLogger.AnyEventRaisedHandler(Object sender, BuildEventArgs e)
at Microsoft.Build.Framework.AnyEventHandler.Invoke(Object sender, BuildEventArgs e)
at Microsoft.Build.BackEnd.Logging.EventSourceSink.RaiseAnyEvent(Object sender, BuildEventArgs buildEvent)
--- End of inner exception stack trace ---
at Microsoft.Build.Exceptions.InternalLoggerException.Throw(Exception innerException, BuildEventArgs e, String messageResourceName, Boolean initializationException, String[] messageArgs)
at Microsoft.Build.BackEnd.Logging.EventSourceSink.RaiseAnyEvent(Object sender, BuildEventArgs buildEvent)
at Microsoft.Build.BackEnd.Logging.EventSourceSink.RaiseMessageEvent(Object sender, BuildMessageEventArgs buildEvent)
at Microsoft.Build.BackEnd.Logging.EventSourceSink.Consume(BuildEventArgs buildEvent)
at Microsoft.Build.BackEnd.Logging.EventSourceSink.Consume(BuildEventArgs buildEvent, Int32 sinkId)
at Microsoft.Build.BackEnd.Logging.EventRedirectorToSink.Microsoft.Build.Framework.IEventRedirector.ForwardEvent(BuildEventArgs buildEvent)
at Microsoft.Build.BackEnd.Logging.CentralForwardingLogger.EventSource_AnyEventRaised(Object sender, BuildEventArgs buildEvent)
at Microsoft.Build.BackEnd.Logging.EventSourceSink.RaiseAnyEvent(Object sender, BuildEventArgs buildEvent)
===================
i am really thankful for your fast interference 😊 is there any fix soon 😀?
Any good news yet @radekdoulik ?
@radekdoulik could you file an issue on dotnet/msbuild, we might get them to look into this?
System.Text.EncoderFallbackException: Unable to translate Unicode character \uDDD0 at index 147 to specified code page.
Is the obfuscator putting a \uDDD0
character in a type or method name?
No, Dotfuscator Community is only using lower case Latin characters a-z
for the actual type and member names. Dotfuscator Professional allows more types of characters, but we are careful to not introduce unpaired surrogates in type and member names.
MSBuild Team Triage: This does look like an MSBuild issue, but can the linker consider escaping or otherwise validating the strings it is logging?
Steps to Reproduce
Minimal repro:
string ReturnsTypeName(string x)
, where the returned value is something like"MyNamespace.MyType"
.string ReturnsTypeName(string x, int y)
.Type.GetType
. For the argument to the method, use a string literal that has an unpaired UTF-16 surrogate. E.g.,var x = Type.GetType(ReturnsTypeName("\uDDD0"))
.Attached is a ZIP file with this minimal repro.
InvalidUTF16.zip
Actual scenario that triggered this bug filing:
I work for the Dotfuscator team at PreEmptive, a Xamarin partner, and am filing this issue on behalf of my employer.
Dotfuscator supports protecting Xamarin Android projects. However, it seems like a recent update to the Xamarin Android linker has caused it to fail when processing certain kinds of protected code. This issue was originally reported to us by a Stack Overflow question.
The core issue appears to be that the Xamarin linker is analyzing a call in the form
Type.GetType(ReturnsTypeName("some_string"))
by assuming that"some_string"
is the type name being passed toType.GetType
, when in reality it is being passed to another method (ReturnsTypeName
) first, which then returns the type name to look up.Normally this might not be a problem, but Dotfuscator takes advantage of a technical fact of how the .NET runtime treats user strings (those stored in the assembly's
#US
blob heap). Specifically, while strings are stored as sequences of UTF-16 code units, the runtime allows any sequence of 16-bit values, not just a sequence that is a well-formed UTF-16 string. For instance, .NET (and the C# compiler) allow strings to have unpaired UTF-16 surrogates. Dotfuscator's knowledge of this fact results in scenarios in protected code similar to the minimal repro: an ill-formed UTF-16 string constant is passed to one method, which returns a well-formed UTF-16 string (only known at runtime) for use in reflection.Based on the stack trace, it looks like the linker is attempting to translate the string constant into UTF-8, and erroring because the constant is ill-formed UTF-16. Thus, this issue might be fixed by handling this error gracefully and/or by not treating
"some_string"
as a type name in the first place.This blocks our customers from using our Runtime Checks (e.g., root detection, tamper detection, etc.) features when also using the Xamarin linker. A workaround is to disable the Xamarin linker when using Dotfuscator, but we think some customers will want to have both enabled. Our team is also investigating other possible workarounds for the Runtime Checks issue specifically, but we suspect some customers code may still trigger this scenario with their own code after it has been put through Dotfuscator.
Expected Behavior
The build succeeds, with the linker ignoring the ill-formed UTF-16 string constant (possibly emitting a warning) and continuing to analyze other call sites.
Actual Behavior
The build errors during the
LinkAssemblies
task, with a message similar to:Version Information
Log File
(Included is an MSBuild log, not an ADB log.)
VS-detailed-log.txt MSBuild_pid-17500_3c3a53c096dc4f9fa9f0261a2d0d8842.failure.txt