search

xamarin / xamarin-macios

.NET for iOS, Mac Catalyst, macOS, and tvOS provide open-source bindings of the Apple SDKs for use with .NET managed languages such as C#
Other
2.47k stars 514 forks source link

Objective C binding library fails to build when attempting to wrap large (>2GB) native libraries #13136

Closed stephensheehy closed 3 years ago

stephensheehy commented 3 years ago

Objective C binding library fails to build when attempting to wrap large (>2GB) native libraries. Libraries of this size are not so unusual when compiling with debug info for multiple architectures.

The portable executable builder seems to have an internal bottleneck, as shown by the callstack in the build log below.

I would imagine this is the same PE builder issue discussed in these stack overflow posts (one of which is more than 10 years old!) https://stackoverflow.com/questions/67407062/visual-studio-embed-large-resource-file-almost-4gb https://stackoverflow.com/questions/6976693/what-is-the-maximum-size-of-a-pe-file-on-64-bit-windows

Steps to Reproduce

  1. Create an objective C binding library project
  2. Reference native libraries larger than 2GB
  3. Try to compile

Expected Behavior

Compilation success

Actual Behavior

Compilation failure

Environment

Microsoft Visual Studio Professional 2019
Version 16.11.5
VisualStudio.16.Release/16.11.5+31729.503
Microsoft .NET Framework
Version 4.8.04084

Installed Version: Professional

Visual C++ 2019   00435-60000-00000-AA153
Microsoft Visual C++ 2019

ASP.NET and Web Tools 2019   16.11.75.64347
ASP.NET and Web Tools 2019

Azure App Service Tools v3.0.0   16.11.75.64347
Azure App Service Tools v3.0.0

C# Tools   3.11.0-4.21403.6+ae1fff344d46976624e68ae17164e0607ab68b10
C# components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.

Common Azure Tools   1.10
Provides common services for use by Azure Mobile Services and Microsoft Azure Tools.

Cookiecutter   16.11.21196.2
Provides tools for finding, instantiating and customizing templates in cookiecutter format.

Extensibility Message Bus   1.2.6 (master@34d6af2)
Provides common messaging-based MEF services for loosely coupled Visual Studio extension components communication and integration.

IntelliCode Extension   1.0
IntelliCode Visual Studio Extension Detailed Info

Linux Core Dump Debugging   1.0.9.31727
Enables debugging of Linux core dumps.

Microsoft Azure Tools for Visual Studio   2.9
Support for Azure Cloud Services projects

Microsoft JVM Debugger   1.0
Provides support for connecting the Visual Studio debugger to JDWP compatible Java Virtual Machines

Microsoft MI-Based Debugger   1.0
Provides support for connecting Visual Studio to MI compatible debuggers

Microsoft Visual C++ Wizards   1.0
Microsoft Visual C++ Wizards

Microsoft Visual Studio VC Package   1.0
Microsoft Visual Studio VC Package

Mono Debugging for Visual Studio   16.10.15 (552afdf)
Support for debugging Mono processes with Visual Studio.

NuGet Package Manager   5.11.0
NuGet Package Manager in Visual Studio. For more information about NuGet, visit https://docs.nuget.org/

NVIDIA Nsight Visual Studio Edition   2020.1.0.20134
NVIDIA Nsight Visual Studio Edition provides tools for GPGPU and graphics development.  Copyright © NVIDIA 2010 - 2020.
•Direct3D® and DirectX® are registered trademarks of Microsoft Corporation in the United States and/or other countries.
•Microsoft Detours is used under the Professional license (http://research.microsoft.com/en-us/projects/detours/).
•Gardens Point Parser Generator Copyright 2005 Queensland University of Technology (QUT). All rights reserved.
•Icons from Axialis Software used under the licensing terms found here: www.axialis.com
•NLog Copyright © 2004-2006 Jaroslaw Kowalski (jaak@jkowalski.net)
•zlib and libpng used under the zlib/libpnc license (http://opensource.org/licenses/Zlib) 
•Breakpad Copyright ©2006, Google Inc. All rights reserved.
•The OpenGL Extension Wrangler Library
Copyright ©2008-2016, Nigel Stewart (nigels@users.sourceforge.net), Copyright ©2002-2008, Milan Ikits (milan.ikits@ieee.org), Copyright ©2002-2008, Marcelo E. Magallon (mmagallo@debian.org), Copyright ©2002, Lev Povalahev. 
All rights reserved. 
•LIBSSH2 Copyright ©2004-2007 Sara Golemon (sarag@libssh2.org), Copyright ©2005,2006 Mikhail Gusarov (dottedmag@dottedmag.net),Copyright ©2006-2007 The Written Word, Inc.,Copyright ©2007 Eli Fant (elifantu@mail.ru),Copyright ©2009-2014 Daniel Stenberg., Copyright ©2008, 2009 Simon Josefsson.
All rights reserved. 
•Protobuf Copyright ©2014, Google Inc. All rights reserved.
•xxHASH Library Copyright ©2012-2014, Yann Collet. All rights reserved. 
•FMT Copyright ©2012 - 2016, Victor Zverovich 
•Font Awesome Copyright 2018 Fonticons, Inc. 
•ELF Definitions Copyright (c) 2010 Joseph Koshy, All rights reserved.

Warning: This computer program is protected by copyright law and international treaties. Unauthorized reproduction or distribution of this program, or any portion of it, may result in severe civil and criminal penalties, and will be prosecuted to the maximum extent possible under the law.

NVIDIA Nsight Visual Studio Edition - CUDA support   2020.1.0.20134
NVIDIA Nsight Visual Studio Edition - CUDA support provides tools for CUDA development and debugging.

NVIDIA® Nsight™ Tegra Development Platform, Visual Studio Edition   3.6.19217.7511
NVIDIA® Nsight™ Tegra Visual Studio Edition provides tools for Android development. 
Copyright © NVIDIA Corporation 2016.
User Guide available at: http://docs.nvidia.com/gameworks/index.html#developertools/mobile/nsight_tegra/nsight_tegra_main.htm

Includes technology developed by Mentor Embedded (http://www.mentor.com/embedded-software).

vs-android is Copyright © 2011 Gavin Pugh (http://www.gavpugh.com). Vs-android contains minor modifications from the original source produced by Gavin Pugh.

Android XML parser for .NET is Copyright © 2012 Markus Jarderot. Android XML parser for .NET contains modifications from the original source produced by Markus Jarderot.

Warning: This computer program is protected by copyright law and international treaties. Unauthorized reproduction or distribution of this program, or any portion of it, may result in severe civil and criminal penalties, and will be prosecuted to the maximum extent possible under the law.

ProjectServicesPackage Extension   1.0
ProjectServicesPackage Visual Studio Extension Detailed Info

Python   16.11.21196.2
Provides IntelliSense, projects, templates, debugging, interactive windows, and other support for Python developers.

Python - Conda support   16.11.21196.2
Conda support for Python projects.

Python - Django support   16.11.21196.2
Provides templates and integration for the Django web framework.

Python - Profiling support   16.11.21196.2
Profiling support for Python projects.

Test Adapter for Boost.Test   1.0
Enables Visual Studio's testing tools with unit tests written for Boost.Test.  The use terms and Third Party Notices are available in the extension installation directory.

Test Adapter for Google Test   1.0
Enables Visual Studio's testing tools with unit tests written for Google Test.  The use terms and Third Party Notices are available in the extension installation directory.

TypeScript Tools   16.0.30526.2002
TypeScript Tools for Microsoft Visual Studio

Visual Basic Tools   3.11.0-4.21403.6+ae1fff344d46976624e68ae17164e0607ab68b10
Visual Basic components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.

Visual C++ for Cross Platform Mobile Development (Android)   16.0.31506.176
Visual C++ for Cross Platform Mobile Development (Android)

Visual C++ for Linux Development   1.0.9.31727
Visual C++ for Linux Development

Visual F# Tools   16.11.0-beta.21322.6+488cc578cafcd261d90d748d8aaa7b8b091232dc
Microsoft Visual F# Tools

Visual Studio Code Debug Adapter Host Package   1.0
Interop layer for hosting Visual Studio Code debug adapters in Visual Studio

Visual Studio Tools for CMake   1.0
Visual Studio Tools for CMake

VisualGDB   5.5
Allows developing and debugging Embedded, Linux, Android and other GCC/GDB-based applications with Visual Studio.

VisualStudio.DeviceLog   1.0
Information about my package

VisualStudio.Foo   1.0
Information about my package

VisualStudio.Mac   1.0
Mac Extension for Visual Studio

VSColorOutput   2.73
Color output for build and debug windows - https://mike-ward.net/vscoloroutput

Xamarin   16.11.000.190 (d16-11@2391ed9)
Visual Studio extension to enable development for Xamarin.iOS and Xamarin.Android.

Xamarin Designer   16.11.0.17 (remotes/origin/11e0001f0b17269345e80b58fb3adf1ba4efe2cd@11e0001f0)
Visual Studio extension to enable Xamarin Designer tools in Visual Studio.

Xamarin Templates   16.10.5 (355b57a)
Templates for building iOS, Android, and Windows apps with Xamarin and Xamarin.Forms.

Xamarin.Android SDK   11.4.0.5 (d16-11/7776c9f)
Xamarin.Android Reference Assemblies and MSBuild support.
    Mono: c633fe9
    Java.Interop: xamarin/java.interop/d16-11@48766c0
    ProGuard: Guardsquare/proguard/v7.0.1@912d149
    SQLite: xamarin/sqlite/3.35.4@85460d3
    Xamarin.Android Tools: xamarin/xamarin-android-tools/d16-11@683f375

Xamarin.iOS and Xamarin.Mac SDK   15.0.0.8 (0796d78dc)
Xamarin.iOS and Xamarin.Mac Reference Assemblies and MSBuild support.

Build Logs

/Library/Frameworks/Mono.framework/Versions/6.12.0/lib/mono/msbuild/Current/bin/Roslyn/Microsoft.CSharp.Core.targets(59,5): error : [ERROR] FATAL UNHANDLED EXCEPTION: System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
Parameter name: mappedFieldDataStreamRva 
  at System.Reflection.Throw.ArgumentOutOfRange (System.String parameterName) [0x00000] in <4bd432d0a09845d3867347e567b864a4>:0
  at System.Reflection.Metadata.Ecma335.MetadataRootBuilder.Serialize (System.Reflection.Metadata.BlobBuilder builder, System.Int32 methodBodyStreamRva, System.Int32 mappedFieldDataStreamRva) [0x0001f] in <4bd432d0a09845d3867347e567b864a4>:0
  at System.Reflection.PortableExecutable.ManagedPEBuilder.SerializeTextSection (System.Reflection.PortableExecutable.SectionLocation location) [0x000ab] in <4bd432d0a09845d3867347e567b864a4>:0
  at System.Reflection.PortableExecutable.ManagedPEBuilder.SerializeSection (System.String name, System.Reflection.PortableExecutable.SectionLocation location) [0x00029] in <4bd432d0a09845d3867347e567b864a4>:0 
  at Microsoft.Cci.ExtendedPEBuilder.SerializeSection (System.String name, System.Reflection.PortableExecutable.SectionLocation location) [0x00016] in <3ba6de6130814b0eae8f8ef21e9e5ac4>:0 
  at System.Reflection.PortableExecutable.PEBuilder.SerializeSections () [0x00073] in <4bd432d0a09845d3867347e567b864a4>:0
  at System.Reflection.PortableExecutable.PEBuilder.Serialize (System.Reflection.Metadata.BlobBuilder builder) [0x00000] in <4bd432d0a09845d3867347e567b864a4>:0 
  at Microsoft.Cci.ExtendedPEBuilder.Serialize (System.Reflection.Metadata.BlobBuilder peBlob, System.Reflection.Metadata.Blob& mvidSectionFixup) [0x00000] in <3ba6de6130814b0eae8f8ef21e9e5ac4>:0 
  at Microsoft.Cci.PeWriter.WritePeToStream (Microsoft.CodeAnalysis.Emit.EmitContext context, Microsoft.CodeAnalysis.CommonMessageProvider messageProvider, System.Func`1[TResult] getPeStream, System.Func`1[TResult] getPortablePdbStreamOpt, Microsoft.Cci.PdbWriter nativePdbWriterOpt, System.String pdbPathOpt, System.Boolean metadataOnly, System.Boolean isDeterministic, System.Boolean emitTestCoverageData, System.Nullable`1[T] privateKeyOpt, System.Threading.CancellationToken cancellationToken) [0x003ae] in <3ba6de6130814b0eae8f8ef21e9e5ac4>:0  [/Users/build/buildAgent/work/mMap_git_master/Platforms/Xamarin/mMap/mMap.iOS/mMap.iOS.csproj]
  at Microsoft.CodeAnalysis.Compilation.SerializePeToStream (Microsoft.CodeAnalysis.Emit.CommonPEModuleBuilder moduleBeingBuilt, Microsoft.CodeAnalysis.DiagnosticBag metadataDiagnostics, Microsoft.CodeAnalysis.CommonMessageProvider messageProvider, System.Func`1[TResult] getPeStream, System.Func`1[TResult] getMetadataPeStreamOpt, System.Func`1[TResult] getPortablePdbStreamOpt, Microsoft.Cci.PdbWriter nativePdbWriterOpt, System.String pdbPathOpt, System.Boolean metadataOnly, System.Boolean includePrivateMembers, System.Boolean isDeterministic, System.Boolean emitTestCoverageData, System.Nullable`1[T] privateKeyOpt, System.Threading.CancellationToken cancellationToken) [0x0002b] in <3ba6de6130814b0eae8f8ef21e9e5ac4>:0
  at Microsoft.CodeAnalysis.Compilation.SerializeToPeStream (Microsoft.CodeAnalysis.Emit.CommonPEModuleBuilder moduleBeingBuilt, Microsoft.CodeAnalysis.Compilation+EmitStreamProvider peStreamProvider, Microsoft.CodeAnalysis.Compilation+EmitStreamProvider metadataPEStreamProvider, Microsoft.CodeAnalysis.Compilation+EmitStreamProvider pdbStreamProvider, System.Func`2[T,TResult] testSymWriterFactory, Microsoft.CodeAnalysis.DiagnosticBag diagnostics, System.Boolean metadataOnly, System.Boolean includePrivateMembers, System.Boolean emitTestCoverageData, System.String pePdbFilePath, System.Nullable`1[T] privateKeyOpt, System.Threading.CancellationToken cancellationToken) [0x00150] in <3ba6de6130814b0eae8f8ef21e9e5ac4>:0 
  at Microsoft.CodeAnalysis.CommonCompiler.CompileAndEmit (Microsoft.CodeAnalysis.TouchedFileLogger touchedFilesLogger, Microsoft.CodeAnalysis.Compilation& compilation, System.Collections.Immutable.ImmutableArray`1[T] analyzers, System.Collections.Immutable.ImmutableArray`1[T] additionalTextFiles, Microsoft.CodeAnalysis.AnalyzerConfigSet analyzerConfigSet, System.Collections.Immutable.ImmutableArray`1[T] sourceFileAnalyzerConfigOptions, System.Collections.Immutable.ImmutableArray`1[T] embeddedTexts, Microsoft.CodeAnalysis.DiagnosticBag diagnostics, System.Threading.CancellationToken cancellationToken, System.Threading.CancellationTokenSource& analyzerCts, System.Boolean& reportAnalyzer, Microsoft.CodeAnalysis.Diagnostics.AnalyzerDriver& analyzerDriver) [0x0049e] in <3ba6de6130814b0eae8f8ef21e9e5ac4>:0 
  at Microsoft.CodeAnalysis.CommonCompiler.RunCore (System.IO.TextWriter consoleOutput, Microsoft.CodeAnalysis.ErrorLogger errorLogger, System.Threading.CancellationToken cancellationToken) [0x001b3] in <3ba6de6130814b0eae8f8ef21e9e5ac4>:0 
  at Microsoft.CodeAnalysis.CommonCompiler.Run (System.IO.TextWriter consoleOutput, System.Threading.CancellationToken cancellationToken) [0x00041] in <3ba6de6130814b0eae8f8ef21e9e5ac4>:0 
  at Microsoft.CodeAnalysis.CSharp.CommandLine.Csc+<>c__DisplayClass1_0.<Run>b__0 (System.IO.TextWriter tw) [0x00000] in <e6a47c0fff294ce1bf49efef8e13be1f>:0
chamons commented 3 years ago

I believe this is due to a limitation with us trying to embed the native library inside the binding library.

Can you try adding this to your binding project:

  <PropertyGroup>
    <NoBindingEmbedding>true</NoBindingEmbedding>
  </PropertyGroup>

and rebuilding?

That will stop embedding the binary, and create an additional binding resource folder you will need to distribute next to your binding library, but will significantly speed up your builds and very likely fix this issue.

stephensheehy commented 3 years ago

Hi, thanks for the reply :)

I added <NoBindingEmbedding>true</NoBindingEmbedding> and encountered the error:

error MT7068: Can't create a binding resource package unless there are native references in the binding project.

Which sounded like it was related to how the native libraries are referenced. This is how I have been doing it:

<ItemGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
    <ObjcBindingNativeLibrary Include="..\..\..\..\install\iOS_fat_10.2\lib\libmMap.a">
      <TargetPath>libmMap.a</TargetPath>
    </ObjcBindingNativeLibrary>
</ItemGroup>

I tried removing the above and adding the libraries as 'NativeReference' since the error seem to imply this may be expected:

  <ItemGroup>
    <NativeReference Include="Native References/libmMap.a">
      <Kind>Static</Kind>
    </NativeReference>
  </ItemGroup>

While this did result in the expected <project>.resources folder, it contained only a zero byte file called manifest and no copies of the static libraries.

If you have any thoughts on what prevents the external resource folder from being properly formed I'd really appreciate it.

Meanwhile I'll just see if I can find any more different ways to reference the native libraries.

chamons commented 3 years ago

Change this:

<NativeReference Include="Native References/libmMap.a">

to be

<NativeReference Include="..\..\..\..\install\iOS_fat_10.2\lib\libmMap.a">

The Include needs to be a path to your native library in question.

stephensheehy commented 3 years ago

Since I had added the native library reference through the IDE it had copied the file into Native References/libmMap.a (project relative) so the path was valid... In any case changing it gave the same result, an empty .resources folder and zero byte manifest file.

It occurred to me to see if Visual Studio on mac would behave differently and indeed it does. There the resource file and manifest are created as expected, so this last issue seems to be a bug particular to Visual Studio on Windows using a mac build agent.

As long as the command line msbuild invocation works equally well on the mac (I haven't tested this yet) our continuous integration system will be happy and I can work around the issue for local builds.

Thanks for all the help. Would love to see the Windows issue fixed though, I'm more at home there :)

chamons commented 3 years ago

That is very helpful information. Unfortunately the team that handles iOS remote builds from Windows VS is not available via github.

Could you please file an issue using the instructions here to let them know.

If you run into more trouble unrelated to that, please reopen this issue.

Thanks!

stephensheehy commented 2 years ago

Just a note for any who come later looking to do this sidecar deployment thing:

when the library is embedded the assembly linker properties can be set in code eg. [assembly: LinkWith("myLib.a", Frameworks = "Metal MetalKit CoreGraphics CoreVideo MetalPerformanceShaders", ForceLoad = true, SmartLink = true, IsCxx = true)]

But when we deploy the native libs in an external resource folder the the assembly linker properties must be set in the .csproj file so they will be correctly copied to the output manifest file, eg.

    <NativeReference Include="..\..\..\..\install\iOS_fat_10.2\lib\myLib.a">
      <Kind>Static</Kind>
      <SmartLink>true</SmartLink>
      <ForceLoad>true</ForceLoad>
      <IsCxx>true</IsCxx>
      <Frameworks>Metal MetalKit CoreGraphics CoreVideo MetalPerformanceShaders</Frameworks>
    </NativeReference>

Took me a while to figure this out. And as of writing neither building libraries with side-by-side resource folders, nor building applications which link to such libraries works in Visual Studio on Windows, it has to be Visual Studio Mac.