search

ikvmnet / ikvm

A Java Virtual Machine and Bytecode-to-IL Converter for .NET
Other
1.15k stars 109 forks source link

Support for JARs with size greater than 2 Gigs #522

Open rizistt opened 2 months ago

rizistt commented 2 months ago

Currently the method WriteResources under IKVM.Tools.Importer.CompilerClassLoader uses MemoryStream which is limited to 2 gigs in size whereas I'm working with JAR files (that contain CUDA dlls) having size greater than 2 gigs which causes IO_StreamTooLong exception.

Are there any workarounds?

rizistt commented 2 months ago

Turns out it's not really a limitation of IKVM at all. I changed the MemoryStream to FileStream in the WriteResources method and got this:

System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'mappedFieldDataStreamRva')
    at System.Reflection.Throw.ArgumentOutOfRange(String parameterName)
    at System.Reflection.Metadata.Ecma335.MetadataRootBuilder.Serialize(BlobBuilder builder, Int32 methodBodyStreamRva, Int32 mappedFieldDataStreamRva)
wasabii commented 2 months ago

Taking a deeper look, there's probably a hard limit of 3 or 2 GB for .NET assembly sizes as well, at least on Windows, as they are PE files, and ultimately loaded with LoadLibrary.

I don't have a workaround for you other than to not put this particular data in a JAR, etc.

rizistt commented 2 months ago

Thank you for your response, @wasabii. As a workaround what I did was to modify the Jar itself and split it into 2 Jars and changed the metadata and got it working, I haven't fully tested it but it seems to be working. Unfortunately, I am not the author of those packages (yes it's more than 2 source packages and around 4 gigs in size) so I have very little control over what I can do with them.

wasabii commented 2 months ago

As a feature, it might be useful to have the ability for ikvmc to generate assemblies with external resources as files or something.

rizistt commented 2 months ago

It's actually a great idea.