dotnet / android

.NET for Android provides open-source bindings of the Android SDK for use with .NET managed languages such as C#
MIT License
1.93k stars 528 forks source link

ByteBuffer is not exposing the Array() method #5347

Open ajpinedam opened 3 years ago

ajpinedam commented 3 years ago

Steps to Reproduce

  1. Create a ByteArray object allocating any number of items var byteBuffer = ByteBuffer.Allocate(2).PutShort(timeout);
  2. byteBuffer should have an Array() method (or Property) to get the array from the ByteBuffer object but it's not.

byteBuffer.Array()

Java documentation: https://docs.oracle.com/javase/7/docs/api/java/nio/ByteBuffer.html

C# / Xamarin.Android Doc: https://docs.microsoft.com/en-us/dotnet/api/java.nio.bytebuffer?view=xamarin-android-sdk-9

Expected Behavior

Array method is present in the ByteBuffer object

Actual Behavior

Array method is not present in the ByteBuffer object

Version Information

Visual Studio Community 2019 for Mac Version 8.7.9 (build 9) Installation UUID: 68a8f36e-286f-4402-ad89-4d3d8e5c8fe8 GTK+ 2.24.23 (Raleigh theme) Xamarin.Mac 6.18.0.23 (d16-6 / 088c73638)

Package version: 612000093
Mono Framework MDK Runtime: Mono 6.12.0.93 (2020-02/620cf538206) (64-bit) Package version: 612000093 Roslyn (Language Service) 3.7.0-6.20427.1+18ede13943b0bfae1b44ef078b2f3923159bcd32 NuGet Version: 5.7.0.6702 .NET Core SDK SDK: /usr/local/share/dotnet/sdk/3.1.403/Sdks SDK Versions: 3.1.403 3.1.402 3.1.300 3.1.202 3.1.200 3.1.101 3.1.100 3.0.101 3.0.100 2.2.402 2.1.701 2.1.700 2.1.505 2.1.504 2.1.503 2.1.302 MSBuild SDKs: /Library/Frameworks/Mono.framework/Versions/6.12.0/lib/mono/msbuild/Current/bin/Sdks .NET Core Runtime Runtime: /usr/local/share/dotnet/dotnet Runtime Versions: 3.1.9 3.1.8 3.1.4 3.1.2 3.1.1 3.1.0 3.0.1 3.0.0 2.2.7 2.1.23 2.1.22 2.1.18 2.1.16 2.1.15 2.1.14 2.1.13 2.1.12 2.1.11 2.1.9 2.1.8 2.1.7 2.1.2 Xamarin.Profiler Version: 1.6.12.29 Location: /Applications/Xamarin Profiler.app/Contents/MacOS/Xamarin Profiler Updater Version: 11 Apple Developer Tools Xcode 12.1 (17222) Build 12B45b Xamarin.Mac Version: 6.20.2.2 (Visual Studio Community) Hash: 817b6f72a Branch: d16-7 Build date: 2020-07-18 18:44:59-0400 Xamarin.Android Version: 11.0.2.0 (Visual Studio Community) Commit: xamarin-android/d16-7/025fde9 Android SDK: /Users/andrespineda/Library/Developer/Xamarin/android-sdk-macosx Supported Android versions: 4.4 (API level 19) 5.1 (API level 22) 7.1 (API level 25) 8.0 (API level 26) 8.1 (API level 27) SDK Tools Version: 26.1.1 SDK Platform Tools Version: 30.0.0 SDK Build Tools Version: 29.0.2 Build Information: Mono: 83105ba Java.Interop: xamarin/java.interop/d16-7@1f3388a ProGuard: Guardsquare/proguard/proguard6.2.2@ebe9000 SQLite: xamarin/sqlite/3.32.1@1a3276b Xamarin.Android Tools: xamarin/xamarin-android-tools/d16-7@017078f Microsoft OpenJDK for Mobile Java SDK: /Users/andrespineda/Library/Developer/Xamarin/jdk/microsoft_dist_openjdk_1.8.0.25 1.8.0-25 Android Designer EPL code available here: https://github.com/xamarin/AndroidDesigner.EPL Android SDK Manager Version: 16.7.0.13 Hash: 8380518 Branch: remotes/origin/d16-7~2 Build date: 2020-09-16 05:12:24 UTC Android Device Manager Version: 16.7.0.24 Hash: bb090a3 Branch: remotes/origin/d16-7 Build date: 2020-09-16 05:12:46 UTC Xamarin Designer Version: 16.7.0.495 Hash: 03d50a221 Branch: remotes/origin/d16-7-vsmac Build date: 2020-08-28 13:12:52 UTC Xamarin.iOS Version: 14.2.0.12 (Visual Studio Community) Hash: e0ce49680 Branch: xcode12.1 Build date: 2020-10-20 09:52:06-0400 Build Information Release ID: 807090009 Git revision: d29d54665a1fc79986687a540a7a4676f97ad3da Build date: 2020-10-21 15:05:18-04 Build branch: release-8.7 Xamarin extensions: d29d54665a1fc79986687a540a7a4676f97ad3da Operating System Mac OS X 10.15.7 Darwin 19.6.0 Darwin Kernel Version 19.6.0 Mon Aug 31 22:12:52 PDT 2020 root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64 ### Log File
jpobst commented 3 years ago

Looks like we explicitly removed it 10+ years ago: https://github.com/xamarin/xamarin-android/blob/4cd3913def29cb445e07a1e0c3a0e03877ad8ee8/src/Mono.Android/metadata#L473.

I'll need to investigate a bit to see if I can figure out why.

You can probably work around it using one of:

Get(byte[] dst);
Get(byte[] dst, int offset, int length);
jonpryor commented 3 years ago

I'll need to investigate a bit to see if I can figure out why.

I suspect part of the problem is our old favorite covariant return types: ByteByffer.array() overrides [Buffer.array()](https://developer.android.com/reference/java/nio/Buffer#array()), which is abstract:

abstract class Buffer {
    public abstract Object array();
}
class ByteBuffer extends Buffer {
    public byte[] array() {…}
}

What should a binding for this look like? I can think of only three solutions:

  1. Java.Lang.Object all the things!
  2. Hide Buffer.array(), bind ByteBuffer.array() and other overrides "normally"
  3. [Requires prototyping, testing, sanity checking]: pursue one of our long-standing binding ideas of "unifying" java.lang.Object and System.Object -- emitting object everywhere java.lang.Object exists -- then using C#9 covariant return types in the override.

Java.Lang.Object all the things!

This "works", but really results in a fugly binding:

abstract class Buffer {
    public abstract Java.Lang.Object Array();
}
class ByteBuffer : Buffer {
   public override Java.Lang.Object Array();
}

Hide Buffer.array(), bind ByteBuffer.array() and other overrides "normally"

This makes for a "nicer" API, but means Buffer in-and-of-itself is a potentially "less useful" abstraction:

abstract class Buffer {
    public abstract Java.Lang.Object Array();
}
class ByteBuffer : Buffer {
   public byte[] Array();
}

C#9!

C#9 adds covariant return types!

This is a more "pie-in-the-sky" idea, but What If™

abstract class Buffer {
    public abstract object Array();
}
class ByteBuffer : Buffer {
   public override byte[] Array();
}

This assumes that C#9 covariant return types can have ByteBuffer.Array() return a byte[] when the base method returns object. I don't know if this is actually supportable.

Then there's the "ABI break" concern: we can't bind all instances of java.lang.Object with System.Object, it'll break everything, so if we wanted to consider this we'd have to only do this for members added since API-30.

(Then there's the other question of what happens with older compilers trying to use this…)

ajpinedam commented 3 years ago

What a great explanation @jonpryor ! Thanks.

Just wondering if we can avoid breaking things by creating an Extension method (or any Utility method) the same way we have done in other cases (I don't have any example now but sure that I have seen it).

This way we can leave things as they are in terms of the Java APIs but expose our own implementation that will return the desired value.

btw: I tried @jpobst suggestion (Thanks for this)

Get(byte[] dst);

but I am having an exception of type Java.Nio.BufferUnderflowException

This is what I am doing:

var byteBuffer = ByteBuffer.Allocate(2).PutShort(1);
byte[] result = new byte[10];
byteBuffer.Get(result);

I am getting the same exception even while using just

var tbyte = byteBuffer.Get();
jpobst commented 3 years ago

I think you are trying to read from the end of the buffer. You will need to "rewind" so that it is pointed at the beginning:

var byteBuffer = ByteBuffer.Allocate (10).PutShort (33);
var result = new byte [byteBuffer.Capacity ()];

byteBuffer.Rewind ();

byteBuffer.Get (result);
ajpinedam commented 3 years ago

Thank you @jpobst .

andreinitescu commented 2 years ago

I think you are trying to read from the end of the buffer. You will need to "rewind" so that it is pointed at the beginning:

var byteBuffer = ByteBuffer.Allocate (10).PutShort (33);
var result = new byte [byteBuffer.Capacity ()];

byteBuffer.Rewind ();

byteBuffer.Get (result);

If I understand correctly, without the [array()](https://developer.android.com/reference/java/nio/ByteBuffer#array()) method, in order to access the buffer of a ByteBuffer instance, I have to allocate and copy to an intermediary byte buffer (in your example, it's the result buffer).

Due to performance reasons, I'd like to avoid the allocation and copying, is it possible? Stupid question, can I make a binding myself for the array() method somehow?

jpobst commented 2 years ago

If you want to access the array in C# code, yes, you will need to allocate memory in C# and copy the array there. If there were an array() method binding it would do exactly this for you, but it would still allocate the memory in C# and copy.

Note that it's actually worse than that. If you then want the ByteBuffer to have the changes you made in C# you will need to copy your changes back to the ByteBuffer.

The problem is that the backing byte[] lives in the Java VM. In order to manipulate it in the .NET runtime you either have to copy it to .NET runtime and then copy it back, or you can manipulate it purely using the Java methods defined on ByteBuffer like the various get* and put* methods.

andreinitescu commented 2 years ago

@jpobst Thanks. But what if the ByteBuffer was directly allocated? (ByteBuffer.directAllocate()) I was hoping that the combination of direct memory + being able to access that memory by the array() would have avoided any allocation and copying.

jpobst commented 2 years ago

No, there is no way for the .NET runtime to directly access memory owned by the Java runtime.

A few other options that may or may not help:

andreinitescu commented 2 years ago

Thanks again. I was looking for a way to serialize some an array of some data (integer and strings) and send the bytes from .NET to Java where it's deserialized in the fastest way possible through some kind of ByteBuffer. It does not have to be pretty at all, speed is the most important.

afriscic commented 5 months ago

Recently i had to deal with ByteBuffer so I stumbled upon this issue. Is this abandoned?

jpobst commented 5 months ago

Recently i had to deal with ByteBuffer so I stumbled upon this issue. Is this abandoned?

The status of this issue has not changed. If you need to copy the contents of a ByteBuffer into a C# array, use the workaround code above.

afriscic commented 5 months ago

Thank you for you answer. The thing is that in my use case, I have to edit the content of the ByteBuffer directly and it is quite large (up to 4K video frame). To copy the data out and back into the buffer is quite time consuming. The second alternative is going into direct memory manipulation using unsafe code.

Frankly there are workarounds, but it would be easier to modify the underlying array directly through safe code if there is one...

jpobst commented 5 months ago

Even if the array () method was available, I do not think it would do what you want. The C# byte[] it would return would be a copy and would no longer be linked to the Java one, so changing it would not update the Java one.

I think you have 2 options: