search

pmem / llpl

Low Level Persistence Library
Other
98 stars 38 forks source link

[Proposal] Add support for DirectByteBuffer #10

Closed Sunny-Island closed 2 years ago

Sunny-Island commented 3 years ago

Background: For I/O intensive java application, we usually use DirectByteBuffer to manage data flow from(to) network and file, Java.nio support using DirectBuffer and channels to avoid copy data into java heap. DirectByteBuffer is a common used byte buffer. When using pmem to accerate I/O with java application, we should avoid copying data into java heap, which will cause GC and latency. Therefore, llpl lib should support write from DirectByteBuffer to PMEM.

Details: DirectByteBuffer will hold a address(located out of Java heap), we could pass this address to PMDK copy_from_array function, and change DirectByteBuffer 's position after copy success.

Sunny-Island commented 3 years ago

@asonje Could you give me some advice about this proposal? I can contribute this part if it works.

asonje commented 3 years ago

This is an interesting proposal and we are thinking about how to support direct ByteBuffer as a source for copying bytes to persistent memory in LLPL. Are you aware that from JDK 14 there is support for persistent MappedByteBuffer? This means that you could write the bytes from the network directly to persistent memory via a MappedByteBuffer. If all you need to do is store the network data in persistent memory then this is a good solution. If you need to manipulate the data with LLPL then you will need an efficient way to copy bytes from a direct ByteBuffer to an LLPL heap.

Sunny-Island commented 3 years ago

@asonje Thank you for your reply! I have read JDK 14's support(jep-352), the best way is to use MappedByteBuffer as buffer writing to pmem. But our project have used DirectByteBuffer and updating JDK version seems unpossible. Are you groups working on DirectByteBuffer supporting now?

asonje commented 3 years ago

@Sunny-Island please take a look at the copy methods added in PR #12 and let me know how it works

Sunny-Island commented 3 years ago

Ok!

Sunny-Island commented 3 years ago

I will try these methods in my project.

Sunny-Island commented 3 years ago

Hi! I just wonder why use native method to get address of DirectByteBuffer instead of DirectByteBuffer.address()? Which one can get the right address of array?

asonje commented 3 years ago

I used JNI to get the address of the DirectByteBuffer as it is the more straight forward approach and definitely gets the correct address. I have also just modified the PR to advance the buffer's position after a copyToByteBuffer.