search

marykdb / maryk

Maryk is a Kotlin Multiplatform library which helps you to store, query and send data in a structured way over multiple platforms. The data store stores any value with a version, so it is possible to request only the changed data or live listen for updates.
Apache License 2.0
8 stars 1 forks source link

support for Kotlin/Native Windows and Linux targets #6

Open aSemy opened 1 year ago

aSemy commented 1 year ago

Continuation of the discussion in #2

  • The linuxX64 and mingwX64 Kotlin Native targets aren't supported by io.maryk.rocksdb:rocksdb-multiplatform - would it possible to enable those targets?
  • linuxArm64 is a newer Kotlin Native target that isn't supported by kotlinx-datetime yet

I see it as quite difficult to fully support the linuxX64 and mingwX64 targets for RocksDB. RocksDB is written in C++ and Kotlin does not provide C++ interop. So we would need to fall back to the way more limited C compatibility layer of RocksDB to provide support. I did once try to do an implementation with the C RocksDB calls but found it too limited for the MarykDB use case.

I could manage to make it work with Darwin/Apple targets thanks to an Objective-C library which wrapped RocksDB which I then expanded. I could then call RocksDB on those platforms thanks to the Kotlin Obj-C interop. As far as I am aware, Kotlin does not provide ObjC compilation for other platforms than the Apple platform. Since both Linux and Windows can always run the JVM I feel that those native targets were less needed to run MarykDB there, unlike the more limited iOS/ipadOS platforms.

So I am still hoping for Kotlin to expand to a limited C++ interop so it is easier to support those targets. Or somebody with more C/C++ experience needs to expand the C api of RocksDB. Or do you see alternatives which I missed?

Originally posted by @jurmous in https://github.com/marykdb/maryk/issues/2#issuecomment-1631052468

Windows and Linux support is quite important for me, since I don't always work on a Mac.

I thought creating a new issue would be best to gather discussions. But feel free to close as "won't fix" if that's the situation!

I've previously worked on creating Kotlin/Native bindings for RocksDB https://github.com/aSemy/kotlin-on-the-rocksdb, and I did get quite far. I just got stuck on trying to build the RocksDB C library for the different targets while still complying with the weird restrictions Kotlin/Native has regarding generally unavailable gcc versions. I can try picking this up again, and maybe I'll have more luck with Kotlin 1.9.

@jurmous Would you be able to be more specific about the functionality that the C library is lacking? This doesn't have to be an exhaustive list, but some specifics to focus on would help. Cheers!

jurmous commented 1 year ago

I once started a RocksDB Native library version with native C calls here: https://github.com/marykdb/rocksdb-multiplatform/compare/master...native-rework

I got quite far but as far as I remember I got stuck with the custom Comparator I need to implement for MarykDB. https://github.com/marykdb/maryk/blob/909e2f4cd451dd06593e1078733c12abcb27daf9/store/rocksdb/src/commonMain/kotlin/maryk/datastore/rocksdb/processors/VersionedComparator.kt But I am not 100% certain but it had to do with those more custom parts. But maybe I overlooked something in the C api back then.

jurmous commented 1 year ago

With RocksDBMultiplatform I try to stay as compatible as possible with the Java SDK of RocksDB so I could use that one for the JVM version. At least I got the RocksDB compilation working for Android, Apple platforms and JVM. Don't know what issues you walked into for the other platforms?

aSemy commented 1 year ago

I'd like to make a Kotlin/Native CLI tool for Mac/Windows/Linux, so the issue is that Maryk isn't available. RocksDB isn't critical, it's just a hobby project, but I thought I'd try and help out if I could since I previously tried to bind Kotlin/Native and RocksDB.

jurmous commented 1 year ago

Maybe the native C RocksDB Multiplatform implementation could be looked into again to see if we could get it to work? It could also be that I got stuck in how to fit it with the Java RocksDB api now I think about it. But we could make platform specific classes to help with certain use cases.

I once started it to support more platforms but also to be more in control of memory instead of leaving things to objC which caused the coroutines issue I mentioned in the coroutines PR.

For me it is also a hobby project btw. But I have less time to work on it with full focus currently, but I can certainly help with all the things I learned along the way.