search

tursodatabase / libsql

libSQL is a fork of SQLite that is both Open Source, and Open Contributions.
https://turso.tech/libsql
MIT License
9.17k stars 239 forks source link

Integrate mvsqlite as a distributed database on foundationdb and libsqlite #116

Open fire opened 1 year ago

fire commented 1 year ago

Since libsql is a sqlite derivative, I am trying to explore ways of embedding @losfair's amazing work with libsql.

https://github.com/losfair/mvsqlite

psarna commented 1 year ago

Oh we are exploring that as well! https://github.com/libsql/sqld/blob/main/sqld/src/libsql/mwal/mod.rs , right now it works on top of a fork of mvSQLite which implements a frontend with virtual WAL instead of VFS: https://github.com/psarna/mvsqlite/tree/mwal

psarna commented 1 year ago

And we're interested in the other way round, i.e. making libSQL better suited to run mvSQLite as well, e.g. https://github.com/libsql/libsql/issues/11, https://github.com/libsql/libsql/issues/12, #20

fire commented 1 year ago

Did you document the steps needed to convert libsql to mvsqlite?

fire commented 1 year ago

Also I am investigating how to have a limit of 15 seconds or less configuration option for timeouts in wasmtime

psarna commented 1 year ago

Did you document the steps needed to convert libsql to mvsqlite?

What do you mean by convert? One of the ways mvSQLite is used is to just LD_PRELOAD an application that uses SQLite with a mvSQLite library - you can do exactly the same with an application that uses libSQL, and it will work (https://su3.io/posts/mvsqlite#drop-in-addition), because libSQL is compatible with SQLite ABI.

fire commented 1 year ago

My goal is to make a static library and merge it with my variation of the Godot Engine.

Problems

  1. static library linking is broken in the official release of libsql for wasm
  2. libsql as sqlite3.h / sqlite3.cpp for mvsqlite requires changes to rename "sqlite3_step" to "real_sqlite3_step" and "sqlite3_open_v2" to "real_sqlite3_open_v2"
psarna commented 1 year ago

static library linking is broken in the official release of libsql for wasm @fire can you post the steps to reproduce your issue with static linking? It works for libsql/sqlite3 shell just fine on my x86-64, maybe it's an arch issue?

It's also worth noting that Wasm UDF support, being written in Rust, is not part of the single amalgamation sqlite3.c file, and is instead produced as a static library, libwblibsql.a, which you can find precompiled in .libs directory of libsql-0.1.0-wasm-udf.tar.gz file from our official release.

Alternatively, if you compile from source, the static library can also be found in ./libwblibsql.a.

And if you're integrating Wasm support with a Rust application, it's best to skip linking with the static library and instead integrate directly with the Rust crate: https://crates.io/crates/libsql-wasmtime-bindings.

It's now definitely clear to me that the process is not documented, which makes it hard for users to consume, and we should fix that. But please also let me know if any of the paragraphs above helped!