This PR is to add WASM foreign data wrapper prototype with 3 new FDWs developed using this new framework.
Snowflake FDW, supports data read and modify
Paddle FDW, supports data read only
HelloWorld FDW, a simple hello world FDW for demo purpose
All those 3 FDWs are WASM FDWs.
What is the current behavior?
In current framework, all FDWs needs to be compile into a single Postgres extension and then released to Supabase Postgres image and then deployed to Supabase platform. This architecture has many drawbacks:
Wrappers Release time is too long, a new FDW may need several weeks to be landed on Supabase platform
Wrappers extension size get bigger and bigger with more FDWs added in, but customers normally only need 1-2 FDWs, all the others are wasting resources
Customers cannot use their own FDW easily on Supabase platform, they needs to rely on Supabase team to include their FDWs into Wrappers and deployed to platform
Upgrade to new version of Wrappers isn't easy, customers need to upgrade their whole Postgres server and then can use the new version of Wrappers. This process is time consuming and error-prone, and also impossible to use older version after upgrade.
What is the new behavior?
The new architecture is based on WebAssembly (WASM), each FDW is complied into a WASM component package (Guest) and it is dynamically loaded and run by a new WASM FDW (Host). This consists of 3 parts:
Host: a new FDW using current Wrappers framework and embedding a WASM runtime, which run one or many guest FDWs
Guest: the actual FDW which does the actual data loading and transformation, compiled into a WASM component (.wasm file)
Registry: a WASM FDW registry based on Warg protocol, which serves as a FDW publish and distribution platform. It runs as a Docker compose cluster and has 3 components:
API server
Postgres Database
Frontend
Those 3 parts are quite similar as Docker Host, Container and Docker Hub in concept. Host FDW loads and runs guest FDW when user runs query against a foreign table, and the guest FDWs are published and distributed on FDW registry.
The guest FDW doesn't have to be published on registry, it can be uploaded to anywhere on the internet, as long as the host FDW can access it. The guest FDW can also be a local .wasm file, which is quite useful when developing a new FDW.
Pros:
Each guest FDW is a WASM file, size is small and can easily download and cached in local
Guest FDW can be individually loaded and run on demand
User can develop and use their own guest FDW without go through Supabase
Guest FDW runs in restricted WASM env, only can access outside through HTTP, no system call, no file or socket IO and etc.
User can post their own FDW to Registry (with Supabase authorization), which can be shared with other users
Cons:
Only support HTTP for guest FDW, so users can only develop API-based FDW
WASM env is restricted, many existing Rust libs cannot be compiled to wasm32-unknown-unknown target, especially those ones need to access system calls or IOs
Developing WASM using Rust may not very familiar for developers
Additional context
This PR is mainly to add initial prototype of WASM FDW, test cases and documents for the new FDW are not ready yet.
The FDW Registry code hasn't been included in this PR, I haven't decided to save it this repo or create a new repo, any ideas?
What kind of change does this PR introduce?
This PR is to add WASM foreign data wrapper prototype with 3 new FDWs developed using this new framework.
Snowflake FDW, supports data read and modifyPaddle FDW, supports data read onlyHelloWorld FDW, a simple hello world FDW for demo purposeAll those 3 FDWs are WASM FDWs.
What is the current behavior?
In current framework, all FDWs needs to be compile into a single Postgres extension and then released to Supabase Postgres image and then deployed to Supabase platform. This architecture has many drawbacks:
What is the new behavior?
The new architecture is based on WebAssembly (WASM), each FDW is complied into a WASM component package (Guest) and it is dynamically loaded and run by a new WASM FDW (Host). This consists of 3 parts:
Host: a new FDW using current Wrappers framework and embedding a WASM runtime, which run one or many guest FDWsGuest: the actual FDW which does the actual data loading and transformation, compiled into a WASM component (.wasm file)Registry: a WASM FDW registry based on Warg protocol, which serves as a FDW publish and distribution platform. It runs as a Docker compose cluster and has 3 components:Those 3 parts are quite similar as Docker Host, Container and Docker Hub in concept. Host FDW loads and runs guest FDW when user runs query against a foreign table, and the guest FDWs are published and distributed on FDW registry.
The guest FDW doesn't have to be published on registry, it can be uploaded to anywhere on the internet, as long as the host FDW can access it. The guest FDW can also be a local
.wasmfile, which is quite useful when developing a new FDW.Pros:
Cons:
wasm32-unknown-unknowntarget, especially those ones need to access system calls or IOsAdditional context
The Registry front page: