Tracking issues of OpenDAL wasm32-unknown-unknown target support

Xuanwo commented 10 months ago

WebAssembly (WASM) is a modern binary instruction format for a stack-based virtual machine. It is designed as a portable compilation target for high-level languages like C/C++ and Rust, enabling deployment on the web for client and server applications. WASM aims to enable near-native performance for web applications, providing a more efficient and secure alternative to traditional JavaScript. Its key features include fast loading times, efficient execution, and the ability to run alongside JavaScript, allowing developers to build complex client-side web applications that perform better than those written solely in JavaScript.

By adding WASM support for opendal, we will allow users to use opendal in targets like wasm32-unknown-unknown. In a short word, users can use opendal in web browser!

We have addressed most blockers by https://github.com/apache/incubator-opendal/pull/3796 and adding a basic test for s3 https://github.com/apache/incubator-opendal/pull/3802. It's time for us to move forward.

This tracking issue is used to track our current progress of wasm support. Welcome to join in us!

Tasks

Add OPFS support

After opendal is ready for WASM, we can start to work for opfs now!

[ ] https://github.com/apache/incubator-opendal/issues/2442

Add support for more services

Please run the following command to check if opendal can build under wasm32 arch:

cargo build --target wasm32-unknown-unknown --no-default-features --features=services-s3

Visit https://github.com/apache/incubator-opendal/pull/3796 for possible fix or ask directly in this issue.

It's possible that some services can't support wasm at all, like fs. Please also provide your feedback here. We will collect them into a seperate doc.

[ ] alluxio
[ ] atomicserver
[x] azblob in https://github.com/apache/incubator-opendal/pull/3806
[ ] azdls
[ ] azfile
[ ] b2
[x] ~~cacache~~ (can't support)
[ ] cloudflare_kv
[ ] cos
[ ] d1
[ ] dashmap
[ ] dbfs
[ ] dropbox
[x] ~~etcd~~ (can't support)
[x] ~~foundationdb~~ (can't support)
[x] fs (can't support)
[ ] ftp
[x] gcs: https://github.com/apache/incubator-opendal/pull/3816
[x] gdrive in https://github.com/apache/incubator-opendal/pull/3808
[ ] ghac
[ ] gridfs
[x] ~~hdfs~~ (can't support)
[ ] http
[ ] huggingface
[ ] ipfs
[ ] ipmfs
[ ] libsql
[ ] memcached
[ ] memory
[ ] mini_moka
[ ] moka
[x] ~~mongodb~~ (can't support)
[x] ~~mysql~~ (can't support)
[ ] obs
[ ] onedrive
[ ] oss
[x] ~~persy~~ (can't support)
[x] ~~postgresql~~ (can't support)
[x] ~~redb~~ (can't support)
[x] ~~redis~~ (can't support)
[x] ~~rocksdb~~ (can't support)
[x] s3 (supported in #3802)
[ ] seafile
[x] ~~sftp~~ (can't support)
[x] ~~sled~~ (can't support)
[x] ~~sqlite~~ (can't support)
[ ] supabase
[ ] swift
[x] ~~tikv~~ (can't support)
[ ] upyun
[ ] vercel_artifacts
[ ] webdav
[ ] webhdfs

Testing

[ ] Integrate behavior test for wasm target

Benchmark

[ ] Add benchmarks for opendal wasm

Existing issues

[ ] wasm-pack test --node doesn't work for s3_read_on_wasm
[x] #3809

Zheaoli commented 10 months ago

FS style target can supported by wasm32-wasi.

I will take handle of this

Xuanwo commented 10 months ago

FS style target can supported by wasm32-wasi.

I will take handle of this

You can start a new issue for this. It's out of the goal of this tracking issue.

Zheaoli commented 10 months ago

For now, I think the service based on the TCP socket is not working because the wasm32-unknown-unknown is not support TCP yet, the wasm32-unknown-emscripten can support TCP socket but need some hackc on Server

The TCP service is some service like following bellow:

MySQL
PostgreSQL
MongoDB
Redis

Zheaoli commented 10 months ago

Can we be used on wasmtime?

Can we be used on wasmedge?

For now, we can't use it on both wasm runtimes, because both of the runtimes is based on wasm32-wasi I have reminded it in the previous comment.

The WASI runtime is not support for the wasm-bindgen which is dependent by reqwest now.

If we want to support the wasi runtime, we may need the other HTTP client which is compatible with the WASI spec

Xuanwo commented 10 months ago

For now, we can't use it on both wasm runtimes, because both of the runtimes is based on wasm32-wasi I have reminded it in the previous comment.

Thanks, let's remove them. I just found that support wasm32-unknown-unkown is totally different from support wasm32-wasi. Lesson learnt.

Zheaoli commented 10 months ago

For now, the memory service can both on wasi(means can run in wasmtime and wasmedge, tested) and wasm32. The fs services need remove the tokio::fs

Zheaoli commented 10 months ago

For the record,

wasm-pack test --chrome --headless and wasm-pack test --chrome is both working.

But need some special S3 settings like

[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "GET",
            "POST",
            "PUT",
            "HEAD"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": [],
        "MaxAgeSeconds": 3000
    }
]

suyanhanx commented 10 months ago

incompatible crate found: ring@0.17.5 When I tried to compile with services-gcs, error reported: No available targets are compatible with triple "wasm32-unknown-unknown"

cargo tree -i ring
ring v0.17.5
├── jsonwebtoken v9.2.0
│   └── reqsign v0.14.6
│       └── opendal v0.43.0 
├── rustls v0.21.9
│   ├── hyper-rustls v0.24.2
│   │   └── reqwest v0.11.22
│   │       ├── opendal v0.43.0 
│   │       └── reqsign v0.14.6 (*)
│   ├── reqwest v0.11.22 (*)
│   └── tokio-rustls v0.24.1
│       ├── hyper-rustls v0.24.2 (*)
│       └── reqwest v0.11.22 (*)
├── rustls-webpki v0.101.7
│   └── rustls v0.21.9 (*)
└── sct v0.7.1
    └── rustls v0.21.9 (*)

Related issue: https://github.com/briansmith/ring/issues/657 It looks like ring isn't going to support wasm for a while.

Xuanwo commented 10 months ago

When I tried to compile with services-gcs, error reported:

Let's mark gcs as unsupported since gcs will require jsonwebtoken (and ring) for JWT.

gabrielgrant commented 10 months ago

@suyanhanx it seems surprising that ring would be failing in that way: the issue you linked seemed to say that it should be able to build for wasm32-unknown-unknown: https://github.com/briansmith/ring/issues/657#issuecomment-509877297

the author (@briansmith) does note that some functionality isn't available yet, but is open to extending support. do we know what is being used that isn't supported?

Xuanwo commented 10 months ago

it seems surprising that ring would be failing in that way

Thanks for raising this up! I re-visit the gcs support on wasm, and confirmed that gcs is able to compile on wasm32-unknown-unknown. I will start a PR to address this.

suyanhanx commented 10 months ago

@suyanhanx it seems surprising that ring would be failing in that way: the issue you linked seemed to say that it should be able to build for wasm32-unknown-unknown: briansmith/ring#657 (comment)

the author (@briansmith) does note that some functionality isn't available yet, but is open to extending support. do we know what is being used that isn't supported?

Thank you for pointing that out! I've tried to compile ring before but it kept failing. Finally, I realized that the llvm comes with macos doesn't have a backend of wasm. I reinstalled llvm and it compiled successfully. Sorry for the confusion!

fyears commented 10 months ago

Hi,

I am very interested in trying the wasm version of opendal for my project But I am not a professional rust developer though (I know something about javascript/typescript but none of rust), so I have some questions/ suggestions:

any doc to use the wasm version? by importing any library in js/typescript? any npm package?
would you like to consider adding hooks for js fetch function? For example, I plan to use opendal in a special js environment and I have to use a special fetch function to get around the annoying cors limitatioms. Moreover, users might need to intercept/monitor/debug some network connections .
would you like to consider adding hooks for js fs functiom? For example, in browser environment, file system might need some mocking by using indexeddb.
in js world, there is a technique called “tree shaking” (by importing part of the package). For example developers only interested in s3 might only import opendal/s3 without importing others. It’s especially useful to support this because users have to download the assets in browser environment. Is it possible for opendal to support tree shaking?

Thank you so much and happy new year!

Young-Flash commented 10 months ago

It is still in a early stage and has not yet reached the stage where it can be released as a lib.

Maybe you could take a look at OpenDAL node.js binding.

Xuanwo commented 10 months ago

Hi @fyears, thank you for your interest! As @Young-Flash mentioned, our WebAssembly support is still in the early stages and not yet ready for practical use.

any doc to use the wasm version? by importing any library in js/typescript? any npm package?

Not yet. WASM support is still in active development, and we need to address existing bugs such as https://github.com/apache/incubator-opendal/issues/3810 before proceeding.

We haven't released it yet. (However, you can create a wasm library by running cargo add opendal and building it.)

would you like to consider adding hooks for js fetch function? For example, I plan to use opendal in a special js environment and I have to use a special fetch function to get around the annoying cors limitatioms. Moreover, users might need to intercept/monitor/debug some network connections .

We are using web-sys now, and I'm not sure if web-sys support this use case.

would you like to consider adding hooks for js fs functiom? For example, in browser environment, file system might need some mocking by using indexeddb.

We will not support the fs service on wasm, which means there's no need to provide hooks either. We have a plan to support opfs: https://github.com/apache/incubator-opendal/issues/2442. If users require IndexedDB, we can offer native support for it.

in js world, there is a technique called “tree shaking” (by importing part of the package). For example developers only interested in s3 might only import opendal/s3 without importing others. It’s especially useful to support this because users have to download the assets in browser environment. Is it possible for opendal to support tree shaking?

Nice question. We want this as well, but I'm not sure how to implement it. We can re-visit this after we have build a usable opendal wasm lib.

fyears commented 10 months ago

web-sys

After a quick search, it seems that web-sys has a clear mapping to js web api. However I think the most dirty work comes to the data crossing boundaries: wasm (prepare the net calls) -> js (fetch hook) -> wasm (reading the result)

Let's wait for the stable wasm releases first... LOL

Furthermore, my intention is to use opendal in my 320k downloaded project (remotely-save if you are interested). If you want to talk about the user case, you are welcome to DM me (username fyears) on discord .

Xuanwo commented 10 months ago

Furthermore, my intention is to use opendal in my 320k downloaded project (remotely-save if you are interested). If you want to talk about the user case, you are welcome to DM me (username fyears) on discord .

Interesting use case! I personally want OpenDAL to be usable in web browsers and Electron as well.

waynexia commented 7 months ago

I've played with the HTTP operator and it works well. Except for the CORS problem mentioned before. It needs the support from server to add CORS header (like the S3 operator)

fyears commented 7 months ago

may i ask is it usable now?

apache / opendal