Added support PQC in the browser with the target wasm32-wasi now supported

[john-sharratt]

As a part of a project I am working on there is a heavy use of quantum resistant crypto.

Now that I am porting the library (https://github.com/john-sharratt/ate) over to WebAssembly so that it can run in a web browser I need the same crypto functions to work in wasm32 under wasi however there are no known libraries in rust that support such routines other than pqcrypto - hence I started porting it.

Given that pqcrypto does not yet work I forked it to add support for compiling to wasm32-wasi (https://github.com/john-sharratt/pqcrypto-wasi) - most of it compiles except for the random generator which is runtime specific thus I have added a function within WASI.

In order to test simply run: cargo build --no-default-features --target wasm32-wasi

Note: While many files are changed in this commit this is due to the nature of the code generation script - in reality the only files I changed where...

pqcrypto-internals/build.rs pqcrypto-internals/include/.h pqcrypto-internals/cfiles/.c pqcrypto-templates/schema/build.rs.j2

The changes that were made was to support an external sys-root which must be supplied in the parent folder outside of the scope of this build.

Due to the changes you made with breaking the hard dependency on STD there are now no more changes required elsewhere upstream.

[thomwiggers]

I'm sorry, I had some commits that I'd used to prepare a release but forgot to push as I got called out to a meeting. Can you rebase?

I will try to make some time for testing after that.

[john-sharratt]

No worrries - I'll rebase it now and retest

[john-sharratt]

There's a problem...

./generate-implementations.py Traceback (most recent call last): File "./generate-implementations.py", line 180, in generate_scheme(name, 'kem', properties) File "./generate-implementations.py", line 95, in generate_scheme metadatas[scheme['name']] = read_scheme_metadata(type, scheme['name']) File "./generate-implementations.py", line 22, in read_scheme_metadata with open(metadata_path) as f: FileNotFoundError: [Errno 2] No such file or directory: 'pqclean/crypto_kem/ntrulpr953/META.yml'

[john-sharratt]

It might actually be my fault - I'll work on it some more

[john-sharratt]

Ok - all fixed up now.

[john-sharratt]

Found a linking error on one of my test runs, this last commit should fix it. Will continue my testing

[john-sharratt]

[john-sharratt]

confirmed that it compiles and works in browsers - awesome

[john-sharratt]

@thomwiggers any news on this merge?

[john-sharratt]

@thomwiggers I published the pqcrypto crates with a postfix "-wasi" and commit to delete them again - my project can't wait any more but whatever you want me to change in the code to get this merge finished... just let me know.

Willing to put the time in to help.

Thanks for the help so far

[thomwiggers]

This should also really be added to CI somehow.

[john-sharratt]

This should also really be added to CI somehow.

I can write some bash scripts if you want me to show you how to test it? The tool chain would have to be downloaded though so I'll split it into a script to install the tool chain and a script to run the tests

You want that in the merge request or just attached to this comment thread?

[john-sharratt]

I looked into setting up CI/CD for this but it was too much work in the end to make a special case for this project. It will of course be picked up in my CI/CD for the ATE project but thats obviously after you publish.

If you want any help in the form of tips and questions/answers let me know.

[john-sharratt]

I just don't have the time to do the CI/CD for WASM on this project - helping with the patch and processing review comments is all I can offer.

[john-sharratt]

@thomwiggers - what do you want to do with this PR as I made the changes but you modified the build.rs file and I can't keep updating this pull request.

In the meantime I've got this all working by republishing the libraries but its not good to fork a WASI version from the mainline so its in the communities interests to sort this.

Cheers

[thomwiggers]

I'm afraid I just don't know enough about WASI's restrictions to comfortably merge this right now, without CI. It doesn't have to be complicated; just running the build and tests is probably enough.

[john-sharratt]

@thomwiggers No worries - let me see what I can do

[john-sharratt]

There we go - all merge conflicts resolved and added some CI/CD that will download the SDK and build the crypto libraries against the WASI build target. This will find 99% of the errors as there really isn't that much else that can go wrong. Executing cargo wasi test will be too hard to get working as its not running the tests in rust.

See here: https://github.com/john-sharratt/pqcrypto-wasi/runs/4330955456?check_suite_focus=true

[john-sharratt]

All green. https://github.com/john-sharratt/pqcrypto-wasi/runs/4379567593?check_suite_focus=true

[john-sharratt]

Cool... let me know when you next publish to cargo.io and I'll kill the other packages and update dependencies for downstream projects.