[feature] Make it easier to use monero-javascript for front end development

[CryptoGrampy]

For such a comprehensive and easy to use library for node, it's not possible to simply npm i monero-javascript and use it for front end/browser apps (yes, this is documented), and it's quite painful to integrate it with even basic create-react-app or vuejs boilerplate code. I think this is a big problem and is likely a bottleneck for a lot of would-be front end development.

I would really like to be able to simply install and use this library with no fussing with webpack and build configs if such a thing is possible.... i.e. I should be able to run create-react-app, or npm init vue@latest, run npm i monero-javascript, and be ready to roll.

Example of what needs to be done to use the library with Vue (see the fallbacks and additional packages necessary in package.json): https://github.com/xmrdotgift/wallet/blob/master/webpack.config.js

[woodser]

I support this and would also like to see typescript implemented.

I'm looking for help on both. Thanks.

[trasherdk]

I was looking to use this thing in sveltekit, but that seems to be blocked by the vite tooling.

https://github.com/vitejs/vite/issues/5075 links to a bunch of issues describing the problem.

It's not really a monero-javascript problem, as it's more a tooling not being ready for wasm problem, I think.

Edit: Maybe this Using WASM in SvelteKit can bring me closer.

[j-berman]

I've been messing around with this off and on. Here's a workaround to get something going with create-react-app:

npx create-react-app <app name>
cd <app name>

# necessary to avoid needing to eject or override create-react-app's defaults
# see: https://github.com/facebook/create-react-app/issues/11756
npm uninstall react-scripts && npm install react-scripts@4

npm i monero-javascript

cp node_modules/monero-javascript/dist/* public
cp node_modules/monero-javascript/src/main/js/common/MoneroWebWorker.js public/MoneroWebWorker.js

[trasherdk]

I'm hoping svelte-emscripten is up for the task.

The required emscripten linker flags got me worried, as I can't find those. What do you think @woodser ?

:heavy_check_mark: -s MODULARIZE=1 :x: -s ENVIRONMENT='web' :x: -s EXTRA_EXPORTED_RUNTIME_METHODS="['specialHTMLTargets', 'JSEvents', 'GL', 'callMain', 'abort']"

How to Use

When you compile your program in Emscripten, use the output filetype *.js and use the linker flags below. These are required to adapt your program into component form:

    -s MODULARIZE=1

    -s ENVIRONMENT='web'

    -s EXTRA_EXPORTED_RUNTIME_METHODS="['specialHTMLTargets',
           'JSEvents', 'GL', 'callMain', 'abort']"

Then, use the component in your Svelte app:

import { default as Emscripten } from 'svelte-emscripten';
import { default as module } from './module.js';

<Emscripten
  module={module}
  canvas={true}
  console={false}
  verticalOrientation={false}
  options={
    { 
      autorun: true,
      wasmPath: 'path/to/module.wasm'
    }
  }
/>

[woodser]

The required emscripten linker flags got me worried, as I can't find those.

These linker flags are defined here.

You get an error if you add them? What's the error?

Sometimes external nodejs dependencies need to be excluded from the build, here.

[trasherdk]

I'm not getting far enough to start tweaking linker flags.

Trying to build standard on Slackware 14.2 (Intel x64) ends with:

-- Configuring done
-- Generating done
-- Build files have been written to: /path/to/build-monero-javascript/github-monero-javascript/build
make: cmake --build . -j4
Unknown argument -j4
Usage: cmake --build <dir> [options] [-- [native-options]]
Options:
  <dir>          = Project binary directory to be built.
  --target <tgt> = Build <tgt> instead of default targets.
  --config <cfg> = For multi-configuration tools, choose <cfg>.
  --clean-first  = Build target 'clean' first, then build.
                   (To clean only, use --target 'clean'.)
  --use-stderr   = Ignored.  Behavior is default in CMake >= 3.0.
  --             = Pass remaining options to the native tool.
emmake: error: 'cmake --build . -j4' failed (returned 1)

[trasherdk]

The build script:

#!/bin/bash

BASE=$(dirname "$(realpath "$0")")

cd "${BASE}" || exit 1

if [ ! -d "${BASE}/emsdk" ]; then
    git clone https://github.com/emscripten-core/emsdk.git
fi

cd "${BASE}/emsdk" || exit 1

git pull \
&& ./emsdk install latest-upstream \
&& ./emsdk activate latest-upstream \
&& source ./emsdk_env.sh

export EMSCRIPTEN="${BASE}/emsdk/upstream/emscripten"
export EMSDK="${BASE}/emsdk"
export EM_CONFIG="${BASE}/emsdk/.emscripten"
export EMSDK_NODE="${BASE}/emsdk/node/14.18.2_64bit/bin/node"

cd "${BASE}/github-monero-javascript" || exit 1

./bin/update_submodules.sh || exit 1

PATCH_FILE="${BASE}/github-monero-javascript/external/monero-cpp/external/monero-project/src/crypto/wallet/CMakeLists.txt"
sed -i 's/set(MONERO_WALLET_CRYPTO_LIBRARY "auto"/set(MONERO_WALLET_CRYPTO_LIBRARY "cn"/g' "${PATCH_FILE}"

HEADER_FILE="${BASE}/../build-monero-pool/staging-area/monero/monero-build/translations/translation_files.h"
HEADER_DEST="${BASE}/github-monero-javascript/external/monero-cpp/external/monero-project/build/release/translations/"

if [ ! -f "${HEADER_DEST}/$(basename "${HEADER_FILE}")" ]; then
    echo "*${WHITE} copy 'translation_files.h' ${RESTORE}"
    sleep 3
    cp "${HEADER_FILE}" "${HEADER_DEST}"
fi

./bin/build_all.sh || exit 1

Edit 1: Forgot to patch MONERO_WALLET_CRYPTO_LIBRARY Edit 2: Copy translation_files.h into include path

[trasherdk]

Removing -j$HOST_NCORES from bin/build_wasm_emscripten.sh moves past that error.

Next error is:

monero-project/src/common/i18n.cpp:36:10: fatal error: 'translation_files.h' file not found
#include "translation_files.h"

That file does not exist in external/monero-cpp/external/monero-project/translations/

Isn't it supposed to be generated by: external/monero-cpp/external/monero-project/translations/generate_translations_header.c ?

[woodser]

That file does not exist in external/monero-cpp/external/monero-project/translations/

I'd try building the monero-project submodule directly to ensure it succeeds, and build twice since some files are not built the first time, e.g.:

cd ./external/monero-cpp/external/monero-project
make release-static -j4
make release-static -j4

Then the translations directory should exist.

[trasherdk]

Building monero in ./bin/build_all.sh should be after boost has been build. Otherwise it will fail if system boost does not have static libraries installed.

Maybe setting BOOST_ROOT to build/boost will do the trick.

[trasherdk]

Nope, no cake.

BOOST_ROOT=$(realpath "/path/to/github-monero-javascript/build/boost") \
BOOST_LIBRARYDIR="${BOOST_ROOT}/lib/" \
make release-static -j4

results in:

CMake Error at /usr/share/cmake-3.5/Modules/FindBoost.cmake:1657 (message):
  Unable to find the requested Boost libraries.

  Boost version: 1.59.0

  Boost include path: /usr/include

[trasherdk]

Building monero standalone is just fine.

[ 98%] Built target gen_multisig
[ 98%] Building CXX object src/daemon/CMakeFiles/daemon.dir/command_server.cpp.o
[ 98%] Building CXX object src/daemon/CMakeFiles/daemon.dir/daemon.cpp.o
[100%] Building CXX object src/daemon/CMakeFiles/daemon.dir/executor.cpp.o
[100%] Building CXX object src/daemon/CMakeFiles/daemon.dir/main.cpp.o
[100%] Building CXX object src/daemon/CMakeFiles/daemon.dir/rpc_command_executor.cpp.o
[100%] Linking CXX executable ../../bin/monerod
[100%] Built target daemon
make :  OK 

Building in external is not. Tried woodser/monero and that failed too.

CMake Error at /usr/share/cmake-3.5/Modules/FindBoost.cmake:1657 (message):
  Unable to find the requested Boost libraries.

  Boost version: 1.76.0

  Boost include path:
  /home/crypto/local/build/build-monero-javascript/github-monero-javascript/build/boost/include

  Could not find the following Boost libraries:

          boost_filesystem
          boost_date_time
          boost_program_options
          boost_locale

Changing bin/build_boost_emscripten.sh

./bootstrap.sh \
- --with-libraries=system,thread,chrono,serialization,regex \
+ --with-libraries=system,thread,chrono,serialization,regex,filesystem,date_time,program_options,locale \
2>&1

Results in:

CMake Error at /usr/share/cmake-3.5/Modules/FindBoost.cmake:1657 (message):
  Unable to find the requested Boost libraries.

  Boost version: 1.76.0

  Boost include path:
  /home/crypto/local/build/build-monero-javascript/github-monero-javascript/build/boost/include

  Could not find the following Boost libraries:

          boost_locale

[trasherdk]

The updated build script:

#!/bin/bash

BASE=$(dirname "$(realpath "$0")")

cd "${BASE}" || exit 1

if [ ! -d "${BASE}/emsdk" ]; then
    git clone https://github.com/emscripten-core/emsdk.git
fi

cd "${BASE}/emsdk" || exit 1

git pull \
&& ./emsdk install latest-upstream \
&& ./emsdk activate latest-upstream \
&& source ./emsdk_env.sh

export EMSCRIPTEN="${BASE}/emsdk/upstream/emscripten"
export EMSDK="${BASE}/emsdk"
export EM_CONFIG="${BASE}/emsdk/.emscripten"
export EMSDK_NODE="${BASE}/emsdk/node/14.18.2_64bit/bin/node"

cd "${BASE}/github-monero-javascript" || exit 1

./bin/update_submodules.sh || exit 1

echo "${WHITE}* Patching ${YELLOW}build_boost_emscripten.sh${RESTORE}"

PATCH_FILE="${BASE}/monero-javascript/bin/build_boost_emscripten.sh"
PATCH_FROM='--with-libraries=system,thread,chrono,serialization,regex \\'
PATCH_TO='--with-libraries=system,thread,chrono,serialization,regex,locale,filesystem,date_time,program_options \\'
sed -i "s/${PATCH_FROM}/${PATCH_TO}/g" "${PATCH_FILE}" || exit 1

./bin/build_boost_emscripten.sh || exit 1

PATCH_FILE="${BASE}/github-monero-javascript/external/monero-cpp/external/monero-project/src/crypto/wallet/CMakeLists.txt"
sed -i 's/set(MONERO_WALLET_CRYPTO_LIBRARY "auto"/set(MONERO_WALLET_CRYPTO_LIBRARY "cn"/g' "${PATCH_FILE}"

MONERO_BUILD="${BASE}/github-monero-javascript/external/monero-cpp/external/monero-project/build"
if [ ! -d "${MONERO_BUILD}" ]; then
    mkdir "${MONERO_BUILD}" || exit 1
fi

cd "${MONERO_BUILD}" || exit 1

export BOOST_ROOT="${BASE}/github-monero-javascript/build/boost"
export BOOST_LIBRARYDIR="${BOOST_ROOT}/lib"
cmake .. || exit 1
make release-static -j2 || exit 1
make release-static -j2 || exit 1

#HEADER_FILE="${BASE}/../build-monero-pool/staging-area/monero/monero-build/translations/translation_files.h"
#HEADER_DEST="${BASE}/github-monero-javascript/external/monero-cpp/external/monero-project/build/release/translations/"

#if [ ! -f "${HEADER_DEST}/$(basename "${HEADER_FILE}")" ]; then
#    echo "*${WHITE} copy 'translation_files.h' ${RESTORE}"
#    sleep 3
#    cp "${HEADER_FILE}" "${HEADER_DEST}"
#fi

cd "${BASE}/github-monero-javascript/" || exit 1

./bin/build_all.sh || exit 1

Edit 1: submodules before boost :) Edit 2: patch build_boost_emscripten.sh Edit 3: Fix patch build_boost_emscripten.sh :)

[trasherdk]

Okay, this is beyond me. I can't get that locale thing to build.

[woodser]

It shouldn't be necessary to modify the boost libraries in bin/build_boost_emscripten.sh.

You still get boost errors without that modification? And have you installed boost with sudo apt install libboost-all-dev?

[trasherdk]

I'm on Slackware 14.2 (x64). Slackware does not include static libraries at all.

[DangerOnTheRanger]

What would need to change in order to remove the need to copy files out of dist/ when targeting the browser? I'm interested in fixing this.

[CryptoGrampy]

What would need to change in order to remove the need to copy files out of dist/ when targeting the browser? I'm interested in fixing this.

This is a bit out of my wheelhouse, but I did a little searching on npm and found a couple examples where they're providing their wasm as browser-importable js modules: https://www.npmjs.com/package/brotli-wasm https://github.com/jungomi/xxhash-wasm

Would be great if there was just one minified / tree-shakable js file that just works on browser or node. We're definitely almost there. Searching for libraries on npm with 'browser and node.js' might give some good examples of what might need to be done. Axios is one that's usable on node and browser without any user-involvement beside installing, and it just works.

[DangerOnTheRanger]

What would need to change in order to remove the need to copy files out of dist/ when targeting the browser? I'm interested in fixing this.

This is a bit out of my wheelhouse, but I did a little searching on npm and found a couple examples where they're providing their wasm as browser-importable js modules: https://www.npmjs.com/package/brotli-wasm https://github.com/jungomi/xxhash-wasm

Would be great if there was just one minified / tree-shakable js file that just works on browser or node. We're definitely almost there. Searching for libraries on npm with 'browser and node.js' might give some good examples of what might need to be done. Axios is one that's usable on node and browser without any user-involvement beside installing, and it just works.

I think you're right, the solution involves bundling plus a package.json that points to the right files (xxhash for example). I think automating the emscripten part of the process to the point where all one needs to do after a fresh git clone should be to run npm run build-wasm should also be a goal, but maybe others have different thoughts on that. Assuming we do that too, the solution then looks like:

1. Automate building wasm bindings (this can be done with Docker and land as a separate PR if need be)
2. Bundle/tree-shake built JS with something like rollup
3. Point to built CJS/UMD/ESM modules with the package.json that gets published to npm

Having laid out everything like this I don't think it's a ton of work (automating all the wasm stuff would probably be the hardest part), but it's possible I've missed something.

[CryptoGrampy]

@woodser @DangerOnTheRanger Apparently, it's possible to generate WASM as a pre-bundled JS file rather than having separate .WASM that needs to be manually copied by downstream library consumers. This would be a huge benefit for monero-javascript browser library users as they would simply import monero-javascript like any other JS library. MyMonero had a very similar setup with their libraries, and they migrated to a 'unified' WASM/JS file in this commit: https://github.com/mymonero/mymonero-utils/commit/07d6a94d1d4cd23b95496243c93fa8d765888190

[woodser]

@CryptoGrampy I made the change from the PR and created single .js files. However, the .js file for the full wallet is 11.2 mbs, whereas the combined size of the old .js and .wasm is 6.7 mbs. Additionally, monero_web_worker.js punches out to 11.6 mbs, instead of the old 2.2 mbs. I need to find a way to optimize these.

[woodser]

Hopefully this issue is resolved with the latest v0.9.2 release (and project rename to monero-ts).