Our `element-desktop` is not built with encrypted chat search support

[colemickens]

Describe the bug

1. Open Riot-Desktop
2. Settings -> Security & Privacy

Under the "Message search" heading, observe:

Riot is missing some components required for securely caching encrypted messages locally. If you'd like to experiment with this feature, build a custom Riot Desktop with search components added.

it links to here: https://github.com/vector-im/riot-web/blob/develop/docs/native-node-modules.md#adding-seshat-for-search-in-e2e-encrypted-rooms

Notify maintainers cc: @pacien @worldofpeace

Metadata

 - system: `"x86_64-linux"`
 - host os: `Linux 5.6.12, NixOS, 20.09pre-git (Nightingale)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.4`
 - nixpkgs: `/home/cole/code/nixpkgs/cmpkgs`

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute: riot-desktop

[livnev]

Would be fantastic to have riot-desktop built with seshat. I tried building it myself but couldn't do it. :cry:

[vbrandl]

Just stumbled across this myself. What's the current state on this? What needs to be done before this can be fixed?

[ajs124]

So, element seems to use its own build system hak to build these native dependencies. It tries to download and build those itself with cargo and stuff. Seems quite complicated to package.

[sagehane]

I would like to add that support for desktop notifications would be also appreciated. (They showed up on Arch's version, at least)

Edit: This problem has been resolved since

[ghost]

I made some progress. This solves the issue of downloading the hakDependencies. Unfortunately they are not locked in any way, so we need to lock them at packaging-time (is that even a term?) by adding them to yarn.lock and yarn.nix. Since yarn2nix always runs yarn with --ignore-scripts, we can safely ignore the upstream reasoning for fetching them outside of yarn dependencies.

I'm now at the point where it tries to build seshat in the element-desktop derivation, but unfortunately they don't provide a Cargo lockfile. I'm working on resolving this problem upstream: https://github.com/matrix-org/seshat/pull/79

``` diff --git a/pkgs/applications/networking/instant-messengers/element/element-desktop.nix b/pkgs/applications/networking/instant-messengers/element/element-desktop.nix index 332563476d1..c73000234e4 100644 --- a/pkgs/applications/networking/instant-messengers/element/element-desktop.nix +++ b/pkgs/applications/networking/instant-messengers/element/element-desktop.nix @@ -1,6 +1,7 @@ { lib, stdenv, fetchFromGitHub , makeWrapper, makeDesktopItem, mkYarnPackage , electron, element-web +, python }: # Notes for maintainers: # * versions of `element-web` and `element-desktop` should be kept in sync. @@ -20,9 +21,31 @@ in mkYarnPackage rec { inherit version src; packageJSON = ./element-desktop-package.json; + yarnLock = ./element-desktop-yarndeps.lock; yarnNix = ./element-desktop-yarndeps.nix; - nativeBuildInputs = [ makeWrapper ]; + nativeBuildInputs = [ makeWrapper python ]; + + postConfigure = '' + rm deps/element-desktop/node_modules + cp -R "$node_modules" deps/element-desktop + chmod -R u+w deps/element-desktop + ''; + + preBuild = '' + ( + cd deps/element-desktop + mkdir -p .hak/matrix-seshat + ln -s $PWD/node_modules/matrix-seshat .hak/matrix-seshat/build + mkdir -p .hak/keytar + ln -s $PWD/node_modules/keytar .hak/keytar/build + node scripts/hak/index.js check + node scripts/hak/index.js build + ) + ''; installPhase = '' # resources diff --git a/pkgs/applications/networking/instant-messengers/element/update-element-desktop.sh b/pkgs/applications/networking/instant-messengers/element/update-element-desktop.sh index 69d0d3d7072..4638997fc9b 100755 --- a/pkgs/applications/networking/instant-messengers/element/update-element-desktop.sh +++ b/pkgs/applications/networking/instant-messengers/element/update-element-desktop.sh @@ -1,5 +1,5 @@ #!/usr/bin/env nix-shell -#!nix-shell -I nixpkgs=../../../../../ -i bash -p wget yarn2nix +#!nix-shell -I nixpkgs=../../../../../ -i bash -p curl wget yarn2nix set -euo pipefail @@ -11,7 +11,28 @@ fi RIOT_WEB_SRC="https://raw.githubusercontent.com/vector-im/element-desktop/$1" -wget "$RIOT_WEB_SRC/package.json" -O element-desktop-package.json -wget "$RIOT_WEB_SRC/yarn.lock" -O element-desktop-yarndeps.lock +# Here we deal with the so-called hakDependencies. They are not part of yarn.lock. +# Upstream doesn't add them to the dependencies field, because they want to prevent +# the install scripts to be run by npm/yarn. Fortunately, yarn2nix doesn't run +# install scripts by default, so it's okay to add them to the dependencies for us. +# For more information, read the description at +# https://github.com/vector-im/element-desktop/tree/v1.7.17/scripts/hak + +TMPDIR="$(mktemp -d)" +trap "rm -rf $TMPDIR;" EXIT + +pushd "$TMPDIR" + +curl "$RIOT_WEB_SRC/package.json" \ + | jq '. + { dependencies: (.dependencies + .hakDependencies) }' \ ```

[ghost]

It turns out to be much more complicated, because the cargo build of seshat-node-native calls npm install, so we have multiple layers of npm calling yarn calling cargo calling npm. With lots of things being downloaded and executed. It's a huge shitshow.

[teutat3s]

Is there anything we could do to help fix building element-desktop with encrypted chat search working on NixOS?

[ghost]

I poured some (many) more hours into this today. I can build the two native modules, matrix-seshat and keytar, both in a seperate derivation and as part of the element-desktop build. However, I can not build them correctly against electron's node lib, so when starting the compiled thing, it refuses to load the module. Attempting to build against electron gives a mystic error.

$ nix-build https://github.com/petabyteboy/nixpkgs/archive/feature/seshat1.tar.gz -A element-desktop.seshat-node

Or the second approach, with everything in one derivation using the hak tool:

$ nix-build https://github.com/petabyteboy/nixpkgs/archive/feature/seshat2.tar.gz -A element-desktop

``` Compiling notify v4.0.15 Compiling r2d2 v0.8.9 Compiling hkdf v0.10.0 Compiling aes-ctr v0.6.0 Compiling tempfile v3.2.0 Compiling r2d2_sqlite v0.17.0 Compiling pbkdf2 v0.6.0 Compiling futures-macro v0.3.12 Compiling thiserror-impl v1.0.23 Compiling failure v0.1.8 Compiling futures-util v0.3.12 The following warnings were emitted during compilation: warning: ar: /build/source/seshat-node/native/target/release/build/neon-sys-f7ab2e53f2e370fb/out/native/build/Release/obj.target/neon/src/neon.o: No such file or directory error: failed to run custom build command for `neon-sys v0.4.0` Caused by: process didn't exit successfully: `/build/source/seshat-node/native/target/release/build/neon-sys-564e91a635bdfcb5/build-script-build` (exit code: 1) --- stdout Skipping node-gyp installation as part of npm install. > @ build-release /build/source/seshat-node/native/target/release/build/neon-sys-f7ab2e53f2e370fb/out/native > node-gyp build make: Entering directory '/build/source/seshat-node/native/target/release/build/neon-sys-f7ab2e53f2e370fb/out/native/build' CXX(target) Release/obj.target/neon/src/neon.o make: Leaving directory '/build/source/seshat-node/native/target/release/build/neon-sys-f7ab2e53f2e370fb/out/native/build' TARGET = Some("x86_64-unknown-linux-gnu") HOST = Some("x86_64-unknown-linux-gnu") AR_x86_64-unknown-linux-gnu = None AR_x86_64_unknown_linux_gnu = None HOST_AR = None AR = Some("ar") running: "ar" "cq" "/build/source/seshat-node/native/target/release/build/neon-sys-f7ab2e53f2e370fb/out/libneon.a" "/build/source/seshat-node/native/target/release/build/neon-sys-f7ab2e53f2e370fb/out/native/build/Release/obj.target/neon/src/neon.o" cargo:warning=ar: /build/source/seshat-node/native/target/release/build/neon-sys-f7ab2e53f2e370fb/out/native/build/Release/obj.target/neon/src/neon.o: No such file or directory exit code: 1 --- stderr In file included from /build/source/seshat-node/.electron-gyp/13.0.1/include/node/node.h:67, from ../../../../../../../../node_modules/nan/nan.h:56, from ../src/neon.cc:2: /build/source/seshat-node/.electron-gyp/13.0.1/include/node/v8.h:1670:79: warning: 'using ResolveCallback = class v8::MaybeLocal (*)(class v8::Local, class v8::Local, class v8::Local)' is deprecated: Use ResolveModuleCallback [-Wdeprecated-declarations] 1670 | ResolveCallback callback); | ^ /build/source/seshat-node/.electron-gyp/13.0.1/include/node/v8.h:1652:9: note: declared here 1652 | using ResolveCallback V8_DEPRECATE_SOON("Use ResolveModuleCallback") = | ^~~~~~~~~~~~~~~ /build/source/seshat-node/.electron-gyp/13.0.1/include/node/v8.h:8652:51: warning: 'using HostImportModuleDynamicallyCallback = class v8::MaybeLocal (*)(class v8::Local, class v8::Local, class v8::Local)' is deprecated: Use HostImportModuleDynamicallyWithImportAssertionsCallback instead [-Wdeprecated-declarations] 8652 | HostImportModuleDynamicallyCallback callback); | ^ /build/source/seshat-node/.electron-gyp/13.0.1/include/node/v8.h:7291:7: note: declared here 7291 | using HostImportModuleDynamicallyCallback V8_DEPRECATE_SOON( | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In file included from ../src/neon.cc:2: ../../../../../../../../node_modules/nan/nan.h: In function 'void Nan::AsyncQueueWorker(Nan::AsyncWorker*)': ../../../../../../../../node_modules/nan/nan.h:2294:7: warning: cast between incompatible function types from 'void (*)(uv_work_t*)' {aka 'void (*)(uv_work_s*)'} to 'uv_after_work_cb' {aka 'void (*)(uv_work_s*, int)'} [-Wcast-function-type] 2294 | , reinterpret_cast(AsyncExecuteComplete) | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In file included from ../src/neon.cc:10: ../src/neon_task.h: In member function 'void neon::Task::complete()': ../src/neon_task.h:62:70: warning: 'v8::Local node::MakeCallback(v8::Isolate*, v8::Local, v8::Local, int, v8::Local*)' is deprecated: Use MakeCallback(..., async_context) [-Wdeprecated-declarations] 62 | node::MakeCallback(isolate_, context->Global(), callback, 2, argv); | ^ In file included from ../../../../../../../../node_modules/nan/nan.h:56, from ../src/neon.cc:2: /build/source/seshat-node/.electron-gyp/13.0.1/include/node/node.h:192:50: note: declared here 192 | NODE_EXTERN v8::Local MakeCallback( | ^~~~~~~~~~~~ /build/source/seshat-node/.electron-gyp/13.0.1/include/node/node.h:108:42: note: in definition of macro 'NODE_DEPRECATED' 108 | __attribute__((deprecated(message))) declarator | ^~~~~~~~~~ In file included from ../src/neon.cc:10: ../src/neon_task.h: In function 'void neon::queue_task(neon::Task*)': ../src/neon_task.h:98:17: warning: cast between incompatible function types from 'void (*)(uv_work_t*)' {aka 'void (*)(uv_work_s*)'} to 'uv_after_work_cb' {aka 'void (*)(uv_work_s*, int)'} [-Wcast-function-type] 98 | (uv_after_work_cb)complete_task); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ../src/neon.cc: In function 'size_t Neon_ArrayBuffer_Data(void**, v8::Local)': ../src/neon.cc:217:20: error: 'Contents' is not a member of 'v8::ArrayBuffer' 217 | v8::ArrayBuffer::Contents contents = buffer->GetContents(); | ^~~~~~~~ ../src/neon.cc:218:15: error: 'contents' was not declared in this scope 218 | *base_out = contents.Data(); | ^~~~~~~~ ../src/neon.cc: In function 'void Neon_Class_SetClassMap(v8::Isolate*, void*, Neon_DropCallback)': ../src/neon.cc:329:41: warning: 'void node::AtExit(void (*)(void*), void*)' is deprecated: Use the three-argument variant of AtExit() or AddEnvironmentCleanupHook() [-Wdeprecated-declarations] 329 | node::AtExit(cleanup_class_map, holder); | ^ In file included from ../../../../../../../../node_modules/nan/nan.h:56, from ../src/neon.cc:2: /build/source/seshat-node/.electron-gyp/13.0.1/include/node/node.h:866:22: note: declared here 866 | NODE_EXTERN void AtExit(void (*cb)(void* arg), void* arg = nullptr)); | ^~~~~~ /build/source/seshat-node/.electron-gyp/13.0.1/include/node/node.h:108:42: note: in definition of macro 'NODE_DEPRECATED' 108 | __attribute__((deprecated(message))) declarator | ^~~~~~~~~~ In file included from ../src/neon.cc:8: ../src/neon_string.h: In member function 'v8::Local neon::Slice::ToJsString(v8::Isolate*, const char*)': ../src/neon_string.h:28:18: warning: ignoring return value of 'bool v8::MaybeLocal::ToLocal(v8::Local*) const [with S = v8::String; T = v8::String]' declared with attribute 'warn_unused_result' [-Wunused-result] 28 | maybe.ToLocal(&result); | ~~~~~~~~~~~~~^~~~~~~~~ make: *** [neon.target.mk:113: Release/obj.target/neon/src/neon.o] Error 1 gyp ERR! build error gyp ERR! stack Error: `make` failed with exit code: 2 gyp ERR! stack at ChildProcess.onExit (/nix/store/009815w1n26nl10rgffgahk7aka80p1m-nodejs-14.17.0/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:194:23) gyp ERR! stack at ChildProcess.emit (events.js:376:20) gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12) gyp ERR! System Linux 5.11.21 gyp ERR! command "/nix/store/009815w1n26nl10rgffgahk7aka80p1m-nodejs-14.17.0/bin/node" "/nix/store/009815w1n26nl10rgffgahk7aka80p1m-nodejs-14.17.0/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "build" gyp ERR! cwd /build/source/seshat-node/native/target/release/build/neon-sys-f7ab2e53f2e370fb/out/native gyp ERR! node -v v14.17.0 gyp ERR! node-gyp -v v5.1.0 gyp ERR! not ok npm ERR! code ELIFECYCLE npm ERR! errno 1 npm ERR! @ build-release: `node-gyp build` npm ERR! Exit status 1 npm ERR! npm ERR! Failed at the @ build-release script. npm ERR! This is probably not a problem with npm. There is likely additional logging output above. npm ERR! A complete log of this run can be found in: npm ERR! /tmp/.npm/_logs/2021-05-29T21_41_18_532Z-debug.log error occurred: Command "ar" "cq" "/build/source/seshat-node/native/target/release/build/neon-sys-f7ab2e53f2e370fb/out/libneon.a" "/build/source/seshat-node/native/target/release/build/neon-sys-f7ab2e53f2e370fb/out/native/build/Release/obj.target/neon/src/neon.o" with args "ar" did not execute successfully (status code exit code: 1). warning: build failed, waiting for other jobs to finish... The following warnings were emitted during compilation: warning: ar: /build/source/seshat-node/native/target/release/build/neon-sys-400bc8aadcaf0046/out/native/build/Release/obj.target/neon/src/neon.o: No such file or directory error: build failed neon ERR! cargo build failed Error: cargo build failed at Target. (/build/source/seshat-node/node_modules/neon-cli/lib/target.js:121:35) at step (/build/source/seshat-node/node_modules/neon-cli/lib/target.js:32:23) at Object.next (/build/source/seshat-node/node_modules/neon-cli/lib/target.js:13:53) at fulfilled (/build/source/seshat-node/node_modules/neon-cli/lib/target.js:4:58) at processTicksAndRejections (internal/process/task_queues.js:95:5) builder for '/nix/store/vs8qn1fzq0cbh5830pci0mxr8fiqdx89-seshat-node-2.2.4.drv' failed with exit code 1 error: build of '/nix/store/vs8qn1fzq0cbh5830pci0mxr8fiqdx89-seshat-node-2.2.4.drv' on 'ssh://pbb@mozarella.petabyte.dev' failed: builder for '/nix/store/vs8qn1fzq0cbh5830pci0mxr8fiqdx89-seshat-node-2.2.4.drv' failed with exit code 1 builder for '/nix/store/vs8qn1fzq0cbh5830pci0mxr8fiqdx89-seshat-node-2.2.4.drv' failed with exit code 1 error: build of '/nix/store/vs8qn1fzq0cbh5830pci0mxr8fiqdx89-seshat-node-2.2.4.drv' failed ```

\ Honestly it's just a horrible build system. hak and neon and so on don't give any consideration to users who want to build the libraries in an environment without an internet connection. \

Is there anything we could do to help fix building element-desktop with encrypted chat search working on NixOS?

@teutat3s If you can find a fix for the error, there shouldn't be much work left.

[ghost]

I think I found the solution to that error: Upstream is using Electron 12.x while we are using Electron 13.x.

[ghost]

PR incoming :sparkles: