Build Signal from source (and enable arm64 builds)

[jnsgruk]

This PR switches to enable build from source for Signal Desktop - the primary motivation is to enable builds for arm64. At present Signal does not provide an upstream deb for other architectures, so this is the only viable option I see.

There is a small difficulty which I've worked around, which is that the upstream package.json uses npm-run-all to run multiple targets as part of yarn generate. This was causing the scriptlets to exit early, so I've broken apart those targets into their component parts - but otherwise the process is as per their documentation.

Fixes #178

[jnsgruk]

@bceverly this is a start - I'm getting an odd error about not being able to prime the opt directory which I can't figure out, but I need to go make some dinner! Will play with this over the coming days!

[jnsgruk]

Error currently:

Executing parts lifecycle: stage cleanup                                                                                                                   
Executing action                                                                                                                                           
Executed: stage cleanup                                                                                                                                    
Executing parts lifecycle: stage desktop-file                                                                                                              
Executing action                                                                                                                                           
Executed: stage desktop-file                                                                                                                               
Executing parts lifecycle: stage gnome/sdk                                                                                                                 
Executing action                                                                                                                                           
Executed: stage gnome/sdk                                                                                                                                  
Executing parts lifecycle: prime signal-desktop                                                                                                            
Executing action                                                                                                                                           
Failed to copy '/root/stage/opt': no such file or directory.                                                                                               
Failed to execute pack in instance.                                                                                                                        
Full execution log: '/home/jon/.local/state/snapcraft/log/snapcraft-20231027-180255.545188.log'    

[jnsgruk]

@bceverly @kenvandine could you folks give this a try? I think I've resolved the build issues and it's getting close.

[lucyllewy]

Unfortunately I believe we're not allowed to build from source per Signal's intervention a couple of years ago. I might be misremembering but I think I recall that Canonical had to remove the Signal snap from the store while some legal work was done behind closed doors, which included committing to using the Signal client as built upstream, as-is.

[lucyllewy]

I might be misremembering tho. I found this about the problem back last year: https://forum.snapcraft.io/t/what-happened-to-signal-desktop/32119/18

[lucyllewy]

Tagging @merlijn-sebrechts in case he can provide more memory of what the situation was and the resolution.

[jnsgruk]

Thanks for the heads up @lucyllewy - will wait for others to respond.

Just noticed this build has failed - was working fine last night! I'll take a look when I get a chance later today or tomorrow.

[merlijn-sebrechts]

I have received no additional info about last year's issue, apart from "whoops, sorry for the DMCA takedown". I don't know why it initially happened, and I don't know why they ultimately accepted the Snap package.

Their policy seems to be "no third party packages because users can't trust them".

• Given that everything is open source, publicly built, and publicly logged, it should be clear that we're just compiling the signal desktop app instead of adding modifications.
• Given the arch package is also building from source, is known by the Signal team, and hasn't been taken down (yet), they seem to have relaxed their stance somewhat.

I'd say to go ahead with this PR. Worst case, we just revert. Once Signal provides upstream arm64 packages, I'd like to switch back to theirs, though.

[jnsgruk]

Thanks @merlijn-sebrechts

One other thing I'm not quite sure about right now, is that this builds fine for me locally, but if I use `snapcraft remote-build' to use the Launchpad builders, I get a strange error in the install process:

:: [5/5] Building fresh packages...
:: error /build/snapcraft-signal-desktop-776fe9bc50bf07baece39c582bdcaff5/parts/signal-desktop/build/node_modules/@signalapp/ringrtc: Command failed.
:: Exit code: 1
:: Command: node scripts/fetch-prebuild.js
:: Arguments:
:: Directory: /build/snapcraft-signal-desktop-776fe9bc50bf07baece39c582bdcaff5/parts/signal-desktop/build/node_modules/@signalapp/ringrtc
:: Output:
:: downloading https://build-artifacts.signal.org/libraries/ringrtc-desktop-build-v2.33.0.tar.gz
:: node:events:491
::       throw er; // Unhandled 'error' event
::       ^
::
:: Error: getaddrinfo ENOTFOUND build-artifacts.signal.org
::     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:107:26)
:: Emitted 'error' event on ClientRequest instance at:
[30/Oct/2023:09:53:29 +0000] "CONNECT nodejs.org:443 HTTP/1.1" 200 8587995 "-" "-"
[30/Oct/2023:09:53:29 +0000] "CONNECT nodejs.org:443 HTTP/1.1" 200 8587959 "-" "-"
[30/Oct/2023:09:53:30 +0000] "CONNECT nodejs.org:443 HTTP/1.1" 200 8123 "-" "-"
[30/Oct/2023:09:53:31 +0000] "CONNECT nodejs.org:443 HTTP/1.1" 200 8123 "-" "-"
[30/Oct/2023:09:53:31 +0000] "CONNECT nodejs.org:443 HTTP/1.1" 200 8587962 "-" "-"
[30/Oct/2023:09:53:32 +0000] "CONNECT nodejs.org:443 HTTP/1.1" 200 8122 "-" "-"
[30/Oct/2023:09:53:32 +0000] "CONNECT github.com:443 HTTP/1.1" 200 3773 "-" "-"
[30/Oct/2023:09:53:44 +0000] "CONNECT playwright.azureedge.net:443 HTTP/1.1" 200 157646321 "-" "-"
[30/Oct/2023:09:53:48 +0000] "CONNECT playwright.azureedge.net:443 HTTP/1.1" 200 1693958 "-" "-"
[30/Oct/2023:09:54:02 +0000] "CONNECT playwright.azureedge.net:443 HTTP/1.1" 200 76834043 "-" "-"
[30/Oct/2023:09:54:16 +0000] "CONNECT playwright.azureedge.net:443 HTTP/1.1" 200 75487844 "-" "-"
::     at TLSSocket.socketErrorListener (node:_http_client:502:9)
::     at TLSSocket.emit (node:events:513:28)
::     at emitErrorNT (node:internal/streams/destroy:151:8)
::     at emitErrorCloseNT (node:internal/streams/destroy:116:3)
::     at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
::   errno: -3008,
::   code: 'ENOTFOUND',
::   syscall: 'getaddrinfo',
::   hostname: 'build-artifacts.signal.org'
:: }

I'm not sure why the LP builder can't see that, but I'll do some digging and update here.

[lucyllewy]

:: error /build/snapcraft-signal-desktop-776fe9bc50bf07baece39c582bdcaff5/parts/signal-desktop/build/node_modules/@signalapp/ringrtc: Command failed. :: Exit code: 1 :: Command: node scripts/fetch-prebuild.js :: Arguments: :: Directory: /build/snapcraft-signal-desktop-776fe9bc50bf07baece39c582bdcaff5/parts/signal-desktop/build/node_modules/@signalapp/ringrtc :: Output: :: downloading https://build-artifacts.signal.org/libraries/ringrtc-desktop-build-v2.33.0.tar.gz :: node:events:491 :: throw er; // Unhandled 'error' event :: ^ :: :: Error: getaddrinfo ENOTFOUND build-artifacts.signal.org :: at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:107:26) :: Emitted 'error' event on ClientRequest instance at:

I suspect this is an issue with the http_proxy and https_proxy environment variables being killed somewhere within the scripts from upstream during the build when calling that specific subproccess.

[jnsgruk]

Yep, you're absolutely right -- there is an attempt to fix it here but no joy as yet. The offending file is obvious, but I can't seem to stop it from trying to fetch without a proxy, even if I pre-fetch the file. Will keep digging.

[jnsgruk]

Okay! I've now managed to work through the build issues on Launchpad, and have successfully launched the snap, built on Launchpad, on an amd64 machine.

The process is quite messy; there are some dependencies in the yarn install process that won't respect the proxy without patching. I've done this as part of the build, and it doesn't feel great, but it should also be more robust than trying to carry patches against the files.

@bceverly could you give this a test?

/cc @kenvandine - as promised :wink:

[tim-hm]

Many thanks for the effort on this @jnsgruk :pray:

I've briefly tested on ubuntu-asahi (arm64) and on an arm64 vm on parallels and can confirm this is working :pray: Are there any specific features to check?

[bceverly]

Thank you @jnsgruk!  It's working great for me on the x13s under Mantic!!! -- Thanks, Bryan

On Tue, 2023-10-31 at 10:13 -0700, Tim Holmes-Mitra wrote:

Many thanks for the effort on this @jnsgruk 🙏 I've briefly tested on ubuntu-asahi (arm64) and on an arm64 vm on parallels and can confirm this is working 🙏 Are there any specific features to check? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[jnsgruk]

Thanks! Yeh that's a good point; I've opened #181 which builds on this PR, and augments the CI to handle this automatically.

[popey]

Is the arm64 supported upstream? If not, are we committing to supporting this build? Bear in mind the relatively limited availability of Linux on (decent) arm64 devices will make supporting arm64 specific issues harder.

[popey]

Just chatted about this with Joe and Im going to +1 this change. Merge at your discretion.

[jnsgruk]

Thanks, Alan! I've also submitted a PR to the upstream repos of those node deps that need patching - hopefully we can remove that piece of complexity from the build in the future.

[merlijn-sebrechts]

Looks good to me. I can merge it now, but know that this won't actually change anything in the store. The signal-desktop release workflow is completely automated from GitHub using GitHub actions. We need to build the arm version here too.

So the next steps are to update the actions use remote-build as explained here: https://github.com/snapcrafters/gimp/issues/244

I can provide the credentials so GH actions can do this. Do you want to do that in this PR or in a next one?

[jnsgruk]

Yep - I can take that on. Let's merge this and I'll do that in another PR.

[jnsgruk]

@merlijn-sebrechts added in #181 - which will be easier to review once this merges!