Dependency updates and some cleanups

infinisil commented 2 years ago

See commit messages for changes

infinisil commented 2 years ago

Two problems:

Darwin fails with

building '/nix/store/98ndy3affdhafiq9k9gj3xp44rfmzpyx-ghc-shell-for-packages-ghc-8.10.7-env.drv'...
/nix/store/dhhbwn09yhr7jqvwkvi0v4m62xzsvdi9-stdenv-darwin/setup: line 23: shopt: inherit_errexit: invalid shell option name

likely is related to https://github.com/NixOS/nixpkgs/pull/139508

Compilation speed slowed down considerably. It now takes about 6 minutes for a cabal build lib:webauthn, whereas previously it only took about 1 minute. Notably there's also some weird lines like this during compilation:
```
exprType TYPE: Text
[...]
exprType TYPE: DOMString
[...]
exprType TYPE: a_ah4f6
[...]
exprType TYPE: a_ah4iL
```
Full output is here:

``` $ time cabal build lib:webauthn Cloning into '/home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/src/hs-certif_-e9a65203a41c944e'... remote: Enumerating objects: 3512, done. remote: Counting objects: 100% (55/55), done. remote: Compressing objects: 100% (37/37), done. remote: Total 3512 (delta 16), reused 48 (delta 13), pack-reused 3457 Receiving objects: 100% (3512/3512), 675.18 KiB | 1.53 MiB/s, done. Resolving deltas: 100% (1802/1802), done. HEAD is now at 6651148 Don't use Monad for validatePure Warning: The package list for 'hackage.haskell.org' is 111 days old. Run 'cabal update' to get the latest list of available packages. Resolving dependencies... Build profile: -w ghc-8.10.7 -O1 In order, the following will be built (use -v for more details): - webauthn-0.1.0.0 (lib) (first run) Configuring library for webauthn-0.1.0.0.. Preprocessing library for webauthn-0.1.0.0.. Building library for webauthn-0.1.0.0.. [ 1 of 31] Compiling Crypto.WebAuthn.DateOrphans ( src/Crypto/WebAuthn/DateOrphans.hs, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/DateOrphans.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/DateOrphans.dyn_o ) [ 2 of 31] Compiling Crypto.WebAuthn.EncodingUtils ( src/Crypto/WebAuthn/EncodingUtils.hs, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/EncodingUtils.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/EncodingUtils.dyn_o ) [ 3 of 31] Compiling Crypto.WebAuthn.Model.Kinds ( src/Crypto/WebAuthn/Model/Kinds.hs, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Model/Kinds.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Model/Kinds.dyn_o ) [ 4 of 31] Compiling Crypto.WebAuthn.PublicKey ( src/Crypto/WebAuthn/PublicKey.hs, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/PublicKey.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/PublicKey.dyn_o ) [ 5 of 31] Compiling Crypto.WebAuthn.Registry ( src/Crypto/WebAuthn/Registry.hs, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Registry.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Registry.dyn_o ) [ 6 of 31] Compiling Crypto.WebAuthn.SubjectKeyIdentifier ( src/Crypto/WebAuthn/SubjectKeyIdentifier.hs, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/SubjectKeyIdentifier.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/SubjectKeyIdentifier.dyn_o ) [ 7 of 31] Compiling Crypto.WebAuthn.Model ( src/Crypto/WebAuthn/Model.hs, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Model.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Model.dyn_o ) [ 8 of 31] Compiling Crypto.WebAuthn.Operations.Common ( src/Crypto/WebAuthn/Operations/Common.hs, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Operations/Common.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Operations/Common.dyn_o ) [ 9 of 31] Compiling Crypto.WebAuthn.Operations.Attestation.TPM ( src/Crypto/WebAuthn/Operations/Attestation/TPM.hs, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Operations/Attestation/TPM.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Operations/Attestation/TPM.dyn_o ) exprType TYPE: Text exprType TYPE: Text [10 of 31] Compiling Crypto.WebAuthn.Operations.Attestation.Packed ( src/Crypto/WebAuthn/Operations/Attestation/Packed.hs, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Operations/Attestation/Packed.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Operations/Attestation/Packed.dyn_o ) [11 of 31] Compiling Crypto.WebAuthn.Operations.Attestation.None ( src/Crypto/WebAuthn/Operations/Attestation/None.hs, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Operations/Attestation/None.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Operations/Attestation/None.dyn_o ) [12 of 31] Compiling Crypto.WebAuthn.Operations.Attestation.FidoU2F ( src/Crypto/WebAuthn/Operations/Attestation/FidoU2F.hs, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Operations/Attestation/FidoU2F.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Operations/Attestation/FidoU2F.dyn_o ) [13 of 31] Compiling Crypto.WebAuthn.Operations.Attestation.Apple ( src/Crypto/WebAuthn/Operations/Attestation/Apple.hs, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Operations/Attestation/Apple.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Operations/Attestation/Apple.dyn_o ) [14 of 31] Compiling Crypto.WebAuthn.Operations.Attestation.AndroidKey ( src/Crypto/WebAuthn/Operations/Attestation/AndroidKey.hs, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Operations/Attestation/AndroidKey.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Operations/Attestation/AndroidKey.dyn_o ) [15 of 31] Compiling Crypto.WebAuthn.Operations.Assertion ( src/Crypto/WebAuthn/Operations/Assertion.hs, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Operations/Assertion.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Operations/Assertion.dyn_o ) [16 of 31] Compiling Crypto.WebAuthn.Model.Binary.Encoding ( src/Crypto/WebAuthn/Model/Binary/Encoding.hs, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Model/Binary/Encoding.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Model/Binary/Encoding.dyn_o ) [17 of 31] Compiling Crypto.WebAuthn.WebIDL ( src/Crypto/WebAuthn/WebIDL.hs, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/WebIDL.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/WebIDL.dyn_o ) [18 of 31] Compiling Crypto.WebAuthn.UAF ( src/Crypto/WebAuthn/UAF.hs, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/UAF.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/UAF.dyn_o ) [19 of 31] Compiling Crypto.WebAuthn.Model.JavaScript ( src/Crypto/WebAuthn/Model/JavaScript.hs, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Model/JavaScript.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Model/JavaScript.dyn_o ) [20 of 31] Compiling Crypto.WebAuthn.Model.JavaScript.Types ( src/Crypto/WebAuthn/Model/JavaScript/Types.hs, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Model/JavaScript/Types.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Model/JavaScript/Types.dyn_o ) [21 of 31] Compiling Crypto.WebAuthn.Model.JavaScript.Encoding ( src/Crypto/WebAuthn/Model/JavaScript/Encoding.hs, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Model/JavaScript/Encoding.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Model/JavaScript/Encoding.dyn_o ) [22 of 31] Compiling Crypto.WebAuthn.Model.Binary.Decoding ( src/Crypto/WebAuthn/Model/Binary/Decoding.hs, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Model/Binary/Decoding.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Model/Binary/Decoding.dyn_o ) [23 of 31] Compiling Crypto.WebAuthn.Model.JavaScript.Decoding ( src/Crypto/WebAuthn/Model/JavaScript/Decoding.hs, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Model/JavaScript/Decoding.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Model/JavaScript/Decoding.dyn_o ) exprType TYPE: DOMString exprType TYPE: DOMString exprType TYPE: DOMString exprType TYPE: DOMString exprType TYPE: DOMString exprType TYPE: DOMString exprType TYPE: DOMString exprType TYPE: DOMString exprType TYPE: DOMString exprType TYPE: DOMString exprType TYPE: a_ah4f6 exprType TYPE: a_ah4f6 exprType TYPE: a_ah4f6 exprType TYPE: a_ah4f6 exprType TYPE: DOMString exprType TYPE: DOMString exprType TYPE: DOMString exprType TYPE: DOMString exprType TYPE: DOMString exprType TYPE: DOMString exprType TYPE: DOMString exprType TYPE: DOMString exprType TYPE: DOMString exprType TYPE: DOMString exprType TYPE: DOMString exprType TYPE: DOMString exprType TYPE: DOMString exprType TYPE: DOMString exprType TYPE: a_ah4iL exprType TYPE: a_ah4iL exprType TYPE: a_ah4iL exprType TYPE: a_ah4iL exprType TYPE: a_ah4iL exprType TYPE: a_ah4iL exprType TYPE: a_ah4iL exprType TYPE: a_ah4iL [24 of 31] Compiling Crypto.WebAuthn.Metadata.Statement.IDL ( src/Crypto/WebAuthn/Metadata/Statement/IDL.hs, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Metadata/Statement/IDL.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Metadata/Statement/IDL.dyn_o ) [25 of 31] Compiling Crypto.WebAuthn.Metadata.Statement.Types ( src/Crypto/WebAuthn/Metadata/Statement/Types.hs, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Metadata/Statement/Types.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Metadata/Statement/Types.dyn_o ) [26 of 31] Compiling Crypto.WebAuthn.Metadata.Statement.Decode ( src/Crypto/WebAuthn/Metadata/Statement/Decode.hs, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Metadata/Statement/Decode.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Metadata/Stat ement/Decode.dyn_o ) [27 of 31] Compiling Crypto.WebAuthn.Metadata.Service.IDL ( src/Crypto/WebAuthn/Metadata/Service/IDL.hs, /home/infinisil/j/tweag/clients/merc ury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Metadata/Service/IDL.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Metadata/Service/IDL.dyn_o ) [28 of 31] Compiling Crypto.WebAuthn.Metadata.Service.Types ( src/Crypto/WebAuthn/Metadata/Service/Types.hs, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Metadata/Service/Types.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Metadata/Service/Types.dyn_o ) [29 of 31] Compiling Crypto.WebAuthn.Metadata.Service.Decode ( src/Crypto/WebAuthn/Metadata/Service/Decode.hs, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Metadata/Service/Decode.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Metadata/Service/Decode.dyn_o ) [30 of 31] Compiling Crypto.WebAuthn.Metadata.Service.Processing ( src/Crypto/WebAuthn/Metadata/Service/Processing.hs, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Metadata/Service/Processing.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Metadata/Service/Processing.dyn_o ) [31 of 31] Compiling Crypto.WebAuthn.Operations.Attestation ( src/Crypto/WebAuthn/Operations/Attestation.hs, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Operations/Attestation.o, /home/infinisil/j/tweag/clients/mercury/repo/main/dist-newstyle/build/x86_64-linux/ghc-8.10.7/webauthn-0.1.0.0/build/Crypto/WebAuthn/Operations/Attestation.dyn_o ) cabal build lib:webauthn 372.82s user 3.65s system 99% cpu 6:17.31 total ```

infinisil commented 2 years ago

Looking at compilation times of individual modules, all modules compile within seconds, except these:

Crypto.WebAuthn.Registry: 1:05
Crypto.WebAuthn.Model.JavaScript: 1:03
Crypto.WebAuthn.Metadata.Statement.IDL: 2:03
Crypto.WebAuthn.Metadata.Service.IDL: 1:07

And the common thing among these modules is that they use deriving-aeson to derive JSON instances for a lot of data types. Safe to say, that package is causing the slowdown with this PR.

I'll continue investigating.

infinisil commented 2 years ago

Here's the full log of a compilation with -v and timestamps: log.txt

infinisil commented 2 years ago

The performance problem doesn't appear to be related to haskell.nix, because it's the same even when I tried to switch to the nixpkgs Haskell infrastructure, see https://github.com/tweag/webauthn/tree/nixpkgs-haskell

infinisil commented 2 years ago

Turns out that the problem only appears with aeson >= 2.0. I'll try to create a small reproducible example to report to probably https://hackage.haskell.org/package/deriving-aeson

infinisil commented 2 years ago

I reported the performance problem in https://github.com/fumieval/deriving-aeson/issues/16, having created a reproducible minimal example in https://github.com/tweag/webauthn/tree/deriving-aeson-slow-repro

infinisil commented 2 years ago

I fixed the bash problem on macOS by just not using nix-shell anymore. But now we're running into a segfault on darwin:

/Users/runner/work/_temp/2190ff6f-868b-4043-9299-11dfd6567307.sh: line 1: 18790 Segmentation fault: 11  nix-build -A webauthn.checks

This seems to be the same as issue https://github.com/NixOS/nix/issues/5125

infinisil commented 2 years ago

Since this seems to be related to import-from-derivation, and I already did the work of getting the nixpkgs haskell infra to work in https://github.com/tweag/webauthn/tree/nixpkgs-haskell, I'm just gonna switch to that. This will also speed up CI and nix-shells

tweag / webauthn

Dependency updates and some cleanups #76