new-install does not seem to handle headers of dependencies appropriately

[vmchale]

When buliding plutus, I can run cabal new-build exe:plc without any problems. However, cabal new-install exe:plc fails with

Wrote tarball sdist to
/home/vanessa/work/haskell/plutus/dist-newstyle/sdist/language-plutus-core-0.1.0.0.tar.gz
Wrote tarball sdist to
/home/vanessa/work/haskell/plutus/dist-newstyle/sdist/marlowe-0.1.0.0.tar.gz
Wrote tarball sdist to
/home/vanessa/work/haskell/plutus/dist-newstyle/sdist/plutus-core-interpreter-0.1.0.0.tar.gz
Wrote tarball sdist to
/home/vanessa/work/haskell/plutus/dist-newstyle/sdist/plutus-exe-0.1.0.0.tar.gz
Wrote tarball sdist to
/home/vanessa/work/haskell/plutus/dist-newstyle/sdist/wallet-api-0.1.0.0.tar.gz
Wrote tarball sdist to
/home/vanessa/work/haskell/plutus/dist-newstyle/sdist/plutus-tx-0.1.0.0.tar.gz
Wrote tarball sdist to
/home/vanessa/work/haskell/plutus/dist-newstyle/sdist/plutus-use-cases-0.1.0.0.tar.gz
Wrote tarball sdist to
/home/vanessa/work/haskell/plutus/dist-newstyle/sdist/plutus-ir-0.1.0.0.tar.gz
Wrote tarball sdist to
/home/vanessa/work/haskell/plutus/dist-newstyle/sdist/plutus-playground-server-0.1.0.0.tar.gz
Wrote tarball sdist to
/home/vanessa/work/haskell/plutus/dist-newstyle/sdist/plutus-playground-lib-0.1.0.0.tar.gz
Wrote tarball sdist to
/home/vanessa/work/haskell/plutus/dist-newstyle/sdist/plutus-tutorial-0.1.0.0.tar.gz
Wrote tarball sdist to
/home/vanessa/work/haskell/plutus/dist-newstyle/sdist/cardano-crypto-1.1.0.tar.gz
Wrote tarball sdist to
/home/vanessa/work/haskell/plutus/dist-newstyle/sdist/servant-purescript-0.9.0.2.tar.gz
Resolving dependencies...
Build profile: -w ghc-8.4.4 -O2
In order, the following will be built (use -v for more details):
 - cardano-crypto-1.1.0 (lib) (requires build)
 - language-plutus-core-0.1.0.0 (lib) (requires build)
 - plutus-core-interpreter-0.1.0.0 (lib) (requires build)
 - plutus-exe-0.1.0.0 (exe:plc) (requires build)
Starting     cardano-crypto-1.1.0 (lib)
Building     cardano-crypto-1.1.0 (lib)
Warning: Some package(s) failed to build. Try rerunning with -j1 if you can't
see the error.

Failed to build cardano-crypto-1.1.0.
Build log (
/home/vanessa/.cabal/logs/ghc-8.4.4/cardano-crypto-1.1.0-cb01cf7f6520ff191ac855c4e142dc93c01f650c0d3916062cdb2b8dba29f7cd.log
):
Configuring library for cardano-crypto-1.1.0..
Preprocessing library for cardano-crypto-1.1.0..
Building library for cardano-crypto-1.1.0..
[ 1 of 19] Compiling Cardano.Crypto.Wallet.Types ( src/Cardano/Crypto/Wallet/Types.hs, dist/build/Cardano/Crypto/Wallet/Types.o )
[ 2 of 19] Compiling Cardano.Crypto.Wallet.Encrypted ( src/Cardano/Crypto/Wallet/Encrypted.hs, dist/build/Cardano/Crypto/Wallet/Encrypted.o )
[ 3 of 19] Compiling Cardano.Internal.Compat ( src/Cardano/Internal/Compat.hs, dist/build/Cardano/Internal/Compat.o )
[ 4 of 19] Compiling Crypto.ECC.Ed25519Donna ( src/Crypto/ECC/Ed25519Donna.hs, dist/build/Crypto/ECC/Ed25519Donna.o )
[ 5 of 19] Compiling Crypto.ECC.P256  ( src/Crypto/ECC/P256.hs, dist/build/Crypto/ECC/P256.o )
[ 6 of 19] Compiling Crypto.DLEQ      ( src/Crypto/DLEQ.hs, dist/build/Crypto/DLEQ.o )
[ 7 of 19] Compiling Cardano.Crypto.Praos.VRF ( src/Cardano/Crypto/Praos/VRF.hs, dist/build/Cardano/Crypto/Praos/VRF.o )
[ 8 of 19] Compiling Crypto.Encoding.BIP39.Dictionary ( src/Crypto/Encoding/BIP39/Dictionary.hs, dist/build/Crypto/Encoding/BIP39/Dictionary.o )
[ 9 of 19] Compiling Crypto.Encoding.BIP39 ( src/Crypto/Encoding/BIP39.hs, dist/build/Crypto/Encoding/BIP39.o )
[10 of 19] Compiling Cardano.Crypto.Encoding.Seed ( src/Cardano/Crypto/Encoding/Seed.hs, dist/build/Cardano/Crypto/Encoding/Seed.o )
[11 of 19] Compiling Cardano.Crypto.Encoding.BIP39 ( src/Cardano/Crypto/Encoding/BIP39.hs, dist/build/Cardano/Crypto/Encoding/BIP39.o )
[12 of 19] Compiling Crypto.Encoding.BIP39.English ( src/Crypto/Encoding/BIP39/English.hs, dist/build/Crypto/Encoding/BIP39/English.o )
[13 of 19] Compiling Crypto.Math.Bits ( src/Crypto/Math/Bits.hs, dist/build/Crypto/Math/Bits.o )
[14 of 19] Compiling Crypto.Math.NatMath ( src/Crypto/Math/NatMath.hs, dist/build/Crypto/Math/NatMath.o )
[15 of 19] Compiling Crypto.Math.Bytes ( src/Crypto/Math/Bytes.hs, dist/build/Crypto/Math/Bytes.o )
[16 of 19] Compiling Crypto.Math.Edwards25519 ( src/Crypto/Math/Edwards25519.hs, dist/build/Crypto/Math/Edwards25519.o )
[17 of 19] Compiling Cardano.Crypto.Wallet.Pure ( src/Cardano/Crypto/Wallet/Pure.hs, dist/build/Cardano/Crypto/Wallet/Pure.o )
[18 of 19] Compiling Cardano.Crypto.Wallet ( src/Cardano/Crypto/Wallet.hs, dist/build/Cardano/Crypto/Wallet.o )
[19 of 19] Compiling Crypto.ECC.Ed25519BIP32 ( src/Crypto/ECC/Ed25519BIP32.hs, dist/build/Crypto/ECC/Ed25519BIP32.o )

cbits/ed25519/ed25519.c:10:10: error:
     fatal error: ed25519-donna.h: No such file or directory
     #include "ed25519-donna.h"
              ^~~~~~~~~~~~~~~~~
   |
10 | #include "ed25519-donna.h"
   |          ^
compilation terminated.
`gcc' failed in phase `C Compiler'. (Exit code: 1)
cabal: Failed to build cardano-crypto-1.1.0 (which is required by exe:plc from
plutus-exe-0.1.0.0). See the build log above for details.

...which leads me to believe that headers are not being copied appropriately or perhaps include paths are not being set appropriately.

Thanks!

[typedrat]

Huh! That's not just a new-install problem, then. new-install works the same way that Hackage installations work (that's why it sdists, there's no difference between a Hackage package being built with the tarball cached and how new-install builds stuff, it's using the exact same codepath to do so), so the issue is more a combination of a bug with new-build that's making it not fail when it's supposed to (maybe a dupe of #5430?) and an issue with how you're using Cabal that's not putting the right stuff in the right places.

[vmchale]

It may also be related to the fact that cardano-crypto is a git-based dependency (in fact, I have not seen this problem without using git-based dependencies).

[typedrat]

Hmm, that might also be it! It is new-sdisting cardano-crypto, though. What's in that tarball?

[vmchale]

Ah that seems to be it! /home/vanessa/work/haskell/plutus/dist-newstyle/sdist/cardano-crypto-1.1.0.tar.gzdoes not contain ed25519-donna.h.

[typedrat]

Then it's an issue with the .cabal file for cardano-crypto.

[vmchale]

Ahh okay. Thanks!

[typedrat]

I think that it's deeper than that. Is it in the tarball you get when you new-sdist in a clone of cardano-crypto? The .cabal file looks right.

Could you paste the output of cabal new-sdist --list-only in the folder?

[vmchale]

The output of cabal new-sdist --list-only in cardano-crypto is

./LICENSE
./README.md
./Setup.hs
./benchs/Bench.hs
./cardano-crypto.cabal
./cbits/ed25519/ed25519.c
./cbits/encrypted_sign.c
./src/Cardano/Crypto/Encoding/BIP39.hs
./src/Cardano/Crypto/Encoding/Seed.hs
./src/Cardano/Crypto/Praos/VRF.hs
./src/Cardano/Crypto/Wallet.hs
./src/Cardano/Crypto/Wallet/Encrypted.hs
./src/Cardano/Crypto/Wallet/Pure.hs
./src/Cardano/Crypto/Wallet/Types.hs
./src/Cardano/Internal/Compat.hs
./src/Crypto/DLEQ.hs
./src/Crypto/ECC/Ed25519BIP32.hs
./src/Crypto/ECC/Ed25519Donna.hs
./src/Crypto/ECC/P256.hs
./src/Crypto/Encoding/BIP39.hs
./src/Crypto/Encoding/BIP39/Dictionary.hs
./src/Crypto/Encoding/BIP39/English.hs
./src/Crypto/Math/Bits.hs
./src/Crypto/Math/Bytes.hs
./src/Crypto/Math/Edwards25519.hs
./src/Crypto/Math/NatMath.hs
./test/GoldenTest.hs
./test/Spec.hs
./test/Test/Cardano.hs
./test/Test/Cardano/Crypto.hs
./test/Test/Cardano/Crypto/Encoding.hs
./test/Test/Cardano/Crypto/Encoding/Seed.hs
./test/Test/Cardano/Crypto/Praos.hs
./test/Test/Cardano/Crypto/Praos/VRF.hs
./test/Test/Crypto.hs
./test/Test/Crypto/Encoding.hs
./test/Test/Crypto/Encoding/BIP39.hs
./test/Test/Orphans.hs
./test/Utils.hs

[typedrat]

Ah! I see it. You've got to have the headers in extra-source-files.

[turbolent]

@typedrat I tried using extra-source-files with new-sdist using Cabal 2.4.1.0 and the tarball still doesn't include the header file, and --list-only doesn't include it either. Shouldn't listing it in install-includes be enough, i.e. shouldn't a file that will be installed in the system also be in the source tarball?