awakesecurity / gRPC-haskell

Haskell gRPC support
https://hackage.haskell.org/package/grpc-haskell
Apache License 2.0
236 stars 74 forks source link

Update/fix primary `nix-shell` entry #129

Closed intractable closed 3 years ago

intractable commented 3 years ago

Prior to this PR, nix-shell was entering via the grpc-haskell.env attribute, and the environment it provided was not sufficient to build both grpc-haskell-core and grpc-haskell (with tests enabled) conveniently.

After this PR, the default entry point is the test-grpc-haskell attribute, which has been extended (albeit in somewhat of an adhoc fashion; feedback welcome and appreciated) to support a working development/testing workflow.

I've also made some miscellaneous cleanups, fixed warnings, etc. Commits are mostly atomic and can be reviewed in order.

intractable commented 3 years ago

Could you explain more detail why grpc-haskell.env doesn't work for a test shell?

This was the problem that led to attempting to fiddle with the additional cabal configure parameters:

$ nix-shell release.nix --attr grpc-haskell.env --command 'cabal configure --enable-tests && cabal build'
Removing cabal.project.local
Removing dist-newstyle/
Resolving dependencies...
Build profile: -w ghc-8.10.3 -O1
In order, the following would be built (use -v for more details):
 - grpc-haskell-0.1.0 (test:tests) (first run)
 - grpc-haskell-core-0.1.0 (lib) (first run)
 - grpc-haskell-core-0.1.0 (test:tests) (first run)
 - grpc-haskell-0.1.0 (lib) (first run)
Build profile: -w ghc-8.10.3 -O1
In order, the following will be built (use -v for more details):
 - grpc-haskell-0.1.0 (test:tests) (first run)
 - grpc-haskell-core-0.1.0 (lib) (first run)
 - grpc-haskell-0.1.0 (lib) (first run)
Configuring library for grpc-haskell-core-0.1.0..
Configuring test suite 'tests' for grpc-haskell-0.1.0..
Preprocessing test suite 'tests' for grpc-haskell-0.1.0..
Building test suite 'tests' for grpc-haskell-0.1.0..
cabal: Missing dependencies on foreign libraries:
* Missing (or bad) header file: include/grpc_haskell.h
* Missing (or bad) C libraries: grpc, gpr
This problem can usually be solved by installing the system packages that
provide these libraries (you may need the "-dev" versions). If the libraries
are already installed but in a non-standard location then you can use the
flags --extra-include-dirs= and --extra-lib-dirs= to specify where they are.If
the library files do exist, it may contain errors that are caught by the C
compiler at the preprocessing stage. In this case you can re-run configure
with the verbosity flag -v3 to see the error messages.
If the header file does exist, it may contain errors that are caught by the C
compiler at the preprocessing stage. In this case you can re-run configure
with the verbosity flag -v3 to see the error messages.

[1 of 2] Compiling GeneratedTests   ( tests/GeneratedTests.hs, /home/joel/w/grpc-haskell/dist-newstyle/build/x86_64-linux/ghc-8.10.3/grpc-haskell-0.1.0/t/tests/
build/tests/tests-tmp/GeneratedTests.o )
[2 of 2] Compiling Main             ( tests/Properties.hs, /home/joel/w/grpc-haskell/dist-newstyle/build/x86_64-linux/ghc-8.10.3/grpc-haskell-0.1.0/t/tests/buil
d/tests/tests-tmp/Main.o )
Linking /home/joel/w/grpc-haskell/dist-newstyle/build/x86_64-linux/ghc-8.10.3/grpc-haskell-0.1.0/t/tests/build/tests/tests ...
cabal: Failed to build grpc-haskell-core-0.1.0 (which is required by
grpc-haskell-0.1.0). The failure occurred during the configure step.

(-v3 does indeed show missing grpc includes, but it's reported as an error for include/grpc_haskell.h)

That said, it looks like some of the other changes (possibly c2hs, but I'll know more soon) I made along the way made the issue go away, so I'll clean this up a bit and see if I can get it to a minimal working state. Thanks!

intractable commented 3 years ago

That said, it looks like some of the other changes (possibly c2hs, but I'll know more soon) I made along the way made the issue go away, so I'll clean this up a bit and see if I can get it to a minimal working state. Thanks!

The above build problem seems to go away after the removal of cabal-install from grpc-haskell's buildDepends (which then exposes my local cabal-install-3.4.0.0 instead of 3.2.0.0 from nixpkgs).

I suppose I can see about pinning a newer cabal-install so that the nix-shell environment can be self-contained.

intractable commented 3 years ago

@evanrelf Yep; thanks. No luck yet with my initial attempts to pin a newer cabal-install, at least without running on the dependency treadmill more than I'd prefer. I'm going to take a shot at bumping the nixpkgs pin and see how that goes.

intractable commented 3 years ago

@Gabriel439 @evanrelf Alright, I danced around the problem with cabal-3.2.0.0 by explicitly pinning a newer version for use in the grpc-haskell.env nix-shell.

@Gabriel439 I'll take a quick stab at the pkgs.fixDarwinDylibNames solution you suggested, but probably not much more than a cursory attempt if it doesn't work out of the box.

intractable commented 3 years ago

On Darwin, the tests don't run on master in the nix-shell environment (and seem to have the same behavior on this branch, with or without my changes at the branch HEAD or the fixDarwinDylibNames attempt mentioned below):

[nix-shell]$ cabal test 
Build profile: -w ghc-8.10.3 -O1
In order, the following will be built (use -v for more details):
 - grpc-haskell-0.1.0 (test:tests) (ephemeral targets)
Preprocessing test suite 'tests' for grpc-haskell-0.1.0..
Building test suite 'tests' for grpc-haskell-0.1.0..
Running 1 test suites...
Test suite tests: RUNNING...
GRPC Unit Tests
  Code generator tests
    server generation: /nix/store/lnvzlhdksnq10g97j86k1ngb94k1mky0-python3.8-setuptools-47.3.1/lib/python3.8/site-packages/pkg_resources/py2_warn.py:15: UserWarning: Setuptools no longer works on Python 2
************************************************************
Encountered a version of Setuptools that no longer supports
this version of Python. Please head to
https://bit.ly/setuptools-py2-sunset for support.
************************************************************
  warnings.warn(pre + "*" * 60 + msg + "*" * 60)
FAIL (3.32s)
      tests/GeneratedTests.hs:36:
      expected: ExitSuccess
       but got: ExitFailure 32
    client generation: /nix/store/lnvzlhdksnq10g97j86k1ngb94k1mky0-python3.8-setuptools-47.3.1/lib/python3.8/site-packages/pkg_resources/py2_warn.py:15: UserWarning: Setuptools no longer works on Python 2
************************************************************
Encountered a version of Setuptools that no longer supports
this version of Python. Please head to
https://bit.ly/setuptools-py2-sunset for support.
************************************************************
  warnings.warn(pre + "*" * 60 + msg + "*" * 60)
FAIL (3.50s)
      tests/GeneratedTests.hs:70:
      expected: ExitSuccess
       but got: ExitFailure 32

2 out of 2 tests failed (6.82s)

For pkgs.fixDarwinDylibNames, I tried:

    usesGRPC = haskellPackage:
      pkgsNew.haskell.lib.overrideCabal haskellPackage (oldAttributes: {
        setupHaskellDepends = (oldAttributes.setupHaskellDepends or []) ++
          pkgsNew.lib.optionals pkgsNew.stdenv.isDarwin [pkgsNew.fixDarwinDylibNames];
      });

The behavior in the nix-shell is the same.

Furthermore, there seem to be issues with the test suite terminating consistently on Darwin; it's hanging in the GHA build environment on this PR, and I'm able to reproduce hangs locally on master via nix-build release.nix --attr grpc-haskell (but have not further diagnosed them).

I suspect that we need to do some more vetting/debugging of the upgrade to grpc 1.34.1, at least on Darwin, and that's outside the intended scope of this PR, which I'm going to close until someone (which might be me...later) has an opportunity to stabilize things a bit.

intractable commented 3 years ago

Also, FWIW, attempting to build the tests in the nix-shell on master produces the same issue that I had worked around with the cabal upgrade pin on this PR:

[nix-shell]$ cabal configure --enable-tests && cabal build
...
Configuring library for grpc-haskell-core-0.1.0..
cabal: Missing dependencies on foreign libraries:
* Missing (or bad) header file: include/grpc_haskell.h
* Missing (or bad) C libraries: grpc, gpr
This problem can usually be solved by installing the system packages that
provide these libraries (you may need the "-dev" versions). If the libraries
are already installed but in a non-standard location then you can use the
flags --extra-include-dirs= and --extra-lib-dirs= to specify where they are.If
the library files do exist, it may contain errors that are caught by the C
compiler at the preprocessing stage. In this case you can re-run configure
with the verbosity flag -v3 to see the error messages.
If the header file does exist, it may contain errors that are caught by the C
compiler at the preprocessing stage. In this case you can re-run configure
with the verbosity flag -v3 to see the error messages.

cabal: Failed to build grpc-haskell-core-0.1.0 (which is required by
test:tests from grpc-haskell-0.1.0). The failure occurred during the configure
step.

(So at least that particular failure is consistent between the two platforms.)