cardano-sl-db Nix build issue

[mlb5000]

Environment:

[nix-shell:~/Documents/Projects/cardano-sl]$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.13.2
BuildVersion:   17C88

[nix-shell:~/Documents/Projects/cardano-sl]$ nix-shell --version
nix-shell (Nix) 1.11.16

[nix-shell:~/Documents/Projects/cardano-sl]$ stack --version
Version 1.6.3, Git revision b27e629b8c4ce369e3b8273f04db193b060000db (5454 commits) x86_64 hpack-0.20.0

Steps:

1. Follow initial environment setup steps in the build steps
2. Update ~/.stack/config.yaml as described here: https://github.com/input-output-hk/cardano-sl/issues/2223#issuecomment-354761767
3. git checkout master
4. Run nix-shell
5. Run stack setup (this was a new Haskell environment)
6. Run ./scripts/build/cardano-sl.sh

Output:

cardano-sl-core builds successfully, then

Completed 150 action(s).
stack build --ghc-options="-DGITREV=9c5394b4d35e638ed1338859792335b8b283754e -Wwarn +RTS -A256m -n2m -RTS" --test --no-haddock-deps --bench --jobs=4 --flag cardano-sl:with-wallet --no-run-tests --no-run-benchmarks --fast  cardano-sl-core

Warning: There were multiple candidates for the Cabal entry " hlint.hs 
         * /Users/matthewbaker/Documents/Projects/cardano-sl/.stack-work/downloaded/XTTklt4ROs4T/hlint.hs

         picking:
         /Users/matthewbaker/Documents/Projects/cardano-sl/.stack-work/downloaded/XTTklt4ROs4T/tests/hlint.hs
cardano-sl-core-1.0.3: build (lib)
Preprocessing library cardano-sl-core-1.0.3...
cardano-sl-core-1.0.3: copy/register
Installing library in
/Users/matthewbaker/Documents/Projects/cardano-sl/.stack-work/install/x86_64-osx-nix/lts-9.1/8.0.2/lib/x86_64-osx-ghc-8.0.2/cardano-sl-core-1.0.3-5rr3gYkhtK844dmHq1IyVc
Registering cardano-sl-core-1.0.3...
Building cardano-sl-db

stack build --ghc-options="-DGITREV=9c5394b4d35e638ed1338859792335b8b283754e -Wwarn +RTS -A256m -n2m -RTS" --test --no-haddock-deps --bench --jobs=4 --flag cardano-sl:with-wallet --no-run-tests --no-run-benchmarks --dependencies-only  cardano-sl-db

Warning: There were multiple candidates for the Cabal entry " hlint.hs 
         * /Users/matthewbaker/Documents/Projects/cardano-sl/.stack-work/downloaded/XTTklt4ROs4T/hlint.hs 
         picking: /Users/matthewbaker/Documents/Projects/cardano-sl/.stack-work/downloaded/XTTklt4ROs4T/tests/hlint.hs
rocksdb-haskell-1.0.0: download
rocksdb-haskell-1.0.0: configure
rocksdb-haskell-1.0.0: build
rocksdb-haskell-1.0.0: copy/register
stack build --ghc-options="-DGITREV=9c5394b4d35e638ed1338859792335b8b283754e -Wwarn +RTS -A256m -n2m -RTS" --test --no-haddock-deps --bench --jobs=4 --flag cardano-sl:with-wallet --no-run-tests --no-run-benchmarks --fast  cardano-sl-db

Warning: There were multiple candidates for the Cabal entry " hlint.hs 
         * /Users/matthewbaker/Documents/Projects/cardano-sl/.stack-work/downloaded/XTTklt4ROs4T/hlint.hs

         picking:
         /Users/matthewbaker/Documents/Projects/cardano-sl/.stack-work/downloaded/XTTklt4ROs4T/tests/hlint.hs
cardano-sl-db-1.0.3: build (lib)
Preprocessing library cardano-sl-db-1.0.3...
[ 5 of 13] Pos.DB.Pure
ERROR: flag 'help' was defined more than once (in files '/tmp/nix-build-google-gflags-2.2.0.drv-0/gflags-v2.2.0-src/src/gflags_reporting.cc' and '/tmp/gflags-20170916-46768-ftgkio/gflags-2.2.1/src/gflags_reporting.cc').

--  While building custom Setup.hs for package cardano-sl-db-1.0.3 using:
      /Users/matthewbaker/.stack/setup-exe-cache/x86_64-osx-nix/Cabal-simple_mPHDZzAJ_1.24.2.0_ghc-8.0.2 --builddir=.stack-work/dist/x86_64-osx-nix/Cabal-1.24.2.0 build lib:cardano-sl-db --ghc-options " -ddump-hi -ddump-to-file"
    Process exited with code: ExitFailure 1

RocksDB was installed using homebrew

[nix-shell:~/Documents/Projects/cardano-sl]$ brew info rocksdb
rocksdb: stable 5.9.2 (bottled)
Embeddable, persistent key-value store for fast storage
http://rocksdb.org
/usr/local/Cellar/rocksdb/5.9.2 (104 files, 100.2MB) *
  Poured from bottle on 2018-01-02 at 12:50:09
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/rocksdb.rb
==> Dependencies
Required: snappy ✔, lz4 ✔, gflags ✔, jemalloc ✔

[mlb5000]

Looks like the same basic thing as here: https://github.com/input-output-hk/cardano-sl/issues/1731

[mlb5000]

Confirmed that this also occurs on/release/1.0.4 and develop, so I'm not sure what's up

[mlb5000]

Still tracking this down. Looks like maybe the gflags version brought in by RocksDB, or some other dependency (2.2.1) is conflicting with the version brought in from the Nix cache (2.2.0)

[mlb5000]

I forced brew install of rocksdb with --ignore-dependencies and installed an old Formula for gflags 2.2.0 (brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/32b221ad628c48655e73e0083d2310471605fd37/Formula/gflags.rb) , but stack must realize that rocksdb points to gflags as a dylib, and add a new build entry for it (or something), because it's still pointing at both even though their version numbers match:

cardano-sl-db-1.0.3: build (lib)
Preprocessing library cardano-sl-db-1.0.3...
[ 5 of 13] Pos.DB.Pure
ERROR: flag 'help' was defined more than once (in files '/tmp/nix-build-google-gflags-2.2.0.drv-0/gflags-v2.2.0-src/src/gflags_reporting.cc' and '/tmp/gflags-20180102-74339-xlzs26/gflags-2.2.0/src/gflags_reporting.cc').

The first one is the package pulled down from the IOHK Nix cache, and the latter is the wrapper that stack created(I guess) based on librocksdb.dylib's dependencies

[mlb5000]

My investigation took me into the CI scripts where I found the line nix-shell -p moreutils expect --run "unbuffer ./scripts/ci/travis.sh | ts"; in .travis.yml. Using this as a basis for execution:

nix-shell -p moreutils expect --run "unbuffer ./scripts/build/cardano-sl.sh | ts"

The issue surrounding RocksDB and GFlags is gone and the build process completes successfully. Does anyone have any idea why this behaves differently than just running ./scripts/build/cardano-sl.sh from within an existing nix-shell?

[mlb5000]

Correction, now this:

Jan 03 10:11:54 [147 of 205] Pos.Block.Network.Listeners
Jan 03 10:11:54 
Jan 03 10:11:54 /Users/matthewbaker/Documents/Projects/cardano-sl/node/src/Pos/Block/Network/Listeners.hs:12:51: error:
Jan 03 10:11:54     Module ‘Node.Conversation’ does not export ‘sendRaw’
Jan 03 10:11:54 
Jan 03 10:11:54 --  While building custom Setup.hs for package cardano-sl-1.0.3 using:
Jan 03 10:11:54       /Users/matthewbaker/.stack/setup-exe-cache/x86_64-osx-nix/Cabal-simple_mPHDZzAJ_1.24.2.0_ghc-8.0.2 --builddir=.stack-work/dist/x86_64-osx-nix/Cabal-1.24.2.0 build lib:cardano-sl exe:cardano-node-simple test:cardano-test bench:cardano-bench-criterion --ghc-options " -ddump-hi -ddump-to-file"
Jan 03 10:11:54     Process exited with code: ExitFailure 1

I'm switching to master to see if this build procedure has any success there.

[mlb5000]

master was successful, culminating in a successful launch of ./scripts/launch/demo-with-wallet-api.sh