search

HuwCampbell / grenade

Deep Learning in Haskell
BSD 2-Clause "Simplified" License
1.45k stars 84 forks source link

Test suite fails #87

Open jappeace opened 5 years ago

jappeace commented 5 years ago

Hi, I'm trying to build grenade with nix, the test suite fails however:

━━━ Test.Grenade.Recurrent.Layers.LSTM ━━━
  ✗ prop_lstm_reference_forwards failed after 54 tests and 55 shrinks.

       ┏━━ test/Test/Grenade/Recurrent/Layers/LSTM.hs ━━━
    44 ┃ prop_lstm_reference_forwards =
    45 ┃   property $ do
    46 ┃     input :: S.R 3                       <- forAll randomVector
       ┃     │ (vector54319561065,0.7925737992825796,-6.308042121263235e-2] :: R 3)
    47 ┃     cell :: S.R 2                        <- forAll randomVector
       ┃     │ (vector102552342276,-0.5733412138993579] :: R 2)
    48 ┃     net@(LSTM lstmWeights _) :: LSTM 3 2 <- forAll genLSTM
       ┃     │ LSTM
    49 ┃
    50 ┃     let actual          = runRecurrentForwards net (S1D cell) (S1D input)
    51 ┃     case actual of
    52 ┃       (_, (S1D cellOut) :: S ('D1 2), (S1D output) :: S ('D1 2)) ->
    53 ┃         let cellOut'        = Reference.Vector . H.toList . S.extract $ cellOut
    54 ┃             output'         = Reference.Vector . H.toList . S.extract $ output
    55 ┃             refNet          = Reference.lstmToReference lstmWeights
    56 ┃             refCell         = Reference.Vector . H.toList . S.extract $ cell
    57 ┃             refInput        = Reference.Vector . H.toList . S.extract $ input
    58 ┃             (refCO, refO)   = Reference.runLSTM refNet refCell refInput
    59 ┃         in do toList refCO ~~~ toList cellOut'
       ┃         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       ┃         │ Failed (- lhs ~/~ + rhs)
       ┃         │ - [ 8.600932134870043e-2 , 0.10405281905297237 ]
       ┃         │ + [ 8.016375201087303e-2 , 7.653272311821224e-2 ]
    60 ┃               toList refO ~~~ toList output'

    This failure can be reproduced by running:
    > recheck (Size 53) (Seed 10790320388850313198 5063647523751707663) prop_lstm_reference_forwards

  ✗ prop_lstm_reference_backwards failed after 58 tests and 36 shrinks.

       ┏━━ test/Test/Grenade/Recurrent/Layers/LSTM.hs ━━━
    63 ┃ prop_lstm_reference_backwards =
    64 ┃   property $ do
    65 ┃     input :: S.R 3                       <- forAll randomVector
       ┃     │ (vector55046569861,0.7353593240191039,0.10220169420456582] :: R 3)
    66 ┃     cell :: S.R 2                        <- forAll randomVector
       ┃     │ (vector94712659202,8.446059985294041e-2] :: R 2)
    67 ┃     net@(LSTM lstmWeights _) :: LSTM 3 2 <- forAll genLSTM
       ┃     │ LSTM
    68 ┃     let (tape, _ :: S ('D1 2), _ :: S ('D1 2))
    69 ┃                                           = runRecurrentForwards  net (S1D cell) (S1D input)
    70 ┃         actualBacks                       = runRecurrentBackwards net tape (S1D (S.konst 1) :: S ('D1 2)) (S1D (S.konst 1) :: S ('D1 2))
    71 ┃     case actualBacks of
    72 ┃       (actualGradients, _, _ :: S ('D1 3)) ->
    73 ┃         let refNet          = Reference.lstmToReference lstmWeights
    74 ┃             refCell         = Reference.Vector . H.toList . S.extract $ cell
    75 ┃             refInput        = Reference.Vector . H.toList . S.extract $ input
    76 ┃             refGradients    = Reference.runLSTMback refCell refInput refNet
    77 ┃         in toList refGradients ~~~ toList (Reference.lstmToReference actualGradients)
       ┃         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       ┃         │ Failed (- lhs ~/~ + rhs)
       ┃         │   [
       ┃         │ -   -0.2833762365900625
       ┃         │ +   -0.2772291966623182
       ┃         │ - , 0.2792387920721928
       ┃         │ + , 0.273181502212966
       ┃         │ - , 3.8809159964732e-2
       ┃         │ + , 3.796730583209162e-2
       ┃         │ - , -1.8789963351874522e-2
       ┃         │ + , -1.9046776736865983e-2
       ┃         │ - , 1.851561984376432e-2
       ┃         │ + , 1.8768683616069006e-2
       ┃         │ - , 2.5733374902189056e-3
       ┃         │ + , 2.60850879413327e-3
       ┃         │ - , 0.3522269593369693
       ┃         │ + , 0.34458639918017514
       ┃         │ - , 3.2072315003399895e-2
       ┃         │ + , 3.1376597524498194e-2
       ┃         │ - , 2.3355281081871578e-2
       ┃         │ + , 2.36744913261781e-2
       ┃         │ - , 2.1266342964228386e-3
       ┃         │ + , 2.1557002473284216e-3
       ┃         │ - , 0.3797310824129001
       ┃         │ + , 0.3714938986832906
       ┃         │ - , 2.51790100961354e-2
       ┃         │ + , 2.5523146308241297e-2
       ┃         │ - , -0.14407943439166543
       ┃         │ + , -0.14095404163950748
       ┃         │ - , 0.14197579763956925
       ┃         │ + , 0.13889603729210484
       ┃         │ - , 1.9732077340779888e-2
       ┃         │ + , 1.930404615252405e-2
       ┃         │ - , 0.10233133248392495
       ┃         │ + , 0.10372995447129899
       ┃         │ - , -0.10083724033390323
       ┃         │ + , -0.1022154416927028
       ┃         │ - , -1.401455922896567e-2
       ┃         │ + , -1.4206104381415103e-2
       ┃         │ - , 0.17908580369842605
       ┃         │ + , 0.1752010475203034
       ┃         │ - , 1.6306804906884104e-2
       ┃         │ + , 1.5953075243230058e-2
       ┃         │ - , -0.12719434247358946
       ┃         │ + , -0.12893278171536413
       ┃         │ - , -1.1581785295885263e-2
       ┃         │ + , -1.1740080308514117e-2
       ┃         │ - , 0.19306996321689512
       ┃         │ + , 0.1888818605480774
       ┃         │ - , -0.13712648638597202
       ┃         │ + , -0.13900067402973101
       ┃         │ - , -0.11252340904695016
       ┃         │ + , -0.131574427655387
       ┃         │ - , 0.11088050713147728
       ┃         │ + , 0.12965337068552493
       ┃         │ - , 1.5410392325159424e-2
       ┃         │ + , 1.80194820553455e-2
       ┃         │ - , 4.787291093695013e-2
       ┃         │ + , 4.6151157876618276e-2
       ┃         │ - , -4.71739408493592e-2
       ┃         │ + , -4.5477326303979465e-2
       ┃         │ - , -6.556327661366858e-3
       ┃         │ + , -6.320528814073785e-3
       ┃         │ - , 0.1398627446668094
       ┃         │ + , 0.16354250849410923
       ┃         │ - , 1.2735317059880666e-2
       ┃         │ + , 1.489149739912809e-2
       ┃         │ - , -5.9504389135933686e-2
       ┃         │ + , -5.736430903441441e-2
       ┃         │ - , -5.418221013079177e-3
       ┃         │ + , -5.22335426217051e-3
       ┃         │ - , 0.1507841180627999
       ┃         │ + , 0.17631294858261248
       ┃         │ - , -6.415087061319925e-2
       ┃         │ + , -6.184367943473307e-2
       ┃         │ - , -0.6893313089947273
       ┃         │ + , -0.6743782306038787
       ┃         │ - , 0.6792667034381152
       ┃         │ + , 0.6645319479841436
       ┃         │ - , 9.440583078310466e-2
       ┃         │ + , 9.235796530850202e-2
       ┃         │ - , -0.7491175754191286
       ┃         │ + , -0.7593562021102653
       ┃         │ - , 0.7381800584171601
       ┃         │ + , 0.7482691956860943
       ┃         │ - , 0.10259372545373369
       ┃         │ + , 0.10399593371881949
       ┃         │ - , 0.9237208005000677
       ┃         │ + , 0.9036833100206666
       ┃         │ - , 1.00383585861486
       ┃         │ + , 1.0175558686009876
       ┃         │   ]

    This failure can be reproduced by running:
    > recheck (Size 57) (Seed 15557724053009315116 9838685224472302247) prop_lstm_reference_backwards

  ✗ prop_lstm_reference_backwards_input failed after 73 tests and 51 shrinks.

       ┏━━ test/Test/Grenade/Recurrent/Layers/LSTM.hs ━━━
    79 ┃ prop_lstm_reference_backwards_input =
    80 ┃   property $ do
    81 ┃     input :: S.R 3                       <- forAll randomVector
       ┃     │ (vector132148291512,0.2850784935453341,-1.40887764348131e-2] :: R 3)
    82 ┃     cell :: S.R 2                        <- forAll randomVector
       ┃     │ (vector429500642426,0.7481729899291754] :: R 2)
    83 ┃     net@(LSTM lstmWeights _) :: LSTM 3 2 <- forAll genLSTM
       ┃     │ LSTM
    84 ┃     let (tape, _ :: S ('D1 2), _ :: S ('D1 2))
    85 ┃                                           = runRecurrentForwards  net (S1D cell) (S1D input)
    86 ┃         actualBacks                       = runRecurrentBackwards net tape (S1D (S.konst 1) :: S ('D1 2)) (S1D (S.konst 1) :: S ('D1 2))
    87 ┃     case actualBacks of
    88 ┃       (_, _, S1D actualGradients :: S ('D1 3)) ->
    89 ┃         let refNet          = Reference.lstmToReference lstmWeights
    90 ┃             refCell         = Reference.Vector . H.toList . S.extract $ cell
    91 ┃             refInput        = Reference.Vector . H.toList . S.extract $ input
    92 ┃             refGradients    = Reference.runLSTMbackOnInput refCell refNet refInput
    93 ┃         in toList refGradients ~~~ H.toList (S.extract actualGradients)
       ┃         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       ┃         │ Failed (- lhs ~/~ + rhs)
       ┃         │   [
       ┃         │ -   0.41783498961490245
       ┃         │ +   -7.144093770897443e-2
       ┃         │ - , -0.643583384088483
       ┃         │ + , -0.5033482252382088
       ┃         │ - , 0.24741239536602957
       ┃         │ + , 0.2196259366479876
       ┃         │   ]

    This failure can be reproduced by running:
    > recheck (Size 72) (Seed 15511439570864369243 14061565670218007919) prop_lstm_reference_backwards_input

  ✗ prop_lstm_reference_backwards_cell failed after 45 tests and 31 shrinks.

        ┏━━ test/Test/Grenade/Recurrent/Layers/LSTM.hs ━━━
     95 ┃ prop_lstm_reference_backwards_cell =
     96 ┃   property $ do
     97 ┃     input :: S.R 3                       <- forAll randomVector
        ┃     │ (vector957622892669,-0.26370740740732634,-0.8739678849810585] :: R 3)
     98 ┃     cell :: S.R 2                        <- forAll randomVector
        ┃     │ (vector544704658233,-0.82774989764567] :: R 2)
     99 ┃     net@(LSTM lstmWeights _) :: LSTM 3 2 <- forAll genLSTM
        ┃     │ LSTM
    100 ┃     let (tape, _ :: S ('D1 2), _ :: S ('D1 2))
    101 ┃                                           = runRecurrentForwards  net (S1D cell) (S1D input)
    102 ┃         actualBacks                       = runRecurrentBackwards net tape (S1D (S.konst 1) :: S ('D1 2)) (S1D (S.konst 1) :: S ('D1 2))
    103 ┃     case actualBacks of
    104 ┃       (_, S1D actualGradients, _ :: S ('D1 3)) ->
    105 ┃         let refNet          = Reference.lstmToReference lstmWeights
    106 ┃             refCell         = Reference.Vector . H.toList . S.extract $ cell
    107 ┃             refInput        = Reference.Vector . H.toList . S.extract $ input
    108 ┃             refGradients    = Reference.runLSTMbackOnCell refInput refNet refCell
    109 ┃         in toList refGradients ~~~ H.toList (S.extract actualGradients)
        ┃         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        ┃         │ Failed (- lhs ~/~ + rhs)
        ┃         │ - [ 0.3555130556015041 , 4.6592664606872156e-2 ]
        ┃         │ + [ 0.4057801473816573 , -1.403610494537344e-2 ]
    110 ┃
    111 ┃ (~~~) :: (Monad m, Eq a, Ord a, Num a, Fractional a, Show a, HasCallStack) => [a] -> [a] -> PropertyT m ()
    112 ┃ (~~~) x y =
    113 ┃   if all (< 1e-8) (zipWith (-) x y) then
    114 ┃     success
    115 ┃   else
    116 ┃     case valueDiff <$> mkValue x <*> mkValue y of
    117 ┃       Nothing ->
    118 ┃         withFrozenCallStack $
    119 ┃           failWith Nothing $ unlines [
    120 ┃               "━━━ Not Simliar ━━━"
    121 ┃             , showPretty x
    122 ┃             , showPretty y
    123 ┃             ]
    124 ┃       Just diff ->
    125 ┃         withFrozenCallStack $
    126 ┃           failWith (Just $ Diff "Failed (" "- lhs" "~/~" "+ rhs" ")" diff) ""

    This failure can be reproduced by running:
    > recheck (Size 44) (Seed 17734744653642261044 4653266027254713753) prop_lstm_reference_backwards_cell

  ✗ 4 failed.
Test suite test: FAIL
Test suite logged to: dist/test/grenade-0.1.0-test.log
0 of 1 test suites (0 of 1 test cases) passed.
builder for '/nix/store/g2ishp6kxmakfxpxxl0l58m0n6w7dfzi-grenade-0.1.0.drv' failed with exit code 1
cannot build derivation '/nix/store/djy93n6n0wjvbv85a1g9s6nc7hh0nxq9-ghc-8.4.3-with-packages.drv': 1 dependencies couldn't be built
error: build of '/nix/store/djy93n6n0wjvbv85a1g9s6nc7hh0nxq9-ghc-8.4.3-with-packages.drv' failed
make[1]: *** [makefile:70: enter] Fout 100
make[1]: Map '/home/jappie/projects/raster' wordt verlaten

I used cabal2nix to just grab the latest version from github, this one to be precise:

{ mkDerivation, ad, base, bytestring, cereal, constraints
, containers, criterion, deepseq, fetchgit, hedgehog, hmatrix
, MonadRandom, mtl, primitive, random, reflection, singletons
, stdenv, text, transformers, typelits-witnesses, vector
}:
mkDerivation {
  pname = "grenade";
  version = "0.1.0";
  src = fetchgit {
    url = "https://github.com/HuwCampbell/grenade";
    sha256 = "1nzzacpiz86z3svvi2lpczzkv4algjfw3m2h2pqsw5n44xp8b320";
    rev = "f729b291e40a9044d95d49d31bce3149a5b9d599";
    fetchSubmodules = true;
  };
  libraryHaskellDepends = [
    base bytestring cereal containers deepseq hmatrix MonadRandom
    primitive singletons vector
  ];
  testHaskellDepends = [
    ad base constraints hedgehog hmatrix MonadRandom mtl random
    reflection singletons text transformers typelits-witnesses vector
  ];
  benchmarkHaskellDepends = [ base bytestring criterion hmatrix ];
  description = "Practical Deep Learning in Haskell";
  license = stdenv.lib.licenses.bsd2;
}

I'll disable the tests for now so it's not a big deal, but I just wanted to let you know.

erikd commented 5 years ago

I don't use Nix, but i just tested building this with ghc-8.6.3 and cabal-install 2.0.0.1.

I built it and ran the tests using:

./mafia build -w
./mafia test

Unless this can be reproduced outside Nix, I can't work on it.

HuwCampbell commented 5 years ago

I’m running NixOS at home at the moment, I’ll try running it there. Do you have a pure derivation I can test against? (That is, one including all dependencies)

On Thu, 7 Mar 2019 at 12:06 pm, Erik de Castro Lopo < notifications@github.com> wrote:

I don't use Nix, but i just tested building this with ghc-8.6.3 and cabal-install 2.0.0.1.

I built it and ran the tests using:

./mafia build -w ./mafia test

Unless this can be reproduced outside Nix, I can't work on it.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/HuwCampbell/grenade/issues/87#issuecomment-470341077, or mute the thread https://github.com/notifications/unsubscribe-auth/AE9t0e5-q86Ae2wukXOcgME527fov_UKks5vUGYOgaJpZM4bhqww .

jappeace commented 5 years ago

This project fails reproducibly: https://github.com/jappeace/awesome-project-name/tree/show-grenade-fails

You can't enter the shell unless you disable the tests. Enter shell with:

   nix-shell -A shells.ghc
jappeace commented 5 years ago

I'm not sure how to make a pure derivation, but since the project I linked is pinned, that should be good enough?