hackworthltd / primer

A pedagogical functional programming language.
GNU Affero General Public License v3.0
15 stars 1 forks source link

Property test failure (possibly Wasm-related?) #1198

Open dhess opened 10 months ago

dhess commented 10 months ago

The Wasm test suite generated a property test failure. It's not clear yet whether this is Wasm-related:

multiple requests accepted:                   FAIL (47.53s)
--
  | ✗ multiple requests accepted failed at test/Tests/Action/Available.hs:507:42
  | after 257 tests, 18 shrinks and 811 discards.
  | shrink path: 257/811:aGeBgEbEiVbCa4Lh
  | AddTm                                   34% ██████▊·············
  | AddTy                                   26% █████▏··············
  | AvailAct                                73% ██████████████▌·····
  | Eval1                                   10% ██··················
  | EvalFull                                11% ██▏·················
  | Question                                 5% █···················
  | RenameModule                            12% ██▍·················
  | Undo                                    11% ██▏·················
  | Just (Input AddCon)                     12% ██▎·················
  | Just (Input MakeChar)                    0% ····················
  | Just (Input MakeCon)                     1% ▏···················
  | Just (Input MakeForall)                  1% ▏···················
  | Just (Input MakeLAM)                     3% ▌···················
  | Just (Input MakeLam)                     6% █▏··················
  | Just (Input MakeLet)                     1% ▏···················
  | Just (Input MakeLetRec)                  2% ▎···················
  | Just (Input MakeTCon)                    5% █···················
  | Just (Input MakeTVar)                    1% ▏···················
  | Just (Input MakeVar)                     2% ▍···················
  | Just (Input MakeVarSat)                  1% ▏···················
  | Just (Input RenameCon)                   2% ▍···················
  | Just (Input RenameDef)                   2% ▍···················
  | Just (Input RenameLam)                   1% ▏···················
  | Just (Input RenameLet)                   1% ▏···················
  | Just (Input RenameType)                 15% ███·················
  | Just (Input RenameTypeParam)             4% ▋···················
  | Just (NoInput AddConField)               5% █···················
  | Just (NoInput DeleteCon)                 3% ▌···················
  | Just (NoInput DeleteConField)            1% ▏···················
  | Just (NoInput DeleteDef)                 1% ▏···················
  | Just (NoInput DeleteExpr)                3% ▌···················
  | Just (NoInput DeleteKind)                6% █▏··················
  | Just (NoInput DeleteType)                8% █▌··················
  | Just (NoInput DeleteTypeDef)            11% ██▏·················
  | Just (NoInput DuplicateDef)              4% ▋···················
  | Just (NoInput EnterHole)                 0% ····················
  | Just (NoInput LetToRec)                  0% ····················
  | Just (NoInput MakeAPP)                   0% ····················
  | Just (NoInput MakeAnn)                   1% ▏···················
  | Just (NoInput MakeApp)                   2% ▍···················
  | Just (NoInput MakeCase)                  4% ▊···················
  | Just (NoInput MakeFun)                  12% ██▎·················
  | Just (NoInput MakeKFun)                  2% ▍···················
  | Just (NoInput MakeTApp)                  2% ▎···················
  | Just (NoInput Raise)                     4% ▊···················
  | Just (NoInput RaiseType)                 1% ▏···················
  | ignoring - capture                       0% ····················
  | ignoring - clash                         5% ▉···················
  | name-capture with entered name          10% ██··················
  | rename def name clash with entered name  2% ▎···················
  |  
  | ┏━━ test/Tests/Action/Available.hs ━━━
  | 327 ┃ tasty_multiple_requests_accepted :: Property
  | 328 ┃ tasty_multiple_requests_accepted = withTests 500
  | 329 ┃   $ withDiscards 2000
  | 330 ┃   $ propertyWT []
...
507 ┃     (Left err, _) -> annotateShow err >> failure
--
  | ┃     │ ActionError
  | ┃     │   (InternalFailure
  | ┃     │      "transformCaseBranches: scrutinees did not have a cached synthesised type")
  | ┃     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  | 508 ┃     (Right _, a'') -> ensureSHNormal a'' $> Succeed a''
  |  
  | ┏━━ test/Tests/Action/Available.hs ━━━
  | 679 ┃ toProgAction ::
  | 680 ┃   Monad m =>
  | 681 ┃   Level ->
  | 682 ┃   App ->
  | 683 ┃   ( Either (ASTTypeDef TypeMeta KindMeta) Def
  | 684 ┃   , Selection' ID
  | 685 ┃   , Maybe Available.Action
  | 686 ┃   ) ->
  | 687 ┃   PropertyT m PA
  | 688 ┃ toProgAction l a (def, loc, action) = do
  | 689 ┃   def' <- case (defAST <$> def, action) of
  | 690 ┃     (Left d, _) -> pure $ Left d
  | 691 ┃     (Right Nothing, Nothing) -> discard
  | 692 ┃     (Right Nothing, Just act) -> do
  | 693 ┃       annotate "Expected no action to be available on a primitive"
  | 694 ┃       annotateShow def
  | 695 ┃       annotateShow act
  | 696 ┃       failure
  | 697 ┃     (Right (Just d), _) -> pure $ Right d
  | 698 ┃   case action of
  | 699 ┃     Nothing -> discard
  | 700 ┃     Just action' -> case action' of
  | 701 ┃       Available.NoInput act' -> do
  | 702 ┃         progActs <-
  | 703 ┃           either (\e -> annotateShow e >> failure) pure
  | 704 ┃             $ toProgActionNoInput (map snd $ progAllDefs $ appProg a) def' loc act'
  | 705 ┃         pure $ NoOpt progActs
  | 706 ┃       Available.Input act' -> do
  | 707 ┃         Available.Options{Available.opts, Available.free} <-
  | 708 ┃           maybe (annotate "id not found" >> failure) pure
  | 709 ┃             $ Available.options
  | 710 ┃               (map snd $ progAllTypeDefs $ appProg a)
  | 711 ┃               (map snd $ progAllDefs $ appProg a)
  | 712 ┃               (progCxt $ appProg a)
  | 713 ┃               l
  | 714 ┃               def'
  | 715 ┃               loc
  | 716 ┃               act'
  | 717 ┃         let opts' = [Gen.element $ (OptOffered,) <$> opts \| not (null opts)]
  | 718 ┃         let opts'' =
  | 719 ┃               opts' <> case free of
  | 720 ┃                 Available.FreeNone -> []
  | 721 ┃                 Available.FreeVarName -> [(OptGen,) . (\t -> Available.Option t Nothing False) <$> (unName <$> genName)]
  | 722 ┃                 Available.FreeInt -> [(OptGen,) . (\t -> Available.Option t Nothing False) <$> (show <$> genInt)]
  | 723 ┃                 Available.FreeChar -> [(OptGen,) . (\t -> Available.Option t Nothing False) . T.singleton <$> genChar]
  | 724 ┃         case opts'' of
  | 725 ┃           [] -> pure NoOfferedOpts
  | 726 ┃           options -> do
  | 727 ┃             (mkPA, opt) <- forAllWithT (show . snd) $ Gen.choice options
  | ┃             │ Option {option = "x", context = Nothing, matchesType = False}
  | 728 ┃             progActs <- either (\e -> annotateShow e >> failure) pure $ toProgActionInput def' loc opt act'
  | 729 ┃             pure $ mkPA opt progActs
  |  
  | This failure can be reproduced by running:
  | > recheckAt (Seed 8950671910154333759 17555302804257829771) "257/811:aGeBgEbEiVbCa4Lh" multiple requests accepted
  |  
  | Use "--pattern '$NF ~ /multiple requests accepted/' --hedgehog-replay '257/811:aGeBgEbEiVbCa4Lh Seed 8950671910154333759 17555302804257829771'" to reproduce from the command-line.
  |  
  | Use -p '/multiple requests accepted/' to rerun this test only.
dhess commented 10 months ago

Full log is here: https://buildkite.com/hackworthltd/primer-wasm/builds/65#018ce62e-9489-4564-84bf-92f19d828b02

Note that this is on https://github.com/hackworthltd/primer/commit/d31f648a226314d9c2f312292e2c868193de62e6

dhess commented 3 weeks ago

Here's another instance of failure, still generated by the Wasm test suite, but this time running on macOS and with a later version of GHC with the Wasm backend:

             This failure can be reproduced by running:
              > recheckAt (Seed 12646156092523544096 799955637605004819) "61/191:aFbChDhCfDb4A" multiple requests accepted

          Use "--pattern '$NF ~ /multiple requests accepted/' --hedgehog-replay '61/191:aFbChDhCfDb4A Seed 12646156092523544096 799955637605004819'" to reproduce from the command-line.

          Use -p '/multiple requests accepted/' to rerun this test only.

This failure occurred locally, on my MacBook, so there are no Buildkite logs.