ucsd-progsys / liquidhaskell

Liquid Types For Haskell
BSD 3-Clause "New" or "Revised" License
1.19k stars 138 forks source link

Reflection captures bound name #1601

Open wenkokke opened 4 years ago

wenkokke commented 4 years ago

This is a bit of a frustrating bug report to write.

There is a problem with reflection that I've encountered several times, but it's unclear to me how to minimize my code to isolate the issue. I am writing a tool for verifying neural networks using Liquid Haskell. At the moment, my project fails to compile with the following error:

:1:1-1:1: Error
  elaborate PLE1 195 failed on:
    ...
    && Leadbeater.Network.norm == Leadbeater.LinearAlgebra.map lam lam_arg##1 : real . Leadbeater.Prelude.rdiv lam_arg##1 (Leadbeater.LinearAlgebra.sumPos xs)
    ...
  with error
      Unbound symbol xs --- perhaps you meant: fst ?
  in environment
    ...

I've omitted a huge number of lines, basically containing the reflection of every single line of Haskell in my current codebase.

The problem seemingly lies with the following snippet:

{-@ reflect norm @-}
{-@ norm :: xs:VectorNE Rpos -> VectorX R xs @-}
norm :: Vector R -> Vector R
norm xs = let s = sumPos xs in map (`rdiv` s) xs

It seems that Liquid Haskell somehow reflects the xs in the let binding as a literal xs. This problem remains if I rewrite the norm function to inline the definition for s, or to factor the lambda out into a top-level function:

{-@ reflect norm @-}
{-@ norm :: xs:VectorNE Rpos -> VectorX R xs @-}
norm2 :: Vector R -> Vector R
norm2 xs = map (`rdiv` sumPos xs) xs
{-@ reflect norm @-}
{-@ norm :: s:Rpos -> xs:VectorNE Rpos -> VectorX R xs @-}
norm3' :: R -> Vector R -> Vector R
norm3' s xs = map (`rdiv` s) xs

{-@ reflect norm3 @-}
{-@ norm3 :: xs:VectorNE Rpos -> VectorX R xs @-}
norm3 :: Vector R -> Vector R
norm3 xs = norm3' (sumPos xs) xs

This is where things get weird. If I only compile and verify the library, which includes the norm function, everything checks just fine. The error above only happens if I also compile and verify my test. The error itself is the failure to elaborate the following definition, which doesn't even use norm:

{-@ reflect example @-}
{-@ example :: NetworkN 2 1 @-}
example :: Network
example = NLast
          (
            Layer { bias       = 0.184
                  , weights    = (1 >< 2)
                                 [ 0.194 , 0.195
                                 ]
                  , activation = Sigmoid
                  }
          )

(The only function in the above definition is ><, the rest are all constructors and constants.)

Similarly, if I extract the code involved in defining norm, as I've done here, everything checks just fine...

If you have any idea what may be causing this, please let me know, I'd be happy to investigate more and provide further evidence.

ranjitjhala commented 4 years ago

Ouch sorry about this! Is there some way you can point us to your repo so we can try to replicate? Specifically I’m wondering what the definition of the various VectorX aliases look like...

On Mon, Feb 3, 2020 at 8:39 AM Wen Kokke notifications@github.com wrote:

This is a bit of a frustrating bug report to write.

There is a problem with reflection that I've encountered several times, but it's unclear to me how to minimize my code to isolate the issue. I am writing a tool for verifying neural networks using Liquid Haskell. At the moment, my project https://github.com/wenkokke/leadbeater fails to compile with the following error:

:1:1-1:1: Error

elaborate PLE1 195 failed on:

...

&& Leadbeater.Network.norm == Leadbeater.LinearAlgebra.map lam lam_arg##1 : real . Leadbeater.Prelude.rdiv lam_arg##1 (Leadbeater.LinearAlgebra.sumPos xs)

...

with error

  Unbound symbol xs --- perhaps you meant: fst ?

in environment

...

I've omitted a huge number of lines, basically containing the reflection of every single line of Haskell in my current codebase.

The problem seemingly lies with the following snippet https://github.com/wenkokke/leadbeater/blob/master/src/Leadbeater/Network.hs#L60-L63 :

{-@ reflect norm @-}

{-@ norm :: xs:VectorNE Rpos -> VectorX R xs @-} norm :: Vector R -> Vector R

norm xs = let s = sumPos xs in map (rdiv s) xs

It seems that Liquid Haskell somehow reflects the xs in the let binding as a literal xs. This problem remains if I rewrite the norm function to inline the definition for s, or to factor the lambda out into a top-level function:

{-@ reflect norm @-}

{-@ norm :: xs:VectorNE Rpos -> VectorX R xs @-} norm2 :: Vector R -> Vector R

norm2 xs = map (rdiv sumPos xs) xs

{-@ reflect norm @-}

{-@ norm :: s:Rpos -> xs:VectorNE Rpos -> VectorX R xs @-} norm3' :: R -> Vector R -> Vector R

norm3' s xs = map (rdiv s) xs

{-@ reflect norm3 @-}

{-@ norm3 :: xs:VectorNE Rpos -> VectorX R xs @-} norm3 :: Vector R -> Vector R

norm3 xs = norm3' (sumPos xs) xs

This is where things get weird. If I only build and verify my library, which includes the norm function, everything checks out. The error above only happens if I also verify my tests. The error itself is the failure to elaborate the following definition https://github.com/wenkokke/leadbeater/blob/master/test/perceptron.hs#L26-L37, which doesn't even use norm:

{-@ reflect example @-}

{-@ example :: NetworkN 2 1 @-} example :: Network

example = NLast

      (

        Layer { bias       = 0.184

              , weights    = (1 >< 2)

                             [ 0.194 , 0.195

                             ]

              , activation = Sigmoid

              }

      )

(The only function in the above definition is >< https://github.com/wenkokke/leadbeater/blob/master/src/Leadbeater/LinearAlgebra.hs#L137-L140, the rest are all constructors and constants.)

Similarly, if I extract the code involved in defining norm, as I've done here https://gist.github.com/wenkokke/8e46debb9d6e8177e007f1e545110f47, everything checks just fine.

If you have any idea what may be causing this, please let me know, I'd be happy to investigate more and provide further evidence.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ucsd-progsys/liquidhaskell/issues/1601?email_source=notifications&email_token=AAMS4OEHIW7ZXM5MI62DS7LRBBCEXA5CNFSM4KPIKKC2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IKUWXXA, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMS4OGTQAAEHQQHXCJJAY3RBBCEXANCNFSM4KPIKKCQ .

wenkokke commented 4 years ago

There's a bunch of links into my repository in the bug report itself. The aliases for Vector are defined here. I've been trying to follow this tutorial fairly closely.

ranjitjhala commented 4 years ago

Ah just saw the “my project” link will take a look ...

Btw I’m guessing you’ve tried renaming the “xs” in both the signature and code of norm?

On Mon, Feb 3, 2020 at 8:44 AM Ranjit Jhala jhala@cs.ucsd.edu wrote:

Ouch sorry about this! Is there some way you can point us to your repo so we can try to replicate? Specifically I’m wondering what the definition of the various VectorX aliases look like...

On Mon, Feb 3, 2020 at 8:39 AM Wen Kokke notifications@github.com wrote:

This is a bit of a frustrating bug report to write.

There is a problem with reflection that I've encountered several times, but it's unclear to me how to minimize my code to isolate the issue. I am writing a tool for verifying neural networks using Liquid Haskell. At the moment, my project https://github.com/wenkokke/leadbeater fails to compile with the following error:

:1:1-1:1: Error

elaborate PLE1 195 failed on:

...

&& Leadbeater.Network.norm == Leadbeater.LinearAlgebra.map lam lam_arg##1 : real . Leadbeater.Prelude.rdiv lam_arg##1 (Leadbeater.LinearAlgebra.sumPos xs)

...

with error

  Unbound symbol xs --- perhaps you meant: fst ?

in environment

...

I've omitted a huge number of lines, basically containing the reflection of every single line of Haskell in my current codebase.

The problem seemingly lies with the following snippet https://github.com/wenkokke/leadbeater/blob/master/src/Leadbeater/Network.hs#L60-L63 :

{-@ reflect norm @-}

{-@ norm :: xs:VectorNE Rpos -> VectorX R xs @-} norm :: Vector R -> Vector R

norm xs = let s = sumPos xs in map (rdiv s) xs

It seems that Liquid Haskell somehow reflects the xs in the let binding as a literal xs. This problem remains if I rewrite the norm function to inline the definition for s, or to factor the lambda out into a top-level function:

{-@ reflect norm @-}

{-@ norm :: xs:VectorNE Rpos -> VectorX R xs @-} norm2 :: Vector R -> Vector R

norm2 xs = map (rdiv sumPos xs) xs

{-@ reflect norm @-}

{-@ norm :: s:Rpos -> xs:VectorNE Rpos -> VectorX R xs @-} norm3' :: R -> Vector R -> Vector R

norm3' s xs = map (rdiv s) xs

{-@ reflect norm3 @-}

{-@ norm3 :: xs:VectorNE Rpos -> VectorX R xs @-} norm3 :: Vector R -> Vector R

norm3 xs = norm3' (sumPos xs) xs

This is where things get weird. If I only build and verify my library, which includes the norm function, everything checks out. The error above only happens if I also verify my tests. The error itself is the failure to elaborate the following definition https://github.com/wenkokke/leadbeater/blob/master/test/perceptron.hs#L26-L37, which doesn't even use norm:

{-@ reflect example @-}

{-@ example :: NetworkN 2 1 @-} example :: Network

example = NLast

      (

        Layer { bias       = 0.184

              , weights    = (1 >< 2)

                             [ 0.194 , 0.195

                             ]

              , activation = Sigmoid

              }

      )

(The only function in the above definition is >< https://github.com/wenkokke/leadbeater/blob/master/src/Leadbeater/LinearAlgebra.hs#L137-L140, the rest are all constructors and constants.)

Similarly, if I extract the code involved in defining norm, as I've done here https://gist.github.com/wenkokke/8e46debb9d6e8177e007f1e545110f47, everything checks just fine.

If you have any idea what may be causing this, please let me know, I'd be happy to investigate more and provide further evidence.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ucsd-progsys/liquidhaskell/issues/1601?email_source=notifications&email_token=AAMS4OEHIW7ZXM5MI62DS7LRBBCEXA5CNFSM4KPIKKC2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IKUWXXA, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMS4OGTQAAEHQQHXCJJAY3RBBCEXANCNFSM4KPIKKCQ .

wenkokke commented 4 years ago

Btw I’m guessing you’ve tried renaming the “xs” in both the signature and code of norm?

I hadn't tried that, but I just have. It seems the variable xs from the type signature is ending up in the reflection somehow:

{-@ reflect norm @-}
{-@ norm :: bar:VectorNE Rpos -> VectorX R bar @-}
norm :: Vector R -> Vector R
norm foo = let s = sumPos foo in map (`rdiv` s) foo
:1:1-1:1: Error
  elaborate PLE1 195 failed on:
    ...
      && Leadbeater.Network.norm == Leadbeater.LinearAlgebra.map lam lam_arg##1 : real . Leadbeater.Prelude.rdiv lam_arg##1 (Leadbeater.LinearAlgebra.sumPos bar)
    ...
  with error
      Unbound symbol bar --- perhaps you meant: bvor ?
  in environment
    ...
wenkokke commented 4 years ago

If I comment out the type signature, the error message changes to:

:1:1-1:1: Error
  elaborate PLE1 195 failed on:
    ...
      && Leadbeater.Network.norm == Leadbeater.LinearAlgebra.map lam lam_arg##1 : real . Leadbeater.Prelude.rdiv lam_arg##1 (Leadbeater.LinearAlgebra.sumPos lq1)
    ...
  with error
      Unbound symbol lq1 --- perhaps you meant: len ?
  in environment
    ...

Which looks like some Liquid Haskell internals to me?

ranjitjhala commented 4 years ago

Yes, I worry this is some gross PLE related bug...

ranjitjhala commented 4 years ago

oh yikes looking again, I see there are "lam" in the refinements? 

In general, the interaction of PLE and lambdas in refinements is very suspect and I'd advise against using lambdas in refinements at all... 

Even more puzzling, I thought the lam comes from the partially applied rdiv in norm but the following works fine:

{-@ LIQUID "--reflection" @-}
{-@ LIQUID "--ple" @-}

{-@ reflect mmap @-}
mmap :: (a -> b) -> [a] -> [b]
mmap _ [] = []
mmap f (x:xs) = f x : mmap f xs

{-@ reflect add @-}
add :: Int -> Int -> Int
add x y = x + y

{-@ reflect bumpOK @-}
bumpOK :: [Int] -> [Int]
bumpOK xs = mmap (add 1) xs

{-@ testOK :: _ -> {b:_ | b} @-}
testOK :: () -> Bool
testOK () = (bumpOK [0, 1]) == [1, 2]

Even more mysteriously if we use (add1) LH grumbles... I expect the core is different? Any idea @nikivazou ?

{-@ reflect bumpFail @-}
bumpFail :: [Int] -> [Int]
bumpFail xs = mmap (`add` 1) xs

{-@ testFail :: _ -> {b:_ | b} @-}
testFail :: () -> Bool
testFail () = (bumpFail [0, 1]) == [1, 2]

OTOH, @wenkokke -- can you see if you have any luck replacing the (rdivs) with (rdiv' s) defined below?

{-@ reflect rdiv' @-}
rdiv' s x = rdiv x s

clearly undesirable and will be fixed, but may provide some progress in the short term?

wenkokke commented 4 years ago

@wenkokke can you see if you have any luck replacing the (rdiv s) with (rdiv' s) defined below?

{-@ reflect rdiv' @-}
rdiv' s x = rdiv x s

clearly undesirable and will be fixed, but may provide some progress in the short term?

I've tried replacing rdiv with rdiv', both defined as above and using (/) directly. Unfortunately, neither solves the problem.

ranjitjhala commented 4 years ago

Hmm can you remind me how you’re running LH to reproduce this? Will try to take a look today. Thanks!

On Tue, Feb 4, 2020 at 6:31 AM Wen Kokke notifications@github.com wrote:

@wenkokke https://github.com/wenkokke -- can you see if you have any luck replacing the (rdiv s) with (rdiv' s) defined below?

{-@ reflect rdiv' @-} rdiv' s x = rdiv x s

clearly undesirable and will be fixed, but may provide some progress in the short term?

I've tried replacing rdiv with rdiv', defined as you mentioned, and defined using (/) directly. Unfortunately, this does not solve the problem.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/ucsd-progsys/liquidhaskell/issues/1601?email_source=notifications&email_token=AAMS4OBWXARQBQWCIJGNVADRBF327A5CNFSM4KPIKKC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKX2LWQ#issuecomment-581936602, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMS4OH63TVJDYICO5KO3FLRBF327ANCNFSM4KPIKKCQ .

wenkokke commented 4 years ago

I'm running make test, which runs:

stack exec -- liquid -isrc test/*.hs
stack test --flag leadbeater:liquidhaskell

It already fails on the first command, so my cabal setup isn't super relevant.

ranjitjhala commented 4 years ago

Hi @wenkokke -- when I do a fresh clone of your repo and run make test things seem to work out fine (see attached screenshot) -- maybe you've edited it to sidestep the issue? Could you point me to a commit that shows the problem? Thanks!

Screen Shot 2020-02-04 at 12 17 07 PM
wenkokke commented 4 years ago

Ah, darn, I'm sorry. My master branch has the offending reflect pragmas commented out, since I'm continuing work on that.

wenkokke commented 4 years ago

I've created the issue-1601 branch for you, with a version which reproduces the reported error. Thanks so much for looking into this!

ranjitjhala commented 4 years ago

Thanks @wenkokke !

Hmm.. @nikivazou -- I think there is something fishy going on here that may have to do with your recent changes for function equality? Specifically, given the code:

{-@ reflect norm @-}
{-@ norm :: bar:VectorNE Rpos -> VectorX R bar @-}
norm :: Vector R -> Vector R
norm foo = map (plus (sumPos foo)) foo

PLE generates the odd equality:

      && Leadbeater.Network.norm == Leadbeater.LinearAlgebra.map (Leadbeater.Prelude.plus (Leadbeater.LinearAlgebra.sumPos bar))

where indeed bar is out of scope. The above looks like some malformed version of the term:

Leadbeater.Network.norm bar == Leadbeater.LinearAlgebra.map (Leadbeater.Prelude.plus (Leadbeater.LinearAlgebra.sumPos bar)) bar

but where you have somehow eta-contracted (is that a word?) by removing the bar (which is the name for the parameter in the reflected function.)

Does this sound like it may have to do with your recent fun-extensionality related edits?

ranjitjhala commented 4 years ago

Indeed, looking at the .fq I see:

define Leadbeater.Network.norm (bar : (Leadbeater.LinearAlgebra.Vector real)) = (&& [((Leadbeater.Network.norm bar) = (Leadbeater.LinearAlgebra.map (Leadbeater.Prelude.plus (Leadbea
ter.LinearAlgebra.sumPos bar)) bar))])

So I'm wondering where this mysterious Leadbeater.Network.norm == Leadbeater.LinearAlgebra.map (Leadbeater.Prelude.plus (Leadbeater.LinearAlgebra.sumPos bar)) term is coming from ...???

ranjitjhala commented 4 years ago

Aha. @nikivazou -- it is most certainly related to that because when I comment this line out

https://github.com/ucsd-progsys/liquid-fixpoint/blob/develop/src/Language/Fixpoint/Solver/Sanitize.hs#L52

@wenkokke's code works fine...

I am reverting your change right now, to get Wen unstuck I think you may want to use their example to debug your eta-elim?

ranjitjhala commented 4 years ago

Hi @wenkokke -- I made a branch

https://github.com/ucsd-progsys/liquidhaskell/tree/T1601

where your particular problem appears solved; will merge into develop soon pending CI and perhaps some deprecating of tests that depend on the suppressed feature...

Can you LMK if it works?

Best!

ranjitjhala commented 4 years ago

Hi @wenkokke -- this should be fixed in develop please LMK if not!

@nikivazou -- see the above bug in the elimEta the diff in the above PR #1602 should be pretty explanatory?

Thanks!

wenkokke commented 4 years ago

I've just tried, and everything works now! Thank you so much for all your effort!

nikivazou commented 4 years ago

Actually, elimEta on PLE is not mine ;)

@yiyunliu if you have time, can you take a look? In short, the elimEta of fixpoint crashes the code above. It is not urgent since it is flagged now, but if you can take a look at this before me, it would be great!

Let me open the issue again, in hope of providing a proper fix to elimEta.

yiyunliu commented 4 years ago

I can see what's going on here. elimEta takes the function:

norm2 xs = map (`rdiv` sumPos xs) xs

then naively turns it into:

norm2 = map (`rdiv` sumPos xs)

I'll add a check to see if the binder is free so it won't mistakenly get rid of the binder when it's needed.

nikivazou commented 4 years ago

oh thanks!

facundominguez commented 4 months ago

It's is possible to build this repo with the latest LH by

git clone https://github.com/wenkokke/lazuli.gitgit clone -b issue-1601 
cabal exec ghc -- -ilazuli/src -package liquidhaskell lazuli/src/Leadbeater/Network.hs -fplugin=LiquidHaskell

I'm still getting strange failures when reflecting.

lazuli/src/Leadbeater/LinearAlgebra.hs:210:13: error:
    Illegal type specification for `Leadbeater.LinearAlgebra.mXv`
    Leadbeater.LinearAlgebra.mXv :: xss:(Leadbeater.LinearAlgebra.Matrix GHC.Types.Double) -> ys:{VV##0 : (Leadbeater.LinearAlgebra.Vector GHC.Types.Double) | Leadbeater.LinearAlgebra.size VV##0 == Leadbeater.LinearAlgebra.cols xss} -> {VV##0 : (Leadbeater.LinearAlgebra.Vector GHC.Types.Double) | VV##0 == Leadbeater.LinearAlgebra.flatten (xss < Leadbeater.LinearAlgebra.asColumn ys)
                                                                                                                                                                                                                                                                                                          && VV##0 == Leadbeater.LinearAlgebra.mXv xss ys
                                                                                                                                                                                                                                                                                                          && Leadbeater.LinearAlgebra.size VV##0 == Leadbeater.LinearAlgebra.rows xss}
    Sort Error in Refinement: {VV##0 : (Leadbeater.LinearAlgebra.Vector real) | VV##0 == Leadbeater.LinearAlgebra.flatten (xss < Leadbeater.LinearAlgebra.asColumn ys)
                                                                                && VV##0 == Leadbeater.LinearAlgebra.mXv xss ys
                                                                                && Leadbeater.LinearAlgebra.size VV##0 == Leadbeater.LinearAlgebra.rows xss}
    Cannot unify (Leadbeater.LinearAlgebra.Matrix @(11572)) with bool in expression: Leadbeater.LinearAlgebra.flatten (xss < Leadbeater.LinearAlgebra.asColumn ys) 
    Just reflect
    |
210 | {-@ reflect mXv @-}
    |             ^^^^

lazuli/src/Leadbeater/LinearAlgebra.hs:220:13: error:
    Illegal type specification for `Leadbeater.LinearAlgebra.outer`
    Leadbeater.LinearAlgebra.outer :: xs:(Leadbeater.LinearAlgebra.Vector GHC.Types.Double) -> ys:(Leadbeater.LinearAlgebra.Vector GHC.Types.Double) -> {VV##0 : (Leadbeater.LinearAlgebra.Matrix GHC.Types.Double) | VV##0 == Leadbeater.LinearAlgebra.outer xs ys
                                                                                                                                                                                                                      && VV##0 == (Leadbeater.LinearAlgebra.asColumn xs < Leadbeater.LinearAlgebra.asRow ys)
                                                                                                                                                                                                                      && Leadbeater.LinearAlgebra.cols VV##0 == Leadbeater.LinearAlgebra.size ys
                                                                                                                                                                                                                      && Leadbeater.LinearAlgebra.rows VV##0 == Leadbeater.LinearAlgebra.size xs}
    Sort Error in Refinement: {VV##0 : (Leadbeater.LinearAlgebra.Matrix real) | VV##0 == Leadbeater.LinearAlgebra.outer xs ys
                                                                                && VV##0 == (Leadbeater.LinearAlgebra.asColumn xs < Leadbeater.LinearAlgebra.asRow ys)
                                                                                && Leadbeater.LinearAlgebra.cols VV##0 == Leadbeater.LinearAlgebra.size ys
                                                                                && Leadbeater.LinearAlgebra.rows VV##0 == Leadbeater.LinearAlgebra.size xs}
    Cannot unify (Leadbeater.LinearAlgebra.Matrix real) with bool in expression: VV##0 == (Leadbeater.LinearAlgebra.asColumn xs < Leadbeater.LinearAlgebra.asRow ys) 
  because
Invalid Relation VV##0 == (Leadbeater.LinearAlgebra.asColumn xs < Leadbeater.LinearAlgebra.asRow ys) with operand types (Leadbeater.LinearAlgebra.Matrix real) and bool
    Just reflect
    |
220 | {-@ reflect outer @-}
    |             ^^^^^^