WIP attempt to support non-Float pipelines

[deepfire]

Same caveat as in https://github.com/lambdacube3d/lambdacube-gl/pull/13 -- this doesn't work yet, so please consider this a research proto-PoC.

[csabahruska]

If you'd like to add color buffer format control, then this is not the way to go. Can you describe the problem/feature that you'd like to solve?

[deepfire]

@csabahruska, in the end I wanted to implement picking -- so I'd need to be able to render the same scene with integer IDs of the objects rendered, instead of the texture pixels.

So I got that working, except I have to convert integers to floats and back, currently.

And this is sort-of doable (I have to turn off SRGB, since it breaks the Float<->Int correspondence linearity), but it's certainly not a pretty way.

[csabahruska]

The IR should be expressive enough for that out of the box. Maybe even the front-end language is sufficient also. I need some time to check it.

[deepfire]

My understanding was that to write integers to the framebuffer in the fragment shader, the framebuffer color component type must be integer as well (so the types unify), and http://lambdacube3d.com/overview says:

A color image is to store color values. Supported color values are Float, and Float vectors with dimension at most 4.

[deepfire]

Also, there is another hint of trouble I was researching -- https://www.khronos.org/opengl/wiki/Framebuffer says:

Note that glClearColor takes floating-point values. Yet it is perfectly legal to use integer Image Formats for images in framebuffers. Attempting to clear integer buffers with floating-point data will not work.

So this at least suggests the possibility that a fragment shader with an integer output cannot be drawing onto a framebuffer with a floating image format. And since glClearColor/glClear is what lambdacube-gl uses to clear the color buffers, I was concluding that it sets up a them with a float image format.

[csabahruska]

I see. I think that should be written as an example in the overview. The more accurate primop and constraint description is specified in Builtins.lc. The interesting part of the pipeline is where the blending is specified. Each color component type have different blending and interpolation constraints. I.e. Images with Int color component can not be interpolated and must use Flat interpolation.

data Blending :: Type -> Type where
  NoBlending          ::                                   Blending t
  BlendLogicOp        :: (Integral t) => LogicOperation -> Blending t
  Blend               :: (BlendEquation, BlendEquation)
                         -> ((BlendingFactor, BlendingFactor), (BlendingFactor, BlendingFactor))
                         -> Vec 4 Float ->                 Blending Float

data Interpolated t where
  Smooth, NoPerspective
                      :: (Floating t) => Interpolated t
  Flat                ::                 Interpolated t

The ColorImage constructor allows any Num (Int/Word/Float) comonent types.

ColorImage          :: forall a d t color . (Num t, color ~ VecScalar d t)
                    => color  -> Image a (Color color)

[csabahruska]

OpenGL has a distinction between image semantic and internal format. The semantic is relevant from the shader perspective the internal format is relevant from data storage and memory prespective. I.e It is valid to have Float semantic on textures with Float or Integer internal format. But every texture with Integer semantic must have Integer internal format. So the semantic and internal format can be confusing first, but they are different things.

[csabahruska]

From lambdacube language you can control the texture semantic. The internal format is baked in the implementation at the moment. But that should not be a problem, because it only affects the allocated texture size in bytes.

[deepfire]

Curiously, the type system made me figure out that Flat interpolation is necessary for the Integer image.

So what I have are two pipelines that are exremely similar:

1. float-typed: https://github.com/deepfire/holotype/blob/master/lc/PipePickF.lc
2. integer-typed: https://github.com/deepfire/holotype/blob/master/lc/PipePickU.lc

The diff literally is:

$ diff -uN lc/PipePick*
--- lc/PipePickF.lc     2018-09-12 22:08:00.752517145 +0300
+++ lc/PipePickU.lc     2018-09-12 22:09:36.072090972 +0300
@@ -1,6 +1,6 @@
 module Holotype where

-type ColorComponent = Float
+type ColorComponent = Int

 mkaccumulationContext :: (FragmentOperation Depth, FragmentOperation (Color (Vec 4 ColorComponent)))
 mkaccumulationContext = ( DepthOp Always False
@@ -13,7 +13,7 @@
 fragmap :: (Vec 2 Float, Vec 4 ColorComponent) -> ((Vec 4 ColorComponent))
 fragmap (uv, rgba) =
   -- ((V4 255 127 63 255))
-  ((V4 0.9 0.5 0.1 1))
+  ((V4 127 127 1 127))
   -- ((V4 16777215 16777215 16777215 16777215))
   -- ((rgba))

@@ -32,7 +32,7 @@
     (\(pos, uv, id)->
       ( (Uniform "viewProj" :: Mat 4 4 Float) *. (V4 pos%x pos%y 0 1)
       , uv
-      , V4 1 1 1 1))
+      , V4 127 127 1 127))
     $ fetch name ( Attribute "position"   :: Vec 3 Float
                  , Attribute "uv"         :: Vec 2 Float
                  , Attribute "id"         :: Int))
@@ -45,7 +45,7 @@
 main :: Output
 main = ScreenOut $
        scene "portStream" $
-       FrameBuffer (depthImage1 1000, colorImage1 (V4 0.5 0.5 0 0.5))
+       FrameBuffer (depthImage1 1000, colorImage1 (V4 0x7f 0x7f 0 0x7f))

 -- Local Variables:
 -- eval: (progn (haskell-mode) (intero-disable))

These pipelines are fed to Lambdacube.GL.renderFrame one after another, with a call to a slightly modified version of getFrameBuffer from https://github.com/lambdacube3d/lambdacube-gl/blob/master/testclient/client.hs#L132 to read back the pixel values after each one.

The reads come back as follows

• glClear-ed area: both Int and Float pipelines produces values as expected (my theory is because the same float glClearColor/glClear is used in both cases)
• glTriangle-affected area: Float as expected, Int pipeline produces zero values.

[csabahruska]

The PipePickU.lc has the right description for the pipeline. But it seems the compiler and maybe the gl backend will need some fix.

[csabahruska]

It seems that in lambdacube-gl glClearBuffer should be used instead of glClearColor in clearRenderTarget function.

Also lambdacube-compiler has to be fixed. The compValue in CoreToIR.hs must be extended:

diff --git a/src/LambdaCube/Compiler/CoreToIR.hs b/src/LambdaCube/Compiler/CoreToIR.hs
index 4b75c18..fd15562 100644
--- a/src/LambdaCube/Compiler/CoreToIR.hs
+++ b/src/LambdaCube/Compiler/CoreToIR.hs
@@ -437,6 +437,9 @@ compValue x = case x of
   A2 "V2" (EBool a) (EBool b) -> IR.VV2B $ IR.V2 a b
   A3 "V3" (EBool a) (EBool b) (EBool c) -> IR.VV3B $ IR.V3 a b c
   A4 "V4" (EBool a) (EBool b) (EBool c) (EBool d) -> IR.VV4B $ IR.V4 a b c d
+  A2 "V2" (EInt a) (EInt b) -> IR.VV2I $ IR.V2 (fromIntegral a) (fromIntegral b)
+  A3 "V3" (EInt a) (EInt b) (EInt c) -> IR.VV3I $ IR.V3 (fromIntegral a) (fromIntegral b) (fromIntegral c)
+  A4 "V4" (EInt a) (EInt b) (EInt c) (EInt d) -> IR.VV4I $ IR.V4 (fromIntegral a) (fromIntegral b) (fromIntegral c) (fromIntegral d)
   x -> error $ "compValue " ++ ppShow x

[csabahruska]

It can be useful to observe the compiled pipeline:

stack exec -- lc PipePickU.lc
stack exec -- lc pretty PipePickU.json

It will create the PipePickU.ppl:

Pipeline
  { info = "generated by lambdacube-compiler 0.6.1.0"
  , backend = OpenGL33
  , textures = []
  , samplers = []
  , targets =
      [ RenderTarget
          { renderTargets =
              [ TargetItem
                  { targetSemantic = Depth , targetRef = Just (Framebuffer Depth) }
              , TargetItem
                  { targetSemantic = Color , targetRef = Just (Framebuffer Color) }
              ]
          }
      ]
  , programs =
      [ Program
          { programUniforms = fromList [ ( "viewProj" , M44F ) ]
          , programStreams =
              fromList
                [ ( "vi1" , Parameter { name = "position" , ty = V3F } )
                , ( "vi2" , Parameter { name = "uv" , ty = V2F } )
                , ( "vi3" , Parameter { name = "id" , ty = Int } )
                ]
          , programInTextures = fromList []
          , programOutput = [ Parameter { name = "f0" , ty = V4F } ]
          , vertexShader =
              """
              #version 330 core
              vec4 texture2D(sampler2D s,vec2 uv) {
                  return texture(s,uv);
              }
              uniform mat4 viewProj;
              in vec3 vi1;
              in vec2 vi2;
              in int vi3;
              flat out vec2 vo1;
              flat out ivec4 vo2;
              void main() {
                  gl_Position = (viewProj) * (vec4 ((vi1).x,(vi1).y,0.0,1.0));
                  vo1 = vi2;
                  vo2 = ivec4 (127,127,1,127);
              }
              """
          , geometryShader = Nothing
          , fragmentShader =
              """
              #version 330 core
              vec4 texture2D(sampler2D s,vec2 uv) {
                  return texture(s,uv);
              }
              flat in vec2 vo1;
              flat in ivec4 vo2;
              out ivec4 f0;
              void main() {
                  f0 = ivec4 (127,127,1,127);
              }
              """
          }
      ]
  , slots =
      [ Slot
          { slotName = "portStream"
          , slotStreams =
              fromList [ ( "id" , Int ) , ( "position" , V3F ) , ( "uv" , V2F ) ]
          , slotUniforms = fromList [ ( "viewProj" , M44F ) ]
          , slotPrimitive = Triangles
          , slotPrograms = [ 0 ]
          }
      ]
  , streams = []
  , commands =
      [ SetRenderTarget 0
      , ClearRenderTarget
          [ ClearImage { imageSemantic = Depth , clearValue = VFloat 1000.0 }
          , ClearImage
              { imageSemantic = Color , clearValue = VV4I (V4 127 127 0 127) }
          ]
      , SetProgram 0
      , SetRasterContext
          (TriangleCtx (CullFront CCW) PolygonFill NoOffset LastVertex)
      , SetAccumulationContext
          AccumulationContext
            { accViewportName = Nothing
            , accOperations =
                [ DepthOp Always False
                , ColorOp NoBlending (VV4B (V4 True True True True))
                ]
            }
      , RenderSlot 0
      ]
  }

[deepfire-pusher]

I was using aeson-pretty to do something very similar, indeed.. (And actually made a change in this PR to this end: https://github.com/lambdacube3d/lambdacube-compiler/pull/14/files#diff-0a0f088140eb06142b5f04d14aad102aR77)

Thank you for pointing out a simpler way!

[csabahruska]

OpenGL has a default back buffer for the screen named GL_BACK_LEFT. It must be filled with Float semantic. That's why PipePickF.lc works but produces Float ids instead of Int. But PipePickU.lc is a correct pipeline description also that produces Int ids for triangles. It's lambdacube-compiler that needs to be fixed. The issue is that the default back buffer must not be used for render targets with Int semantic. Instead a render texture must be allocated for that. Then you have to lookup a pixel in the render texture. Probably lambdacube-gl API needs some extension for that.

[deepfire-pusher]

Thank you a lot for the explanation, Csaba!

[csabahruska]

Do you plan to make these changes? I think only CoreToIR.hs need some changes.

[deepfire]

Csaba, I filed https://github.com/lambdacube3d/lambdacube-compiler/pull/15, to allow PipePickU.lc to be compilable -- indeed a minimum change.

But also, would you like separate pull requests for minor things like dropping Cabal upper bounds?

Perhaps this pull request now should be closed.

[csabahruska]

If you use stack then the bound check can be disabled easily, just add the following line to stack.yaml:

allow-newer: true

e.g. https://github.com/grin-tech/grin/blob/master/stack.yaml#L4

[csabahruska]

Despite that PipePickU.lc will compile now, the generated pipeline IR will not do the right thing, because it still tries to use the default back buffer (which supports only the Float semantic) to store the render result. I believe the right direction would be to add support for the render texture pipeline output. Right now only ScreenOut is supported (see Builtins.lc):

data Output where
  ScreenOut           :: FrameBuffer a b -> Output

[csabahruska]

If you'd like to work on this issue, then we can create a gitter chat to discuss the details.

[deepfire-pusher]

Csaba, I PM-ed you on Gitter.

[deepfire-pusher]

This PR is considered taken over by https://github.com/lambdacube3d/lambdacube-compiler/pull/15, closing.