Implementing Dropout in Haskell Interface

As an exercise to test my understanding, I've been trying to implement a simple dropout operation in the Haskell interface to Tensorflow. Following the example https://github.com/tensorflow/haskell/blob/master/tensorflow-mnist/app/Main.hs, I tried defining a function

-- | Create random bitmask
randomMask :: Int64 -> TF.Shape -> TF.Build (TF.Tensor TF.Value Float)
randomMask (TF.Shape shape) =
    TF.round (TF.randomUniform (TF.vector shape))

to sample a random bitmask. However, I'm getting the error

    /home/rbharath/tensorflow-haskell/tensorflow-mnist/app/MainWithDropout.hs:49:5:
        Not in scope: ‘TF.round’
        Perhaps you meant one of these:
          ‘TF.run’ (imported from TensorFlow.Core),
          ‘TF.run_’ (imported from TensorFlow.Core)

    /home/rbharath/tensorflow-haskell/tensorflow-mnist/app/MainWithDropout.hs:49:15: Not in scope: ‘TF.randomUniform’

Tensorflow's python API exports tf.round and tf.random_uniform. The existing MNIST example uses TF.truncateNormal as well. What would be the best way for me to surface round and random_uniform from the underlying tensorflow? (Or is my approach here wrong entirely? I don't understand this codebase well, so any general pointers much appreciated :-))

You can import them from TensorFlow.GenOps.Core from the tensorflow-core-ops package; for example: https://tensorflow.github.io/haskell/haddock/tensorflow-core-ops-0.1.0.0/TensorFlow-GenOps-Core.html#v:round

The full Haddocks are here: https://tensorflow.github.io/haskell/haddock

So far the TensorFlow.Ops module only reexports a subset of TensorFlow.Core.Ops. I'm not sure if we have a story around how the two will relate long-term, but both of the functions you mentioned seem reasonable to add to the reexport list.

Thanks for the helpful response! This got me past the first roadblock. I'm now trying to re-export randomUniform from TensorFlow.Core.Ops to TensorFlow.Ops. I followed the template from truncateNormal since these have the same signature in the haddock (see here, here)

-- Random tensor from the unit normal distribution with bounded values.
truncatedNormal :: forall a v . TensorType a
                => Tensor v Int64  -- ^ Shape.
                -> Build (Tensor Value a)
truncatedNormal = buildOp $ opDef "TruncatedNormal"
                          & opAttr "dtype" .~ tensorType (undefined :: a)
                          & opAttr "T" .~ tensorType (undefined :: Int64)

-- Random tensor from uniform distribution
randomUniform :: forall a v . TensorType a
              => Tensor v Int64 -- ^ Shape.
              -> Build (Tensor Value a)
randomUniform = buildOp $ opDef "RandomUniform"
                        & opAttr "dtype" .~ tensorType (undefined :: a)
                        & opAttr "T" .~ tensorType (undefined :: Int64)

When I go to compile however, GHC complains:

    /home/rbharath/tensorflow-haskell/tensorflow-ops/src/TensorFlow/Ops.hs:289:9:
        Could not deduce (t /= GHC.Int.Int8,
                          t /= GHC.Int.Int16,
                          t /= GHC.Word.Word8,
                          t /= ByteString,
                          t /= Bool)
          arising from a use of ‘CoreOps.round’
        from the context (TensorType t)
          bound by the type signature for
                     TensorFlow.Ops.round :: TensorType t =>
                                             Tensor v1 t -> Tensor Value t
          at src/TensorFlow/Ops.hs:288:10-56
        Relevant bindings include
          round :: Tensor v1 t -> Tensor Value t
            (bound at src/TensorFlow/Ops.hs:289:1)
        In the expression: CoreOps.round
        In an equation for ‘round’: round = CoreOps.round

My apologies if this is an obvious Haskell error. I'm new to Haskell development, so nothing leaped out as obviously wrong. Any pointers much appreciated :-)

You can do something simpler: since you're just reexporting the functions without changing them, you can just list them in the export list directly as CoreOps.round, etc., without defining a local version, similar to the treatment of CoreOps.add: https://github.com/tensorflow/haskell/blob/master/tensorflow-ops/src/TensorFlow/Ops.hs#L60

For the error message you ran into: you can see the signature of CoreOps.round here: https://tensorflow.github.io/haskell/haddock/tensorflow-core-ops-0.1.0.0/TensorFlow-GenOps-Core.html#v:round The type signature you've written is missing the OneOf constraint, which is documented here: https://tensorflow.github.io/haskell/haddock/tensorflow-0.1.0.0/TensorFlow-Types.html#t:OneOf

Thanks for the explanations again! I think I'm now comfortable with using the CoreOps, and I've managed to implement a simple dropout function in Tensorflow.NN. I'm trying to put together a reasonable test for it, but the types are defeating me:

NN.hs:

-- | Computes dropout on neural network
dropout :: (OneOf '[Double, Float] a, TensorType a, Num a)
        => Tensor Value a  -- ^ __x__
        -> a               -- ^__keepProb__
        -> Build (Tensor Value a)
dropout x keepProb = do
    x' <- render x 
    keepProb' <- render $ TF.scalar keepProb
    withNameScope "dropout" $ do
        randUnif <- TF.randomUniform (TF.shape x') 
        randomTensor <- render $ keepProb' `TF.add` randUnif
        binaryTensor <- render $ TF.floor randomTensor
        render $ (x' `TF.div` keepProb') `TF.mul` binaryTensor

NNTest.hs:

-- Adapated from https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/nn_test.py
-- Runs dropout with 0-1 tensor 10 times, sum the number of ones and validate
-- that it is producing the right number of ones over a large number of
-- samples, based on the keep prob
testDropout :: Test
testDropout = testCase "testDropout" $ do
    let xDim = 40
        yDim = 30  
        shape = [xDim, yDim]
        keepProbs = [0.1, 0.5, 0.8] :: [Float]
        tfDropped = TF.dropout (TF.ones shape) (0.5 :: Float)
        numOnes = TF.sum tfDropped [0, 1]
    r <- run $ TF.render $ TF.sum tfDropped [0, 1]
    assertAllClose (V.fromList [r]) (V.fromList [600.0])

The test in its current form isn't complete, but I'm not able to make it compile yet. I'm getting type complaints:

    Preprocessing test suite 'NNTest' for tensorflow-nn-0.1.0.0...
    [1 of 1] Compiling Main             ( tests/NNTest.hs, .stack-work/dist/x86_64-linux-dkd1ce2ff9c9560b648268df668d177711/Cabal-1.22.5.0/build/NNTest/NNTest-tmp/Main.o )

    /home/rbharath/tensorflow-haskell/tensorflow-nn/tests/NNTest.hs:111:26:
        Couldn't match type ‘TF.BuildT
                               Data.Functor.Identity.Identity
                               (TensorFlow.Tensor.Tensor TensorFlow.Tensor.Value Float)’
                       with ‘TensorFlow.Tensor.Tensor v10 t’
        Expected type: TensorFlow.Tensor.Tensor v10 t
          Actual type: TF.Build
                         (TensorFlow.Tensor.Tensor TensorFlow.Tensor.Value Float)
        Relevant bindings include
          numOnes :: TensorFlow.Tensor.Tensor TensorFlow.Tensor.Value t
            (bound at tests/NNTest.hs:111:9)
        In the first argument of ‘TF.sum’, namely ‘tfDropped’
        In the expression: TF.sum tfDropped [0, 1]

    /home/rbharath/tensorflow-haskell/tensorflow-nn/tests/NNTest.hs:112:35:
        Couldn't match type ‘TF.BuildT
                               Data.Functor.Identity.Identity
                               (TensorFlow.Tensor.Tensor TensorFlow.Tensor.Value Float)’
                       with ‘TensorFlow.Tensor.Tensor v1 a0’
        Expected type: TensorFlow.Tensor.Tensor v1 a0
          Actual type: TF.Build
                         (TensorFlow.Tensor.Tensor TensorFlow.Tensor.Value Float)
        In the first argument of ‘TF.sum’, namely ‘tfDropped’
        In the second argument of ‘($)’, namely ‘TF.sum tfDropped [0, 1]’

It looks like tfDropped lives in the Build Monad, but TF.sum expects arguments not in the monad. I've been staring at this for a while, but it's defeating my weak Monad knowledge :). Do you have any hints on the right way to structure this test?

(warning: haven't tried to compile any of this)

Loosely speaking, you can use the >>= operator to work with values inside the monad

    let xDim = 40
        yDim = 30  
        shape = [xDim, yDim]
        tfDropped = TF.dropout (TF.ones shape) (0.5 :: Float)
        numOnes = tfDropped >>= \x -> TF.render (TF.sum x [0, 1])
    r <- run numOnes

Or, with do notation

    let xDim = 40
        yDim = 30  
        shape = [xDim, yDim]
    r <- run $ do
        tfDropped <- TF.dropout (TF.ones shape) (0.5 :: Float)
        TF.render (TF.sum tfDropped [0, 1])

tensorflow / haskell

Implementing Dropout in Haskell Interface #63