Ambiguous DCN layer

[hkristof03]

I would like to ask for some help regarding stacked Cross layers.

From this tutorial, the calling of a simple DCN layer looks like this:

if use_cross_layer:
      self._cross_layer = tfrs.layers.dcn.Cross(
          projection_dim=projection_dim,
          kernel_initializer="glorot_uniform")

......

x = tf.concat(embeddings, axis=1)

# Build Cross Network
if self._cross_layer is not None:
    x = self._cross_layer(x)

From the documentation of the DCN Cross layer:

# after embedding layer in a functional model:
input = tf.keras.Input(shape=(None,), name='index', dtype=tf.int64)
x0 = tf.keras.layers.Embedding(input_dim=32, output_dim=6)
x1 = Cross()(x0, x0)
x2 = Cross()(x0, x1)
logits = tf.keras.layers.Dense(units=10)(x2)
model = tf.keras.Model(input, logits)

I really don't get how it is possible to properly call the Cross layer first with one variable then with two. The second looks more reasonable for me according to the DCN paper and the tutorial.

What I would like to do is to stack multiple Cross layers and properly feed them with the data from the embedding layers.

self.cross_layers = []

for i in range(n_cross_layers):
    self.cross_layers.append(tfrs.layers.dcn.Cross(
        projection_dim=projection_dim,
        #kernel_regularizer='glorot_uniform'
    ))

def call(self, inputs: tf.Tensor, training: bool = False) -> tf.Tensor:

    embedding = tf.concat(
        [self.embeddings[key](inputs[key])
         for key in self.embedding_features],
        axis=1
    )
    embedding = tf.concat([
        embedding,
        tf.stack([inputs[feature] for feature in self.features], axis=1),
    ], axis=1)

    if self.cross_layers:
        x1 = self.cross_layers[0](embedding, embedding)

        for cross_layer in self.cross_layers[1:]:
            x1 = cross_layer(embedding, x1)

        embedding = x1

    embedding = self.dense_layers(embedding, training)

    return embedding

[maciejkula]

The example you wrote out looks correct - what do you think is missing?

[hkristof03]

@maciejkula Thanks for your response.

I checked the documentation of the Cross layer now:

def call(self, x0: tf.Tensor, x: Optional[tf.Tensor] = None) -> tf.Tensor:
    """Computes the feature cross.
    Args:
      x0: The input tensor
      x: Optional second input tensor. If provided, the layer will compute
        crosses between x0 and x; if not provided, the layer will compute
        crosses between x0 and itself.
    Returns:
     Tensor of crosses.
    """

From this it is obvious what happens when only one parameter is passed. The question's objective was to validate if I correctly implemented the stacking of the Cross layers.

[patrickorlando]

@hkristof03, Your implementation of the Cross block is correct.

The output of layer x_(l+1) of a Cross stack is the output of the previous layer, x_l, crossed with x_0.

In the DCN tutorial, there is no layer stacking, which means we simply cross x_0 with itself. This is why the layer can be called with a single argument. In the stacked case you need to do as you have done.

From the DCN Paper