Error duing forward pass: Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got torch.FloatTensor instead (while checking arguments for embedding)

[kyegomez]

[2023-08-08 03:29:00,661] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["id2label"]` will be overriden.
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["bos_token_id"]` will be overriden.
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["eos_token_id"]` will be overriden.
Non-A100 GPU detected, using math or mem efficient attention if input tensor is on cuda
torch.Size([1, 257, 1024])
torch.Size([1, 64, 1024])
torch.Size([1, 64, 50304])
torch.Size([1, 50, 50304])
torch.Size([1, 114, 50304])
Error duing forward pass: Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got torch.FloatTensor instead (while checking arguments for embedding)
Output: None

• create a forward embedding function and integrate it into the main class

[kyegomez]

• from Kosmos

  def forward_embedding(
      self,
      tokens,
      token_embedding=None,
      incremental_state=None,
  ):
      if incremental_state is not None and not self.is_first_step(incremental_state):
          tokens = tokens[:, -1:]
  
      if token_embedding is None:
          token_embedding = self.embed_tokens(tokens)
  
      x = embed = self.embed_scale * token_embedding
  
      if self.layernorm_embedding is not None:
          x = self.layernorm_embedding(x)
  
      x = self.dropout_module(x)
  
      return x, embed
  
  def is_first_step(self, incremental_state):
      if incremental_state is None:
          return False
      return incremental_state.get("is_first_step", False)
  
  def forward(
      self,
      prev_output_tokens,
      incremental_state=None,
      features_only=False,
      return_all_hiddens=False,
      token_embeddings=None,
      **kwargs
  ):
      # embed tokens
      x, _ = self.forward_embedding(
          prev_output_tokens, token_embeddings, incremental_state
      )
      is_first_step = self.is_first_step(incremental_state)
  
      if self.chunkwise_recurrent and prev_output_tokens.size(1) % self.recurrent_chunk_size != 0:
          padding_len = self.recurrent_chunk_size - prev_output_tokens.size(1) % self.recurrent_chunk_size
          slen = prev_output_tokens.size(1) + padding_len
          x = F.pad(x, (0, 0, 0, padding_len))
      else:
          slen = prev_output_tokens.size(1)
      # relative position
      retention_rel_pos = self.retnet_rel_pos(slen, incremental_state is not None and not is_first_step, chunkwise_recurrent=self.chunkwise_recurrent)
  
      # decoder layers
      inner_states = [x]
  
      l_aux = []
  
      for idx, layer in enumerate(self.layers):
          if incremental_state is None or is_first_step:
              if is_first_step and incremental_state is not None:
                  if idx not in incremental_state:
                      incremental_state[idx] = {}
          else:
              if idx not in incremental_state:
                  incremental_state[idx] = {}
  
          x, l_aux_i = layer(
              x,
              incremental_state[idx] if incremental_state is not None else None,
              retention_rel_pos=retention_rel_pos,
              chunkwise_recurrent=self.chunkwise_recurrent,
          )
          l_aux.append(l_aux_i)
          inner_states.append(x)
  
      if self.chunkwise_recurrent and prev_output_tokens.size(1) % self.recurrent_chunk_size != 0:
          x = x[:, :prev_output_tokens.size(1), :]
  
      if self.layer_norm is not None:
          x = self.layer_norm(x)
  
      if not features_only:
          x = self.output_layer(x)
  
      return x, {
          "inner_states": inner_states,
          "l_aux": l_aux,
          "attn": None,
      }
  
  def output_layer(self, features):
      return self.output_projection(features)