How to specific customized force_token_ids in whisper

[MonolithFoundation]

ValueError: A custom logits processor of type <class 'transformers.generation.logits_process.ForceTokensLogitsProcessor'> with values <transformers.generation.logits_process.ForceTokensLogitsProcessor object at 0x7f4230cfac50> has been passed to `.generate()`, but it has already been created with the values <transformers.generation.logits_process.ForceTokensLogitsProcessor object at 0x7f422829c510>. <transformers.generation.logits_process.ForceTokensLogitsProcessor object at 0x7f422829c510> has been created by passing the corresponding arguments to generate or by the model's config default values. If you just want to change the default values of logits processor consider passing them as arguments to `.generate()` instead of using a custom logits processor

this way don't work:

inputs = inputs.to(self.model.dtype)
        with torch.no_grad():
            if forced_decoder_ids is not None:
                generated_ids = self.model.generate(
                    inputs, forced_decoder_ids=forced_decoder_ids
                )
            else:
                generated_ids = self.model.generate(inputs)

[LysandreJik]

cc @gante/@ylacombe maybe

[gante]

cc @eustlb , since you're working on Whisper fixes :)