search

huggingface / transformers

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
https://huggingface.co/transformers
Apache License 2.0
132.92k stars 26.52k forks source link

Sequential constraints? #11180

Open kcarnold opened 3 years ago

kcarnold commented 3 years ago

fairseq has an implementation of unordered and ordered multi-token constraints: see their PR and example.

This is more advanced than the single-token constraints that have been occasionally requested here mainly due to the bookkeeping involved; see the papers referenced in the fairseq PR.

Has anyone looked into porting this feature? fairseq's constraint tracking logic looks to be well-factored and could probably be adopted verbatim, license permitting. The beam search modifications (fairseq implementation) may be able to be implemented as a LogitsProcessor, or maybe even just a prefix_allowed_tokens_fn, but the papers propose some additional logic around making sure that a partial constraint stays in the beam; I'm not sure whether those hooks are sufficient to implement that logic (or how essential it is to the functionality).

(I've found one related issue.)

blodstone commented 3 years ago

Has anyone working on this yet?

github-actions[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

kcarnold commented 3 years ago

If no one else beats me to it, I may tackle this mid-summer. No promises.

ashim95 commented 3 years ago

@kcarnold any update?

There is another recent paper on constrained decoding with complex constraints (fairseq only has positive constraints): NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints