Support for sequence tagging tasks

[Khalid-Nabigh]

We are trying to evaluate Named Entity Recognition and Part of Speech tagging tasks, but it is unclear to us how to do that. We've noticed that aclue include a Named Entity Recognition task but it is treated as a multiple choice not a sequence tagging task.

[haileyschoelkopf]

Hi!

If I'm remembering right, the question of supporting sequence labeling tasks has come up before--The primary reason we haven't added any is that we're focusing on prompted evaluation of autoregressive LMs, and I'm not aware of a popular + effective framing of NER or POS tagging tasks as a prompted evaluation.

Maybe there have been proposals recently for this though that have gotten traction, if so we'd be happy to accept additions of such tasks!

[areias]

Hello, I've been working on using llms for NLP tasks like NER and relationship extraction and would be interested in adding these tasks and datasets to the harness. You can see here several papers and datasets on this:

• Qin, C., Zhang, A., Zhang, Z., Chen, J., Yasunaga, M., & Yang, D. (2023). Is ChatGPT a General-Purpose Natural Language Processing Task Solver? arXiv preprint arXiv:2302.06476.
• Wei, X., Cui, X., Cheng, N., Wang, X., Zhang, X., Huang, S., Xie:, P., Xu, J., Chen, Y., Zhang, M., Jiang, Y., & Han, W. (2023a). Zero-Shot Information Extraction via Chatting with ChatGPT. arXiv preprint arXiv:2302.10205.
• Xu, X., Zhu, Y., Wang, X., & Zhang, N. (2023). How to Unleash the Power of Large Language Models for Few-shot Relation Extraction? arXiv preprint arXiv:2305.01555.

What do you think?