Camembert - Experiments in French

[PierreColombo]

Hello, Thanks a lot for your nice work on CamemBERT. Do you plan to release some code to help us to reproduce your results of the experiments describes in the paper (CamemBERT: a Tasty French Language Model). Thanks,

[huihuifan]

@louismartin

[louismartin]

Hi @PierreColombo The experiments were run using fairseq for pretraining and XNLI and HuggingFace's Transformers for the rest. Most of the code is already present in the two librairies.

[stefan-it]

Hi @louismartin , do you plan to add examples for fine-tuning a model for sequence tagging tasks (NER and Pos tagging) 🤔

I wasn't able to reproduce the results for PoS tagging with my extension of the fine-tuning NER code in 🤗/Transformers, so I'm just wondering if I missed something in the implementation.

(It's perfectly working with the feature-based approach in Flair btw 😅)

[benjamin-mlr]

Hi @stefan-it , we do plan to release the full code that will allow you to train and use our NER, POS Parser and NLI models. It will come in the next weeks. Meanwhile, can I ask you how you did it ? My first guess is that is comes from the Optimization : what Optimizer , learning rate, batch size , number of epochs did you use ? Thanks, Benjamin

[stefan-it]

Hi Benjamin,

I re-run the PoS tagging experiment with the latest version of 🤗/Transformers. I used the default parameters of the run_ner script and trained both camemBERT and multilingual BERT models on the ParTUT dataset for 30 epochs (only one run).

The results are consistent with the experiment on ParTUT done in the paper: camemBERT is ~0.24% better 🎉

Btw: do you plan to add support for camemBERT into the fairseq library or into pytext (like it is done for XLM-R) 🤔

Thanks for your help!

Stefan

[benjamin-mlr]

Hi Stefan,

Thanks for reproducing the experiments !

Yes we do plan to release it in faiseq at some point. It should come begining of January

Benjamin

On Tue, Dec 24, 2019 at 12:20 AM Stefan Schweter notifications@github.com wrote:

Hi Benjamin,

I re-run the PoS tagging experiment with the latest version of 🤗/Transformers. I used the default parameters of the run_ner script and trained both camemBERT and multilingual BERT models on the ParTUT dataset for 30 epochs (only one run).

The results are consistent with the experiment on ParTUT done in the paper: camemBERT is ~0.24% better 🎉

Btw: do you plan to add support for camemBERT into the fairseq library or into pytext (like it is done for XLM-R) 🤔

Thanks for your help!

Stefan

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pytorch/fairseq/issues/1450?email_source=notifications&email_token=AEHOJY2MOVKSM2NWZABN7MLQ2FBSJA5CNFSM4JUXIQV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHSDNCI#issuecomment-568604297, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEHOJY75FMWPMI7HVXNFEF3Q2FBSJANCNFSM4JUXIQVQ .

[ericbrunetgouet]

Dear all, I am a newby that just came to camembert and tried to use it following the NLI example from Roberta.

When I type this line, it appears that "mnli" is not recognized (KeyError). tokens = camembert.encode('Salut.', 'Bonjour.') camembert.predict('mnli',tokens).argmax() # 0: contradiction

What would be the correct "head" instead of mnli ? All the best, and thanks a lots for this great piece of work, Eric Brunet Gouet CH Versailles

[louismartin]

Hi @ericbrunetgouet We did not release the NLI model yet, therefore you cannot use the model like this unless you retrain it for NLI yourself. Thanks, Louis

[ericbrunetgouet]

Dear Louis, Thanks a lot. Hope you will do that. I will try my hypotheses with the English version. Best regards Eric

[LeoDeep]

Hi @stefan-it and @louismartin , I successfully used the run_ner.py script in order to evaluate the PoS tagging task for the CamemBERT

Currently I'm working on evaluating the dependency parsing task (evaluating UAS and LAS on the same dataset I used for PoS (e.g. ParTUT). The dataset contains all the information I need since the head and the dependency relation are labelled for each word. However I'm wondering how the depedency parsing tasks can be implemented. Is there a script or change in run_ner.py available that allows this type of task?

Thanks for everything as always. Cheers!

[benjamin-mlr]

Hi,

It wouldn’t be trivial to modify run_ner.py for parsing as it requires a graph prediction layer. We plan to release the code soon.

Benjamin

On Sat, Dec 28, 2019 at 3:08 PM LeoDeep notifications@github.com wrote:

Hi @stefan-it https://github.com/stefan-it and @louismartin https://github.com/louismartin ,

I successfully used the run_ner.py script in order to evaluate the PoS tagging task for the CamemBERT

Currently I'm working on evaluating the dependency parsing task (evaluating UAS and LAS on the same dataset I used for PoS (e.g. ParTUT). The dataset contains all the information I need since the head and the dependency relation are labelled for each word.

However I'm wondering how the depedency parsing tasks can be implemented. Is there a script or change in run_ner.py available that allows this type of task?

Thanks for everything as always. Cheers!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pytorch/fairseq/issues/1450?email_source=notifications&email_token=AEHOJYYWZOT4Q6KN7NRBWO3Q25MW5A5CNFSM4JUXIQV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHYKUPI#issuecomment-569420349, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEHOJYYJ3GF3HXYSBGUF2GDQ25MW5ANCNFSM4JUXIQVQ .

[stale[bot]]

This issue has been automatically marked as stale. If this issue is still affecting you, please leave any comment (for example, "bump"), and we'll keep it open. We are sorry that we haven't been able to prioritize it yet. If you have any new additional information, please include it with your comment!

[stale[bot]]

Closing this issue after a prolonged period of inactivity. If this issue is still present in the latest release, please create a new issue with up-to-date information. Thank you!