dan-zeman
commented
1 year ago There is already a Sindhi dataset in the UD Github by @mazharaliabro. It has never been released primarily because it does not have dependencies. But it is 675 sentences / 6863 tokens with UPOS tags and some features. I suppose someone could use it to train a tagger and apply it to the new data. It should be checked whether the tokenization is compatible.
Regarding dependencies, I imagine that a parser based on XLM Roberta (it seems to contain Sindhi) and a mixture of existing UD treebanks (in the spirit of Udify) could produce something that the annotators could use.
With unexperienced annotators it may be even more advisable to implement a language-specific validator that will check patterns that the universal validator cannot check.
meesumalam
AngledLuffa
muteeurahman
I am wondering, what advice is there for starting a new corpus? Is there a guide for doing so?
There is a team in Pakistan at Isra University who would like to see more Sindhi NLP tools in Stanza, and one of the ways we could make that happen is by annotating more raw data. (Currently there is not very much Sindhi in UD.) We're able to find an annotation company with some amount of linguistics knowledge in Sindhi, and of course there are people at Isra would put together a schema, review annotations, and possibly do some annotation as well.
There's already some tokenized data, so that should be taken care of. I believe the next step would be to label it with POS and dependencies.
Would it make sense to:
Are there better approaches for getting annotators who may not be very familiar with dependencies to label things? For example, I could also imagine breaking sentences into phrases and then trying to describe the relations between phrases as an easier approach for getting high quality annotations. That almost sounds like constituencies, for that matter, so perhaps it would be easier to build a constituency dataset and convert that to dependencies in some way.
@muteeurahman