search

Machine-Learning-for-Medical-Language / cnlp_transformers

Transformers for Clinical NLP
https://cnlp-transformers.readthedocs.io/en/stable/
Apache License 2.0
21 stars 17 forks source link

character level model support #121

Closed etgld closed 2 days ago

etgld commented 1 year ago

Support for character based models, in particular Flair and canine

etgld commented 1 year ago

Clinical genomics stuff on pause for now but when that starts again can try to fold in the CANINE support with all the recent updates

etgld commented 1 year ago

Hoping to do some work on this issue this week since Timelines is on pause for ITCR. Will need to rebase the branch in my fork off of the error analysis branch and then adapt the error analysis code to work with character level.

etgld commented 1 year ago

Still figuring out the most efficient way to do this since a couple of the approaches I tried also have some issues.

Have also been thinking about how to do adapt character level stuff for relation extraction. Tagging has also been more expensive than its wordpiece counterpart, which would mean even more of a blowup for relex. Maybe there's some way to leverage sparse matrices for both

etgld commented 1 year ago

Have something that's basically working modulo an issue with the batches being the wrong shape if one uses say layer 8 out of 12 on canine and some formatting tweaks for the error analysis

The former issue might have something to do with the relatively unique downsampling -> upsampling part of the canine architecture. Actually only found out about this by accident since the layer 8 setting was something I was trying with a Roberta based model. Will look into both of these more though.

etgld commented 2 days ago

This got folded in with my latest PR