search

ruanchaves / hashformers

Hashformers is a framework for hashtag segmentation with Transformers and Large Language Models (LLMs).
MIT License
67 stars 4 forks source link

Proposal: Replace mlm-scoring dependency with better-mlm-scoring #35

Closed ruanchaves closed 1 year ago

ruanchaves commented 1 year ago

Please Describe The Problem To Be Solved

The current dependency on mlm-scoring introduces a challenge due to its mxnet-cu110 dependency. This is causing compatibility issues with Google Colab and prevents us from updating our software stack. Furthermore, there could be potential improvements in scoring quality.

(Optional): Suggest A Solution

I suggest replacing mlm-scoring with better-mlm-scoring. This library doesn't rely on mxnet-cu110, resolving our compatibility issues. It would also allow us to keep our stack current, and is reported to yield better scoring results. The main trade-off will be the time invested in the transition and subsequent testing.

ruanchaves commented 1 year ago

In fact, we might broaden this to outsource our sentence scoring backend entirely to minicons. If scores are preserved or improved, this change should be accepted.

ruanchaves commented 1 year ago

Closed. See #36 .