Can the n-gram indexing use adaptive size for N?

Siderite / lichessTools

Browser extension that turbocharges the lichess.org site with extra functionalities

MIT License

29 stars 6 forks source link

Can the n-gram indexing use adaptive size for N? #890

Open Siderite opened 3 weeks ago

Siderite commented 3 weeks ago

Vague idea at the moment, but one could keep the too generic 3-grams and try to encode only 4 grams that contain them, maybe go to 5-grams or more in some situations.