Closed Nixinova closed 1 year ago
Found a roadblock: unless I were to fetch every sample github-linguist offers, testing against a classifier trained only on samples corresponding to a given language gives the classifier nothing to compare the files to so it can never give a proper rating. I don't think this would be possible while also not taking forever.
Work is at https://github.com/Nixinova/LinguistJS/tree/feat-bayesian-classification but won't be continued.
Actually, I think it should be possible without using too many web requests if I just load one sample per language. So per heuristic the web requests is the number rof associated languages + 1. Maybe this should be off by default.
Well, loading one sample per language makes the results no better than normal, and it'd take too many requests for this to be worthwhile.
Currently around 200 of github/linguist's samples are misclassified:
A lot of these are because github/linguist uses its own Bayesian classifier on these ambiguous files while linguist-js currently doesn't.
There are a couple npm packages like classificator that may be of use here.