search

nlpodyssey / cybertron

Cybertron: the home planet of the Transformers in Go
BSD 2-Clause "Simplified" License
280 stars 26 forks source link

Support for `distilbert` and `spacy transformer` models for text-encondings #7

Open meetsoni15 opened 1 year ago

meetsoni15 commented 1 year ago

Currently its supporting six models as per downloadmodel.go file

var supportedModelsFiles = map[string][]string{
    "bart":       {"pytorch_model.bin", "vocab.json", "merges.txt"},
    "pegasus":    {"pytorch_model.bin", "spiece.model"},
    "marian":     {"pytorch_model.bin", "vocab.json", "source.spm", "target.spm"},
    "bert":       {"pytorch_model.bin", "vocab.txt", "tokenizer_config.json"},
    "electra":    {"pytorch_model.bin", "vocab.txt", "tokenizer_config.json"},
    "flair":      {"pytorch_model.bin"},
}

when we are trying to use GPL/dbpedia-entity-msmarco-distilbert-gpl for text encoding got below error

{"level":"fatal","error":"failed to convert model: unsupported model type: \"distilbert\"","time":"2022-11-30T10:37:21+05:30"}