Closed WeihaoGe1009 closed 1 week ago
prototype written, need to install more packages and perform some test runs. need to test with different models.
writing a general function with AutoTokenizer, AutoModelForCausalLM. Not sure if specific tokenizer and model functions will work better. start with general function at this stage LLMTokenizater does not recognize llama-3.8-1 model.
bert-basic has limited token size, and cannot handle the issue with "unmatched tokens" for input and output when testing tokens with some small prompting example code. after discussion with David Bianchi, this looks like a tensor-flow issue. need to look into it so that we can still utilize legal-bert later.
things to do next:
write functions use AutoTokenizer to process texts with models from hugging face.