-
我在执行cli_demo.py时，报错找不到属性
(base) root@hzhb:/data/fuzi.mingcha-main/src# python3 cli_demo.py --url_lucene_task1 "法条检索对应部署的 pylucene 地址" --url_lucene_task2 "类案检索对应部署的 pylucene 地址"
正在加载模型
Traceback (m…
-
e.g. `word _. .` gets translated to CGEL without the final period
-
The Unicode category of Connector Punctuation (https://www.unicode.org/charts/script/chart_Punctuation-Connector.html), which is a small collection of punctuation-like symbols which are used as connec…
-
Hi,i finetune MGM-2B on coco, but i got the warning that:
`{'loss': 6.9221, 'grad_norm': tensor(18.7422, device='cuda:0', dtype=torch.float64), 'learning_rate': 9.203084832904885e-06, 'epoch': 0.01}…
-
文件"/ft/train.py"，第168行：
`response = predict(messages, model, tokenizer)`
文件"/ft/train.py"，第70行：
`model_inputs = tokenizer([text]， return_tensors="pt").to(device)`
文件"/root/miniconda3/lib/python3.8…
-
Create a basic data preprocessing pipeline for a specific bioinformatics dataset to prepare it for LLM training. The pipeline should include steps for data cleaning, tokenization, and formatting
-
- Investigate whether claude 3 models need a new tokenization method or can we use the old methods for abuse detection
- Collect Data from the experimentation and share results to make the decisions.
-
In #38 @livyreal said that some PART are not correctly tokenized/lemmatized.
Let us try a different approach... The following pages define the PART POS tag (in general and for English).
- http://un…
-
```
When writing a tokenization unit test for the ClearTK wrappers for ClearNLP, I
found an inconsistency between OpenNLP's tokenization and ClearNLP's.
Consider the string:
String s = "\"John & Mar…
-
- [x] Implement token generation
- [ ] Implement pointer mechanism
- [ ] Error analysis
- [ ] Reinforcement learning implementation