Viet1004 / Code4Earth-2024-Challenge-24

The aim of the project is to create a knowledge graph for Scientific documents. This is linked directly to the challenge https://github.com/ECMWFCode4Earth/challenges_2024/issues/7
2 stars 0 forks source link

Implementing KG generation pipeline #12

Closed thinhngo-x closed 1 month ago

thinhngo-x commented 1 month ago

The objective of this issue is to implement a complete pipeline for KG generation and to resolve some problems below. After prototyping (PR https://github.com/Viet1004/Code4Earth-2024-Challenge-24/pull/11), I identified some problems:

thinhngo-x commented 1 month ago

For counting tokens, keep an eye on this issue on Ollama: https://github.com/ollama/ollama/issues/3582. When this api is added, we can rely on ollama native api to count number of tokens instead of relying on huggingface as in https://github.com/ollama/ollama/issues/1716#issuecomment-2023379127.