gptscript-ai / knowledge

Knowledge for GPTScript
https://gptscript-ai.github.io/knowledge/
Apache License 2.0
29 stars 14 forks source link

add: cohere_rerank postprocessor + expand env vars in flows config #31

Closed iwilltry42 closed 4 months ago

iwilltry42 commented 4 months ago

This PR adds the cohere_rerank postprocessor. Cohere offers free trial keys: https://dashboard.cohere.com/api-keys

Additionally, environment variables ${var} and $var are now expanded in the flows-config files, so ${COHERE_API_KEY} in the example config below is inferred from the environment.


flows:
  foo:
    default: false
    ingestion:
      - filetypes: [".md"]
        documentloader:
          name: plaintext
        textsplitter:
          name: markdown
        transformers:
          - name: filter_markdown_docs_no_content
    retrieval:
      retriever:
        name: basic
        options:
          topK: 25
      postprocessors:
        - name: cohere_rerank
          options:
            apiKey: ${COHERE_API_KEY}
            topN: 3
            model: rerank-multilingual-v3.0