truera / trulens

Evaluation and Tracking for LLM Experiments
https://www.trulens.org/
MIT License
1.84k stars 163 forks source link

Add Snowflake's Cortex LLM function as a new feedback provider + Cortex-Finetune on feedback providers #1202

Closed sfc-gh-dhuang closed 1 week ago

sfc-gh-dhuang commented 3 weeks ago

Items to add to release announcement:

Note: cost tracking (token counts) work end to end but needs to figure out how to better compute overall costs for Cortex as it also incurs compute (query) costs on the data warehouses and those cost tracking is only available on Account level in Snowflake. We should put it in a separate PR.

Other details that are good to know but need not be announced:

finally: if snowflake_session:

Close the Snowflake session

        snowflake_session.close()


![image](https://github.com/truera/trulens/assets/171001675/c70e1bdf-e6c6-4998-a74b-d3fabc73a314)
review-notebook-app[bot] commented 3 weeks ago

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

sfc-gh-pmardziel commented 3 weeks ago

Is it still a draft?

sfc-gh-jreini commented 2 weeks ago

Please add api reference for the cortex provider & include in mkdocs.yml. See: https://github.com/truera/trulens/blob/e4813294002bff5a0fe767e5270e22524e5cb0dc/mkdocs.yml#L255