search

guosyjlu / DS-Agent

Official implementation of "DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning" in ICML'24
128 stars 16 forks source link

API issues with CRFM stanford #16

Open SaeedSubayyal opened 2 months ago

SaeedSubayyal commented 2 months ago

Hi, I tried running it but it always keep showing issues with crfm and requires its api key.txt. I checked over the internet its a stanford project and require personal api to acess. Therefore, I cant run the project on colabs even though I follow all the directions mentioned by you in the repository. Also I have a question why not use llama 3.1 instead of chatgpt 4 as it can save money and probably is more efficient.

SaeedSubayyal commented 2 months ago

try: from helm.common.authentication import Authentication from helm.common.request import Request, RequestResult from helm.proxy.accounts import Account from helm.proxy.services.remote_service import RemoteService

setup CRFM API

auth = Authentication(api_key=open("crfm_api_key.txt").read().strip())
service = RemoteService("https://crfm-models.stanford.edu")
account: Account = service.get_account(auth)

except Exception as e: print(e) print("Could not load CRFM API key crfm_api_key.txt.")

guosyjlu commented 2 months ago

Hi,

  1. Currently, this repository is based on OpenAI API. The other LLMs are implemented by the original base repository. If you still find issues for using OpenAI LLMs, you can delete these irrelevant codes.
  2. Yes, of course! Note that Llama 3 was not released yet when we built this project. We do test the performance of Llama3-70b-chat and find it can indeed achieve desired success rate. If you want to use Llama3-70b-chat as your base LLM, you can use vllm framework to simulate OpenAI-alike API service.