Availability of Bird-SQL data

[duyvuleo]

Hi,

Thanks for the exciting work on Bird-SQL data for text-to-SQL.

Any updates on the data availability?

Thanks!

[jacekplocharczyk]

You can download it here: https://bird-bench.github.io/ But I cannot run the code - there is some bug with openai API request

[huybery]

@jacekplocharczyk sorry for it, could you provide some detail error trace for the bug?

[jacekplocharczyk]

After running sh ./run/run_gpt.sh for first 10 questions I got the following predict_dev.json

{
    "0": "error:This is a chat model and not supported in the v1/completions endpoint. Did you mean to use v1/chat/completions?\t----- bird -----\tcalifornia_schools",
    "1": "error:This is a chat model and not supported in the v1/completions endpoint. Did you mean to use v1/chat/completions?\t----- bird -----\tcalifornia_schools",
    "2": "error:This is a chat model and not supported in the v1/completions endpoint. Did you mean to use v1/chat/completions?\t----- bird -----\tcalifornia_schools",
    "3": "error:This is a chat model and not supported in the v1/completions endpoint. Did you mean to use v1/chat/completions?\t----- bird -----\tcalifornia_schools",
    "4": "error:This is a chat model and not supported in the v1/completions endpoint. Did you mean to use v1/chat/completions?\t----- bird -----\tcalifornia_schools",
    "5": "error:This is a chat model and not supported in the v1/completions endpoint. Did you mean to use v1/chat/completions?\t----- bird -----\tcalifornia_schools",
    "6": "error:This is a chat model and not supported in the v1/completions endpoint. Did you mean to use v1/chat/completions?\t----- bird -----\tcalifornia_schools",
    "7": "error:This is a chat model and not supported in the v1/completions endpoint. Did you mean to use v1/chat/completions?\t----- bird -----\tcalifornia_schools",
    "8": "error:This is a chat model and not supported in the v1/completions endpoint. Did you mean to use v1/chat/completions?\t----- bird -----\tcalifornia_schools",
    "9": "error:This is a chat model and not supported in the v1/completions endpoint. Did you mean to use v1/chat/completions?\t----- bird -----\tcalifornia_schools"
}

Maybe the problem is the lib version - could you provide requirements.txt file?

[accpatrick]

Hi @jacekplocharczyk! Thank you for your interest in our work. We appreciate you mentioning this. It seems like you're facing an issue with the OpenAI keys. In our project, we utilized Azure OpenAI API keys to evaluate the performance of ChatGPT. To fetch the results, we used the openai.Completion.create() function, which is consistent with other models such as code-davinci-002 andtext-davinci-003.

However, we observed that for the regular keys, you should refer to the documentation at https://platform.openai.com/docs/api-reference/chat and use openai.ChatCompletion.create() instead. Additionally, for prompt design, please follow the instructions provided in the chatlm tag documentation at https://github.com/openai/openai-python/blob/main/chatml.md#few-shot-prompting. In this case, you will need to place your tag within the message parameter instead of prompt.

During our experiments, we found that the performance of ChatGPT with Azure OpenAI Key is superior to ChatGPT with the regular OpenAI key, and we only reported the best performance in our paper. However, we are actively working on updating the standard methods for calling ChatGPT in the regular account. Thanks a lot

[jacekplocharczyk]

Thanks @accpatrick - using 'text-davinci-003' worked :)