search

HazyResearch / evaporate

This repo contains data and code for the paper "Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes"
480 stars 45 forks source link

[QUESTION] GPT-J-6b #12

Closed albertus-andito closed 1 year ago

albertus-andito commented 1 year ago

Hi,

Thanks for open-sourcing this code. Interesting stuffs!

Have you tried running EVAPORATE-CODE+ with open source huggingface models, such as GPT-J-6b? What's the overall performance? Is it comparable to text-davinci-003?

simran-arora commented 1 year ago

Hi! In preliminary analysis we find the quality with GPT-J-6B to fall short on function generation and thus quality to be worse than text-davinci-003. We are looking at various options and thinking about how to acquire good open-source LMs for that step. If you have pointers as well, let us know!

albertus-andito commented 1 year ago

I'm wondering whether the newly released StarChat can be used for the function generation step...