Optimizing prompts - Githubissues

HEmile / iswc-challenge

MIT License

10 stars 1 forks source link

Optimizing prompts #19

Open miselico opened 2 years ago

miselico commented 2 years ago

Just leaving it here as a perhaps stupid thing we could try.

Currently, we are manually trying prompts with trial and error. We could choose to instead optimize the prompt itself.

This could be done by first getting a set of related words (could be generated by the LM itself). Then, we can use some way to optimize the prompts based on these words. Perhaps using some form of evolutionary computing for this.

thiviyanT commented 2 years ago

I was working on a script to automatically generate the prompts from randomly sampled examples from the training set and @DimitrisAlivas suggested doing stratified sampling.

@HEmile said that it's better to hand pick the prompts and make sure that they are very diverse and representative of the dataset we are working with. I agreed with Emile's suggestion.

JanKalo commented 2 years ago

I think optimizing might work better, but I am not sure, if this is appropriate for GPT-3. We might need lots of API calls to make it work better than manual prompts.

DimitrisAlivas commented 2 years ago

What if we just inversed the prompts using the true labels from the train set, and let the model "figure out" the correct natural language question/prompt?

Is that what you meant @miselico ?

Some things to already take into account here:

It will be another relatively costly experiment ( @JanKalo mentioned it )
There is probably more than 1 strong prompts that could work
Perhaps our hand-engineered prompts are already quite strong, given that they come as the byproduct of people who utilise prior knowledge.

DimitrisAlivas commented 2 years ago

What if we just inversed the prompts using the true labels from the train set, and let the model "figure out" the correct natural language question/prompt?

We could even cherry pick from our own perfect predictions to use as prompts/training samples

thiviyanT commented 2 years ago

Issue #20 is related to this problem

HEmile commented 2 years ago

We might need lots of API calls to make it work better than manual prompts.

yeah I'm mostly worried about this. I feel we already have very good numbers without tuning the prompts automatically, and looking into where the pain points are would probably be a more cost efficient way to improve stuff.