CarperAI / cheese

Used for adaptive human in the loop evaluation of language and embedding models.
MIT License
300 stars 24 forks source link

Add rule based feedback to architext example #33

Closed ehavener closed 1 year ago

ehavener commented 1 year ago

Details This PR adds two new input fields to the architext example. Rule is a randomly chosen design rule and rule score is a likert scale assessment of how well the generated design adheres to the rule. This preference will be used to train a sparrow like rule model. This PR also sets the correct HuggingFace model path.

Screenshots

Screenshot 2022-10-29 at 4 06 47 PM
TheodoreGalanos commented 1 year ago

Looks nice! Added below some cleaner descriptions for the rules:

The rooms are appropriately allocated for comfortable living.
The design is structured around a pronounced room
The design is simple
The design contains no inaccessible rooms
The design is plausible
The design makes sense 

As for the preference ranking, I think we can use something closer to Sparrow, like: disagree, mostly disagree, unsure, mostly agree, agree

Although to be honest, not very experienced in the impact of scales and if they matter.