evaluate

[liujianwei2023]

Try to reproduce your code. Why is the response in your inference code all yes? Does the result still make sense?

[SAI990323]

This looks strange. If all 'yes' values are generated, there might be an error somewhere. Have you checked the specific logits and label distribution during the training process? Also, please check the version of the package and the size of the Lora model. If there is a version mismatch, it may cause the model code to run correctly but Lora may not actually be effective.

[liujianwei2023]

The data set here is provided by you. Can you update the version number corresponding to the implementation of your paper in the requirements? Thank you very much.

[liujianwei2023]

The following is an interception of some intermediate results, and the probability of predicting yes is also very close：

['Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. # noqa: E501\n\n### Instruction:\nGiven the user\'s preference and unpreference, identify whether the user will like the target movie by answering "Yes." or "No.".\n\n### Input:\nUser Preference: "Paris, Texas (1984)", "Rebel Without a Cause (1955)", "Return of the Pink Panther, The (1974)", "Ace Ventura: Pet Detective (1994)", "Magnificent Seven, The (1954)", "Star Trek: The Wrath of Khan (1982)", "Cat People (1982)", "Orlando (1993)", "Dave (1993)"\nUser Unpreference: "Kalifornia (1993)"\nWhether the user will like the target movie "Perez Family, The (1995)"?\n\n### Response:\nYes.\n\n### Explanation:\nThe user prefers "Paris, Texas (1984)", "Rebel Without a Cause (1955)", "Return of the Pink Panther, The (1974)", "Ace Ventura: Pet Detective (1994)", "Magnificent Seven, The (1954)", "Star Trek: The Wrath of Khan (1982)", "Cat People (1982)", "Orlando (1993)", "Dave (199', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. # noqa: E501\n\n### Instruction:\nGiven the user\'s preference and unpreference, identify whether the user will like the target movie by answering "Yes." or "No.".\n\n### Input:\nUser Preference: "Rebel Without a Cause (1955)", "Return of the Pink Panther, The (1974)", "Ace Ventura: Pet Detective (1994)", "Magnificent Seven, The (1954)", "Star Trek: The Wrath of Khan (1982)", "Cat People (1982)", "Orlando (1993)", "Dave (1993)"\nUser Unpreference: "Kalifornia (1993)", "Perez Family, The (1995)"\nWhether the user will like the target movie "Jurassic Park (1993)"?\n\n### Response:\nYes.\n\n### Explanation:\nThe user prefers "Rebel Without a Cause (1955)", "Return of the Pink Panther, The (1974)", "Ace Ventura: Pet Detective (1994)", "Magnificent Seven, The (1954)", "Star Trek: The Wrath of Khan (1982)", "Cat People (1982)", "Orlando (1993)", "Dave (1993)" and unpreferences "Kalifornia'] ['Yes.\n\n### Explanation:\nThe user prefers "Paris, Texas (1984)", "Rebel Without a Cause (1955)", "Return of the Pink Panther, The (1974)", "Ace Ventura: Pet Detective (1994)", "Magnificent Seven, The (1954)", "Star Trek: The Wrath of Khan (1982)", "Cat People (1982)", "Orlando (1993)", "Dave (199', 'Yes.\n\n### Explanation:\nThe user prefers "Rebel Without a Cause (1955)", "Return of the Pink Panther, The (1974)", "Ace Ventura: Pet Detective (1994)", "Magnificent Seven, The (1954)", "Star Trek: The Wrath of Khan (1982)", "Cat People (1982)", "Orlando (1993)", "Dave (1993)" and unpreferences "Kalifornia'] [[0.5731707811355591, 0.4268292486667633], [0.5828027129173279, 0.4171972870826721]] 1it [00:06, 6.24s/it]['Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. # noqa: E501\n\n### Instruction:\nGiven the user\'s preference and unpreference, identify whether the user will like the target movie by answering "Yes." or "No.".\n\n### Input:\nUser Preference: "Return of the Pink Panther, The (1974)", "Ace Ventura: Pet Detective (1994)", "Magnificent Seven, The (1954)", "Star Trek: The Wrath of Khan (1982)", "Cat People (1982)", "Orlando (1993)", "Dave (1993)", "Jurassic Park (1993)"\nUser Unpreference: "Kalifornia (1993)", "Perez Family, The (1995)"\nWhether the user will like the target movie "Manhattan Murder Mystery (1993)"?\n\n### Response:\nYes.\n\n### Explanation:\nThe user prefers "Return of the Pink Panther, The (1974)", "Ace Ventura: Pet Detective (1994)", "Magnificent Seven, The (1954)", "Star Trek: The Wrath of Khan (1982)", "Cat People (1982)", "Orlando (1993)", "Dave (1993)", "Jurassic Park (1993)" and unpreferences "Kalifornia (1', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. # noqa: E501\n\n### Instruction:\nGiven the user\'s preference and unpreference, identify whether the user will like the target movie by answering "Yes." or "No.".\n\n### Input:\nUser Preference: "Ace Ventura: Pet Detective (1994)", "Magnificent Seven, The (1954)", "Star Trek: The Wrath of Khan (1982)", "Cat People (1982)", "Orlando (1993)", "Dave (1993)", "Jurassic Park (1993)", "Manhattan Murder Mystery (1993)"\nUser Unpreference: "Kalifornia (1993)", "Perez Family, The (1995)"\nWhether the user will like the target movie "Sleeper (1973)"?\n\n### Response:\nYes.\n\n### Explanation:\nThe user prefers "Ace Ventura: Pet Detective (1994)", "Magnificent Seven, The (1954)", "Star Trek: The Wrath of Khan (1982)", "Cat People (1982)", "Orlando (1993)", "Dave (1993)", "Jurassic Park (1993)", "Manhattan Murder Mystery (1993)" and unpreferences "Kalifornia (199'] ['Yes.\n\n### Explanation:\nThe user prefers "Return of the Pink Panther, The (1974)", "Ace Ventura: Pet Detective (1994)", "Magnificent Seven, The (1954)", "Star Trek: The Wrath of Khan (1982)", "Cat People (1982)", "Orlando (1993)", "Dave (1993)", "Jurassic Park (1993)" and unpreferences "Kalifornia (1', 'Yes.\n\n### Explanation:\nThe user prefers "Ace Ventura: Pet Detective (1994)", "Magnificent Seven, The (1954)", "Star Trek: The Wrath of Khan (1982)", "Cat People (1982)", "Orlando (1993)", "Dave (1993)", "Jurassic Park (1993)", "Manhattan Murder Mystery (1993)" and unpreferences "Kalifornia (199'] [[0.5806307196617126, 0.41936925053596497], [0.5781132578849792, 0.42188674211502075]]

[SAI990323]

The data set here is provided by you. Can you update the version number corresponding to the implementation of your paper in the requirements? Thank you very much.

We use peft==0.3.0 & transformers==4.28.0. If you use the latest version, a useful link I here.

[SAI990323]

The following is an interception of some intermediate results, and the probability of predicting yes is also very close：

['Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. # noqa: E501\n\n### Instruction:\nGiven the user's preference and unpreference, identify whether the user will like the target movie by answering "Yes." or "No.".\n\n### Input:\nUser Preference: "Paris, Texas (1984)", "Rebel Without a Cause (1955)", "Return of the Pink Panther, The (1974)", "Ace Ventura: Pet Detective (1994)", "Magnificent Seven, The (1954)", "Star Trek: The Wrath of Khan (1982)", "Cat People (1982)", "Orlando (1993)", "Dave (1993)"\nUser Unpreference: "Kalifornia (1993)"\nWhether the user will like the target movie "Perez Family, The (1995)"?\n\n### Response:\nYes.\n\n### Explanation:\nThe user prefers "Paris, Texas (1984)", "Rebel Without a Cause (1955)", "Return of the Pink Panther, The (1974)", "Ace Ventura: Pet Detective (1994)", "Magnificent Seven, The (1954)", "Star Trek: The Wrath of Khan (1982)", "Cat People (1982)", "Orlando (1993)", "Dave (199', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. # noqa: E501\n\n### Instruction:\nGiven the user's preference and unpreference, identify whether the user will like the target movie by answering "Yes." or "No.".\n\n### Input:\nUser Preference: "Rebel Without a Cause (1955)", "Return of the Pink Panther, The (1974)", "Ace Ventura: Pet Detective (1994)", "Magnificent Seven, The (1954)", "Star Trek: The Wrath of Khan (1982)", "Cat People (1982)", "Orlando (1993)", "Dave (1993)"\nUser Unpreference: "Kalifornia (1993)", "Perez Family, The (1995)"\nWhether the user will like the target movie "Jurassic Park (1993)"?\n\n### Response:\nYes.\n\n### Explanation:\nThe user prefers "Rebel Without a Cause (1955)", "Return of the Pink Panther, The (1974)", "Ace Ventura: Pet Detective (1994)", "Magnificent Seven, The (1954)", "Star Trek: The Wrath of Khan (1982)", "Cat People (1982)", "Orlando (1993)", "Dave (1993)" and unpreferences "Kalifornia'] ['Yes.\n\n### Explanation:\nThe user prefers "Paris, Texas (1984)", "Rebel Without a Cause (1955)", "Return of the Pink Panther, The (1974)", "Ace Ventura: Pet Detective (1994)", "Magnificent Seven, The (1954)", "Star Trek: The Wrath of Khan (1982)", "Cat People (1982)", "Orlando (1993)", "Dave (199', 'Yes.\n\n### Explanation:\nThe user prefers "Rebel Without a Cause (1955)", "Return of the Pink Panther, The (1974)", "Ace Ventura: Pet Detective (1994)", "Magnificent Seven, The (1954)", "Star Trek: The Wrath of Khan (1982)", "Cat People (1982)", "Orlando (1993)", "Dave (1993)" and unpreferences "Kalifornia'] [[0.5731707811355591, 0.4268292486667633], [0.5828027129173279, 0.4171972870826721]] 1it [00:06, 6.24s/it]['Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. # noqa: E501\n\n### Instruction:\nGiven the user's preference and unpreference, identify whether the user will like the target movie by answering "Yes." or "No.".\n\n### Input:\nUser Preference: "Return of the Pink Panther, The (1974)", "Ace Ventura: Pet Detective (1994)", "Magnificent Seven, The (1954)", "Star Trek: The Wrath of Khan (1982)", "Cat People (1982)", "Orlando (1993)", "Dave (1993)", "Jurassic Park (1993)"\nUser Unpreference: "Kalifornia (1993)", "Perez Family, The (1995)"\nWhether the user will like the target movie "Manhattan Murder Mystery (1993)"?\n\n### Response:\nYes.\n\n### Explanation:\nThe user prefers "Return of the Pink Panther, The (1974)", "Ace Ventura: Pet Detective (1994)", "Magnificent Seven, The (1954)", "Star Trek: The Wrath of Khan (1982)", "Cat People (1982)", "Orlando (1993)", "Dave (1993)", "Jurassic Park (1993)" and unpreferences "Kalifornia (1', 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. # noqa: E501\n\n### Instruction:\nGiven the user's preference and unpreference, identify whether the user will like the target movie by answering "Yes." or "No.".\n\n### Input:\nUser Preference: "Ace Ventura: Pet Detective (1994)", "Magnificent Seven, The (1954)", "Star Trek: The Wrath of Khan (1982)", "Cat People (1982)", "Orlando (1993)", "Dave (1993)", "Jurassic Park (1993)", "Manhattan Murder Mystery (1993)"\nUser Unpreference: "Kalifornia (1993)", "Perez Family, The (1995)"\nWhether the user will like the target movie "Sleeper (1973)"?\n\n### Response:\nYes.\n\n### Explanation:\nThe user prefers "Ace Ventura: Pet Detective (1994)", "Magnificent Seven, The (1954)", "Star Trek: The Wrath of Khan (1982)", "Cat People (1982)", "Orlando (1993)", "Dave (1993)", "Jurassic Park (1993)", "Manhattan Murder Mystery (1993)" and unpreferences "Kalifornia (199'] ['Yes.\n\n### Explanation:\nThe user prefers "Return of the Pink Panther, The (1974)", "Ace Ventura: Pet Detective (1994)", "Magnificent Seven, The (1954)", "Star Trek: The Wrath of Khan (1982)", "Cat People (1982)", "Orlando (1993)", "Dave (1993)", "Jurassic Park (1993)" and unpreferences "Kalifornia (1', 'Yes.\n\n### Explanation:\nThe user prefers "Ace Ventura: Pet Detective (1994)", "Magnificent Seven, The (1954)", "Star Trek: The Wrath of Khan (1982)", "Cat People (1982)", "Orlando (1993)", "Dave (1993)", "Jurassic Park (1993)", "Manhattan Murder Mystery (1993)" and unpreferences "Kalifornia (199'] [[0.5806307196617126, 0.41936925053596497], [0.5781132578849792, 0.42188674211502075]]

Have you verified your stored LoRA model? Is the size of the model appropriate?