BlinkDL / RWKV-LM

RWKV is an RNN with transformer-level LLM performance. It can be directly trained like a GPT (parallelizable). So it's combining the best of RNN and transformer - great performance, fast inference, saves VRAM, fast training, "infinite" ctx_len, and free sentence embedding.
Apache License 2.0
12.05k stars 827 forks source link

为什么生成的内容经常会重复 #190

Open bigcat26 opened 9 months ago

bigcat26 commented 9 months ago

model是RWKV-4-World-0.1B-v1-20230520-ctx4096。

比如这种

User:Generate a JSON file to describe an automation action.
Assistant:[ To describe an automation action, you can use the `describe` method. Here's an example of how you can use the `describe` method:

describe("User", function () {
  it("should show the user", function () {
    expect(User.findOne({_id: 1}).first()).toEqual({_id: 1});
  });
});
describe("User", function () {
  it("should show the user", function () {
    expect(User.findOne({_id: 1}).first()).toEqual({_id: 1});
  });
});

This will show the user's name, email address, and password. The `User.findOne({_id: 1})` method will return the user's name, email address, and password. The `User.findOne({_id: 1})` method will return the user's email address, password, and password. The `User.find]

还有这种

User:Generate a JSON file to describe an automation action.
Assistant:[ To describe an automation action, you can use the `Actions` module in the `Action` module. Here's an example of how you can create a `Actions` module:

import { Actions } from '@/actions/actions';
const actions = [
  {
    name: 'User',
    email: 'user@example.com',
    password: 'password',
    emailAddress: 'email@example.com',
    password: 'password',
    emailAddress: 'email@example.com',
    password: 'password',
    emailAddress: 'email@example.com',
    password: 'password',
    emailAddress: 'email@example.com',
    password: 'password',
    emailAddress: 'email@example.com',
    password: 'password',
    emailAddress: 'email@example.com',
    password: 'password',
    emailAddress: 'email@example.com',
    password: 'password',
    email]

后面试了,RWKV-4-Raven-1B5-v12-Eng98%-Other2%-20230520-ctx4096

好像会稍微好点。参数量太低就会有这种问题?

BlinkDL commented 9 months ago

https://pypi.org/project/rwkv/ Try: temperature = 1.0, top_p = 0.3, top_k = 0, alpha_frequency = 1, alpha_presence = 0, alpha_decay = 0.996 For alpha_frequency and alpha_presence, see "Frequency and presence penalties": https://platform.openai.com/docs/api-reference/parameter-details

shouldsee commented 4 days ago

我在跑chatRWKV的时候也经常碰到循环的问题,这是主要跟采样策略有关系吗?