Severe output quality difference between 4096 and 8192 models

[GenTxt]

Thanks for this great code and models.

I've been testing long form test generation from prompts with:

1) RWKV-4-Pile-14B-20230228-ctx4096-test663.pth

2) RWKV-4-Pile-14B-20230313-ctx8192-test1050.pth

I've made the necessary simple edits to chat.py, API_DEMO.py, and your gradio app.py to generate 4000+ tokens with both models using the same settings and found the 4096 model generates relatively coherent output for the entire 4000+ tokens. No small task.

Unfortunately, the 8192 model starts well then severely breaks down at around 2048 max tokens until by the end of 4000-8000 tokens it's simply repeating words. e.g. "... The Prince and the Pauper , The Prince and the Pauper , _The Prince and"

I understand most users are interested in chat-style generation and will never be interested in long form replies but I'm still wondering if there are there are special settings to improve model 8192 output relative to 4096?

Also, I saw a post where a 50b model is in the works? Fantastic if that happens. That would be a great time to have a version of the API_DEMO.py that runs on multiple gpus.

Cheers,

[BlinkDL]

Please send me your prompt so I can check it. :)

[GenTxt]

Thanks for the quick reply. Prompt is below. +gen used for 'chat.py' only.

Provided settings for the 3 scripts mentioned above. Ran a few more tests and the results are similar.

One useful addition would be the generation and replication of seeds to repeat output on the user's machine.

+gen It was about eleven o'clock in the morning, mid October, with the sun not shining and a look of hard wet rain in the clearness of the foothills. I was wearing my powder-blue suit, with dark blue shirt, tie and display handkerchief, black brogues, black wool socks with dark blue clocks on them. I was neat, clean, shaved and sober, and I didn't care who knew it. I was everything the well-dressed private detective ought to be. I was calling on four million dollars.

All scripts launching on single RTX 3090

Ubuntu 22.04, python3.10

Loading models from local folder:

args.MODEL_NAME = 'models/RWKV-4-Pile-14B-20230313-ctx8192-test1050'

args.ctx_len = 1024 (for all)

1) chat.py settings:

args.FLOAT_MODE = "cuda fp16i8 *20 -> cuda fp16"

CHAT_LEN_SHORT = 40 CHAT_LEN_LONG = 150 FREE_GEN_LEN = 8190 # 4090-8190 set for max generation

GEN_TEMP = 1.0 GEN_TOP_P = 0.85

PROMPT_FILE = f'{current_path}/prompt/default/{CHAT_LANG}-2.py'

2) API_DEMO.py

strategy='cuda fp16i8 *20 -> cuda fp16'

args = PIPELINE_ARGS(temperature = 1.0, top_p = 0.7, top_k=0, # top_k = 0 then ignore alpha_frequency = 0.1, # change to 0.2 or 0.1 alpha_presence = 0.1, # change to 0.1 token_ban = [0], # ban the generation of some tokens token_stop = []) # stop generation whenever you see any token here chunk_len = 256) # split input into chunks to save VRAM (shorter -> slower)

pipeline.generate(ctx, token_count=8190, args=args, callback=my_print)

3) app.py

strategy='cuda fp16i8 *20 -> cuda fp16'

def infer( ctx, token_count=8190, temperature=1.0, top_p=0.85, presencePenalty = 0.1, countPenalty = 0.1, ): args = PIPELINE_ARGS(temperature = max(0.2, float(temperature)), top_p = float(top_p), alpha_frequency = countPenalty, alpha_presence = presencePenalty, token_ban = [0], # ban the generation of some tokens token_stop = []) # stop generation whenever you see any token here

gr.Slider(10, 8190, step=10, value=150), # token_count

[BlinkDL]

Ok I'll take a look.

API_DEMO.py already runs on multiple gpus btw. Try strategy = 'cuda:0 fp16 -> cuda:1 fp16'

[BlinkDL]

Ok I tried it and no issue here :)

Make sure you are using latest ChatRWKV v2 and latest rwkv pip package (pip install rwkv --upgrade) v0.7.2

I am using GEN_TEMP = 1.0 GEN_TOP_P = 0.8 GEN_alpha_presence = 0.2 # Presence Penalty GEN_alpha_frequency = 0.2 # Frequency Penalty

FREE_GEN_LEN = 1000

args.strategy = 'cuda fp16'

args.MODEL_NAME = '/fsx/BlinkDL/HF-MODEL/rwkv-4-pile-14b/RWKV-4-Pile-14B-20230313-ctx8192-test1050'

Bob: +gen It was about eleven o'clock in the morning, mid October, with the sun not shining and a look of hard wet rain in the clearness of the foothills. I was wear ing my powder-blue suit, with dark blue shirt, tie and display handkerchief, black brogues, black wool socks with dark blue clocks on them. I was neat, clean, shaved and sober, and I didn't care who knew it. I was everything the well-dressed private detective ought to be. I was calling on four million dollars.

The building that housed The San Francisco Daily Bugle was not at all what I had expected. It was no black-brick structure with a ground-floor door like those of the other offices I had visited. It was one of the swankiest buildings in the city, all white marble and chrome steel. Its entrance was across the street from the newspaper office and it had a green door that bore the printed sign: Daily Bugle —Editorial, Advertising and Legal Offices.

A very fine elevator took me to the seventh floor. I got out and crossed the lobby to a glass door which led into a long, broad room that had desks, editors' offices, a typesetting room, an art department, offices for reporters and an editorial department. And a lot of people sitting around reading papers, talking on telephones, typing letters and making corrections on galleys. A radio played dance music in the background.

I looked around for someone who looked like Don Juan Tenorio and then remembered he was Mexican. The person I saw most like him was an old lady with iron-gray hair who sat at a desk against one wall and stared into space while she wrote with her fountain pen on a pad in front of her. She wore green eyeshade, cat's-eye glasses, suit of black crepe and white stockings. Her shoes were low-heeled black pumps. I went over and spoke to her and she lifted her head off her pad with some difficulty. She had never heard of Parker Faine before but after a bit of chin music from me she remembered him well enough to tell me he was in his office on the floor above but occupied with another client right now. She told me his name was Kenneth Hooper MacNulty III but that didn't mean anything to me except it sounded like it should be spelled Kiner or Kinerneer or something like that. I said okay and went back across the lobby to the elevators again and took one up to the fifth floor. I went through a door marked Kane Enterprises into an area devoted entirely to waiting rooms where a man in a double-breasted blue suit sat behind a desk smoking cigarettes and tapping his pencil on the glass top of one of them. His name was Mr Prentiss Kane who was away somewhere on business so I took the elevator up to Mr MacNulty's floor again, found his room easily enough, pushed open the door without knocking and went inside. The first thing I saw in there was Ellen Faine sitting behind his desk reading an afternoon edition of a newspaper through her horn-rimmed glasses which were tilted low down on her nose. Her hair wasn't up in curlers any more but it wasn't down either. She wore pearl earrings, bright green lipstick which made her lips look dark against them, long gray slacks with leg-of-mutton sleeves and buckled sandals with bright red bottoms so bright they hurt my eyes. They were shapely legs, smooth as silk under dark gray wool stockings without any run in them anywhere. I turned around as though to leave before she could look up at me but she called out to me:

'Who is it?'

I said: 'Parker Faine.'

She smiled politely as though she hadn't been able to recognize my voice over the telephone only half an hour before because she'd been so damn busy talking herself while he'd been speaking for her over the telephone. Then she said: 'Sit down,' and kept on reading about herself in The San Francisco Daily Bugle. The desk was covered with bills, correspondence papers, copies of local newspapers and two different colored cigarette boxes full of small bits of rock that looked like grass but weren't because they had scales on them instead of seeds like grass has. There were three gold-plated ashtrays full of cigarette stubs piled around them on top of a large mahogany desk crowded with files of newspapers from all over America both east and west, with magazines too with names printed in gilt letters across their spines: Life, Look , New York Times , Chicago Tribune , Los Angeles Times , Chicago Sun Times. Two bookcases filled with paperbacks were built into one wall between two windows that faced west across Market Street towards Sutter Street where traffic moved like a swarm of bees between lunchtime and dinnertime in all directions carrying people in cars from job to job—and yet another flyway filled with books ran along behind these desks right up against the wall so you couldn't see through it or anything else unless you pushed them aside yourself because you couldn't push a window open wide enough for you to stick your head out far enough to do any good when you were tired of looking at those flyers stuck together everywhere through

Bob: +++ the office like a lot of leeches. On the top shelf of one bookcase there were a couple of framed photographs, one of them with some men I guessed were MacNulty's relatives and another one with the inscription on it that said MacNulty, Kenneth Hooper , Photographer. It was signed Oakley Cinthrop Tuttle III, B.S. Camera.

On the opposite side of the room was a large plate-glass window behind which the whole sky was darkening into black as evening came on, with lights beginning to show in the buildings across the street and in other buildings across the park to the south of it. The windows showed everything as though they were mirrors: traffic signals on both ends of the street at the head of Market Street, cars and taxis crossing those streets through red lights. And in the middle distance to the east there was a large building with tall columns in front of it and a statue of a naked girl in bronze standing on top of it. She had breasts, an hourglass waist and wasn't wearing anything except for gold chains that hung around her neck and a pair of heavy golden handcuffs on her wrists. I couldn't make up my mind whether or not I wanted to be standing behind her trying to put my hands up between her legs when I looked at her, so I kept on looking at MacNulty's photograph. It was a good one for this kind of job: well lighted and clear without being flashy.

I heard him clear his throat behind me and said without turning around: 'You're not MacNulty.'

'Certainly not,' he said, but still without turning around. 'I'm Leonard Faine, Mr MacNulty's business manager.' He took a newspaper out from under his hand and laid it aside carefully so he wouldn't mark it with his fingerprints. He put both hands flat down on his desk top so I could see his name-plate: L. Faine , Chief Editor. He had been reading some sort of newspaper but he turned his head away from me again to look at it.

'How do you do?' I said. 'My name is Parker Faine.'

'Yes,' he said gravely, 'I know who you are.' Then he raised his head slowly until his eyes met mine over his glasses which had slipped down on his nose so that they looked as though they were lying sideways in a deep pocket between them. 'What can I do for you?' he asked pleasantly enough but not very warmly either. He wore no earrings, no rings, no watch chain nor any sign that he ever had worn any on either wrist or anywhere else where jewelry might have shown up if he'd ever had any jewelry on him that he didn't like or wanted to get rid of. His fingernails were clean but not manicured; only the pink color had been taken off their edges with an emery board and it looked as though he was the kind of man who did all his own manicuring himself instead of letting anyone else do it for him because he knew exactly what results he wanted to achieve and knew exactly what techniques were necessary to achieve them too. He wore tortoise-shell rimless spectacles that looked good with his face and would probably have looked good with anyone else's face too but they didn't look good with his face because they seemed too small for him or too plain or something because they gave him an expression that wasn't just shy but made him look as though he didn't quite know how to go about things socially or even how to go about getting along in general so that all those new ways he'd picked up from newspaper people seemed like a kind of protective coloration he'd grown while living by himself out there in Wild Cat Canyon where maybe nobody ever came who was exactly like him or even understood exactly what sort of man he really was underneath all those bright colors and silver buttons on his shirtfront and blue tie under his jutting chin. It was hard to tell whether all this gaudiness was put on by him deliberately for show because that's how people who work out there usually are or whether it was put on for effect by somebody else who had dressed him up that way before sending him out into civilization again where everyone would judge him by standards entirely different from those which prevailed in Wild Cat Canyon where no one would notice these strange new things about him anyway because no one ever goes there except cowboys and trappers, mountaineers and hikers and weekend flyers who spend their lives hanging over precipices ten thousand feet up among peaks covered with snow half the year round.

He said: 'My name is Leonard Faine,' pointing to himself with one hand while keeping the other one under the glass shield above his head so no light could shine through its oval lenses onto the sides of his eyes where

Bob: +++ the fine lines of age and sleeplessness were. 'You must have heard of my father's name,' he went on.

I said I had and thanked him for giving me the assignment on the MacNulty piece and asked if he'd mind if I made a note of that to give me something to hang on to while I was working on it. He smiled very affably and lifted his hands up in front of him as though he were begging me not to.

'Not at all,' he said, 'not at all.' Then he said: 'I like your idea about MacNulty myself. I don't think it would be a bad idea to give MacNulty some publicity now that the old son of a bitch is getting so old and feeble. Maybe we could even put it into one of our books as a chapter heading.' He laughed and slapped his thigh gently as though he were trying to get some cramp out of it. 'I was just about to go out to dinner,' he said, 'but why don't you wait here for me and I'll fix you a cup of coffee and we'll talk this thing over before we get started.'

'That's fine with me,' I said, 'but what about all that Wild Cat Canyon stuff? What will we do with that?'

'Leave it in,' he said. 'Leave it in until you're ready to go. Then we'll send it out to San Francisco for the same department that does the book reviews.'

'Okay,' I said, 'but won't it take up too much room?'

'How big is your piece?' he asked. When I told him how long it was he said: 'That's all right. Just put a line under your byline and then tell Mr Faine you're going out for a sandwich or something and be back here when you're through.'

I took my leave of him in a sort of wary way which seemed the best way to take with him since there was no telling what kind of a guy this MacNulty was or whether Faine would like what I was going to do with his book or not, although personally I thought it would work fine as far as having some fun with him went but there wasn't much chance of being able to use any in the way of literary criticism because this wasn't the kind of story that would lend itself to any literary treatment at all. It was just about as complicated as a poker game where somebody else can see your cards better than you can see them yourself. MacNulty had never done anything worth doing except read newspaper columns written by other people and stuff they'd written in them and spent the rest of his time running from one newspaper office to another whenever one of them got out anything against him and then defending himself in his own little world that included his son Kenneth who worked for him too but never seemed interested in doing anything but sit there smoking cigarettes while his father played cards with friends over coffee until time for supper when Kenneth would go home leaving MacNulty alone with himself again but much happier than when he'd been sitting alone in the other room listening to himself and being delighted with himself for the things he thought about himself and then being glad about himself afterwards because he'd been so delighted by them and feeling sad because his friends didn't appreciate him as much as they should so that his son couldn't understand why they never came around any more. Maybe someday someone might say something nice about him in print so that he wouldn't have any trouble understanding why people wrote good things about people who did things like that; but then maybe not; because sometimes newspaper people did things like that too—for reasons nobody ever could understand unless they were detectives, which they weren't. But not me; I was only a private detective with too many irons in too many fires; no, I wasn't going to read any more about this case or this guy, so maybe I wouldn't even read what Don Juan Faine had written about MacNulty before deciding whether or not it was worth writing an article about him. Maybe Faine had decided against writing an article after reading MacNulty's book which would be just one more indication that Faine was making more money out of the newspaper business than anybody else was even if they all had paid him handsomely for the honor of being associated with him on these big deals; or maybe he hadn't even finished reading it yet since Wild Cat Canyon had come out only yesterday morning but maybe somebody else had already turned it down on account of being afraid it would bring them trouble instead of fame, like Donovan Parker Bowles's book Beyond Hope. Maybe Faine wouldn't even bother reading Wild Cat Canyon at all because what he really wanted wasn't anything like that at all but something else: something like... like... well, like _Crime

Bob: +++ at Blackbury High_ , by Owen Hooper MacNulty.

'Hello, Mr Parker,' Faine said. 'We were beginning to wonder whether you'd forgotten all about us.'

I told him I hadn't forgotten about him at all and hadn't been out to see him for two or three days for a very good reason: he'd told me he didn't want to be bothered with anybody at the moment and especially not with reporters. He had enough trouble as it was without having reporters pestering him all the time just because he'd lost a lawsuit and he didn't need any more of that now; so I said I wouldn't bother him any more unless he asked me to but if he wanted me to help him on his new book there was nothing I would enjoy doing more. I told him that if it weren't for my wife and son who were in the other room waiting for me, I'd go right over to his office now and get to work on his book instead of talking to him here where they might overhear us. He looked as though he would like to ask me why the hell didn't I go over there if that was where he lived but instead he just thanked me again for my interest in him and his book and wished me luck with it. Then he started down the corridor towards his office so fast I had to run after him and catch up with him, which was all right with me because I was enjoying myself, especially when he put his arm through mine again and we went down the corridor towards his office together like two guys walking together along an avenue of parked cars who aren't in the same car or even any kind of car at all except one is a Dusenberg while the other is a Studebaker and they're both going as fast as they can to get away from something on their mind. We got into the elevator together and rode up to the third floor together without saying anything while we listened to that elevator-man laughing at us in an exaggerated falsetto, making funny remarks about us being two old bachelors living up here by ourselves with nobody coming up here but us. It seemed that we weren't being laughed at but only made fun of because we were two happy old bachelors out for a ride on our private elevator instead of standing around in lines at public elevators with everybody else making fun of us, only this was just as good as having people come up and ask you what it felt like inside one of those high-priced apartments up there in the sky and having somebody say: 'It must be great living up here in one of these million dollar places.' So we got out on the third floor, walked along a hallway past a few doors until we came to MacNulty's door and Faine stopped there for a minute with his hand on the knob, turning around to face me again, leaning his head against his hand as though he were very tired and there was something wrong with his legs or back. But when he saw that I wasn't watching him anymore but looking down the corridor toward his son's door he straightened himself out again, knocked on MacNulty's door and turned away towards his own office again without giving me a chance to ask him what was wrong with MacNulty or why had he asked me instead of asking MacNulty himself? But I wasn't going to ask him because if he didn't want me asking questions then it didn't make any difference whether I did or not so I decided not to do it.

I thought it strange that Faine should have wanted me along on this assignment just because I happened to be in the neighborhood at the same time but then maybe that wasn't strange at all because some newspapermen are always looking for an angle where they can make some money while they're working on their story, especially newspapermen like Faine who make a lot more money than most of them do anyway; maybe he had been sent here by someone above him who wanted MacNulty investigated carefully before Faine wrote anything about it because it might be somebody else's turn next. After all, there was no telling who owned The San Francisco Daily Bugle , even if this were its real name; some multimillionaire industrialist might have bought it after buying half-a-dozen other newspapers so there might not even be a millionaire among them anymore; although I doubted whether even one millionaire could afford to buy a newspaper now that inflation was ruining everybody so much; it seemed more likely now that millionaires had fallen into such hard times themselves that they couldn't afford anything at all unless it was right under their nose where they could grab it; something you could throw away or sell on street corners like copper wire or aluminum cans that no one could have any use for except those who could still use them; although perhaps gold coins would do just as well if you knew how to sell

Bob: +++ them at a good price.

I knocked on MacNulty's door and then walked into the office as if I owned the place, turning on all the lights as I went so he could see who it was and what I wanted and make him feel at home right away. When I walked into his office I found him sitting behind his desk looking down at the city with his eyes wide open, staring out of that room out of a pair of eyes so big and so green and so far apart that I wondered why anybody would want to get in between them when there were so many other things in life that were made for you to look at with those eyes: mountains, sunsets, women, cities. I sat down in the client's chair that MacNulty had pulled up for me but it was too far back for me so I got up and put it right back where he liked it and put my hands on the desk behind me. I had to keep my voice low to be quiet when I talked because MacNulty had set his telephone down on the floor next to his chair instead of setting it on top of his desk where somebody might trip over it or kick it over by mistake. He looked across at me through his horn-rimmed glasses with those funny little circles in front of his eyes like old-fashioned ice skates and smiled very softly with both corners of his mouth coming up slightly. His hair was parted very neatly right down the middle and combed smooth on each side so that he looked as though he had two smooth tracks running straight down from his forehead to meet each other somewhere below the level of his chin, leaving a strip of white skin between them; but there wasn't any white skin on him except for a little bit around his eyes where they were deep blue as cornflowers, glinting under an arched black eyebrow that arched up over them into two blue hollows above each eye like two very small houses whose roofs came together in a point above their walls; but his skin looked just as pale, smooth and fresh as though all these white walls were some sort of a magic carpet which had carried him through space all by itself while some unseen attendant waited behind to touch up the paint on the buildings when he finished moving across them. He said: 'How do you do?' and raised one hand slowly as though he didn't have much strength in it and had to make every move slowly so it wouldn't shake too much before he made it; then he nodded slightly with both hands resting lightly on his desk top as though he didn't want to let go of it even for an instant because that might knock something off it or cause one of those tremors to run through its delicate structure like a spider's web across its top surface; then he lowered one hand again towards a book that lay open before him on the desk top next to his other hand which rested flat on top of it. He said: 'Are you Mr Parker Faine?'

I said I was Parker Faine and sat down in a chair which looked too comfortable to be legal—a padded black leather chair with rounded armrests; although when you sat down in one like that you felt more like sitting up than being comfortable because the padding gave your buttocks an unnatural prominence. Faine leaned forward across the desk towards me again and asked me if I wouldn't like to take off my coat; when I did so, revealing my suit coat underneath, he sat back again with one hand clutching a book on which there was a picture of a yacht off a beach somewhere and asked me how long I'd been a reporter; since when; what had brought me here today; whether or not this assignment was new for me; how long my newspaper expected me to stay in town; what hours they kept here at The San Francisco Daily Bugle ; whether this would be an expensive assignment for them; how many words they would pay me for writing about it; whether I intended to write something long or short about their case for publication later on; what pictures they could expect from me in connection with this story. As soon as I mentioned pictures he reached into one of those drawer-like compartments built into the right armrest of his chair and pulled out a big glossy envelope with four smaller envelopes inside each one of them; he handed them all to me without saying anything at all until I opened them up and saw four large glossy photos inside each envelope: two men in bathing suits lying on beach blankets doing some very muscular things while other people stood around watching them or taking their pictures or both. Then Faine picked up one picture after another out of those envelopes until there were three others sitting together next to each other like players lining up along their respective bases ready for some game they couldn't see anywhere else but only in their own minds; then turned one over carefully so that we could see who had taken all these pictures—Robert Alexander

Bob:

[GenTxt]

Looks like I'm not using FREE_GEN_LEN = limit correctly?

FREE_GEN_LEN = 1000 does work as per your examples and mine.

As previously mentioned above this can be raised to 4000+

I was trying to generate 8000+ tokens with the ctx8192 model

FREE_GEN_LEN = 8190 # 4090-8190 set for max generation

That appears to be the error on my end.

Cheers