Output diverging on different GPUs using same prompt?

[jasonrohrer]

🐛 Bug

Wondering if anyone else is noticing this, or I'm missing something.

GPT-2 transformer run_generation.py

Running on both AWS p2 and p3, which have different GPUs.

Same text seed, same numerical seed (default: 42)

The output is identical for a long segment, and then suddenly diverges, picking a different word on the P3, and then carrying on in a different direction after that.

I'll do some more investigation and try to post examples. But first: Has anyone else noticed anything like this? Is this normal behavior?

[BramVanroy]

How do you set your seed? Personally I use the following, which sets... a lot of seeds but also useful variables for the backend.

    def set_seed(seed: Optional[int]):
        """ Set all seeds to make results reproducible (deterministic mode).
             When seed is None, disables deterministic mode. """
        if seed is not None:
            torch.manual_seed(seed)
            torch.cuda.manual_seed_all(seed)
            torch.backends.cudnn.deterministic = True
            torch.backends.cudnn.benchmark = False
            np.random.seed(seed)
            random.seed(seed)
            os.environ['PYTHONHASHSEED'] = str(seed)

Also make sure that you are setting your model to .eval().

[jasonrohrer]

Output is deterministic and reproducible on each GPU.

I.e., on the AWS P3, I get Output A consistently whenever I run it with the same input

On AWS P2, I get Output B consistently whenever I run it with the same input.

Could there be any chance that the default seeding is somehow based on the GPU identifier?

The other weird thing is that the divergence between Output A and Output B happens very far along in the output stream. They are the same for a long while, many hundreds of tokens, and then they diverge.

I would think that a seed difference would cause them to diverge right away.

[BramVanroy]

Hm, as far as I know different GPUs should still give consistent results with a fixed seed. Does it also happen with other generative models?

[julien-c]

gpu-specific numerical imprecision?

[jasonrohrer]

Haven't tried other models, just GPT2.

It's also hard to provide a reproduction case, because this happens with a big seed (something like 500 words), and happens a few thousand words into the output.

To achieve this long output, which is longer than the 1024 tokens in the model vector, I'm reseeding over and over with the output, daisy chaining the last output to generate the next input. There is no nondeterminism in this reseeding process, though, and I've verified that the input to the sampling function is identical every step of the way, until that one word where one GPU finally makes a different choice.

And this different choice is deterministic on each GPU. It always picks a different word at that spot.

I can at least provide the output samples to show you what I mean.

Also, I'll leave this open for now in case someone can shed some light on this or has the same problem. But after quite a bit of testing, I've determined that it's nothing on my end that is causing the divergence. Probably some GPU-specific imprecision that I can't control. It's also not a problem, exactly, but just something unexpected that made me take a closer look at my own code.

Anyway, search for the word "hunting" in each of these to see where the output diverges.

AWS P2 output sample:

When Oedipa got home she cried, didn't know how much, for many long minutes, and then came up with that often enough, two people's wanton moments being too much. Wendell lay across the back seat of her car and was sitting up when she arrived home. She started holding him more carefully, so as not to bump his head on the window, and almost drifted off until she could think of little else. "In other words, I won't remember my husband's name, Wendell," she told herself.

"Or maybe it won't be him, and I'll know who died." She opened her window and sat down with her arms on the steering wheel and turned the key. "I won't be able to take it, won't be able to get there."

She lost two hours, wasn't able to drive. She was sad. And maybe this wasn't the time to get angry. Oedipa had something of a position of privilege; she had just come through a dozen solid months with the murder of one man and a whole quarter of a century of acquaintance with that same man, when the only memory she could derive was of his unholy ghost that seemed to hide away as often as it was borne up to meet it. Was that at all real, her itchy sense that somebody was out there who wasn't quite supposed to be there, trailing slowly across the sun-kissed fields of lowlands and behind the straight and narrow lanes of what appeared to be an English village?

What happened in the fall of 1966 and early 1967 wasn't the best of times, but if she had to go by only the state's case and two sworn affidavits, it was bad, she thought, with things festering under her. I think it's worse here than when I came here, she said herself, and then shifted her viewpoint; people are really bad here, people are all over the map, and yet she sees them all the same. It could have all been better, she thought. "He was even worse than he was before," she thought, "and I'm the mother of his child." I really wish that he were here, she said, and felt a rumbling in her, remembering and disbelieving this last sentence, old hat.

By the time she finished cooking breakfast the next morning, she felt a familiar course of fatigue (she knew this feeling, had felt it before and gone through it) but not quite because of anything she'd done. It was the same stuff of the weeks gone by. By three o'clock in the afternoon she was feverishly studying the latest installment in the The Winnipeg Free Press and wondering, if she was going to have another baby, what time they were going to have the baby. Later, after Wendell had gone to bed, and she had fallen into a restless, frenzied sleep, it became clear that those thoughts had been heading away from the child, toward her father. Her husband was supposed to come out and hug her one more time. And something strange happened: no hug.

Part 1

Fifteen-thirty on that sunny October day in the early nineteen-seventies in Lake of the Woods, an hour's drive from Indiana, was a normal day. It was a typical county-issue late afternoon: a burst of snow, mostly covering the ground at eight or nine o'clock; an overweight man riding a first-class brown ticket train, in cotton dress and gold plate badge, who was carrying a sapphire metallic briefcase into St. Martinville, Oklahoma City; he stood before the crescent-shaped office of the Mulberry County Auditor; one of those large stainless steel doors opened and a ruggedly handsome man walked out in his tan, easy-looking suit, without his outer appearance a warning sign to any observer. The man was Wendell Sams, chief of police of Mulberry County, and his name was Earl Sams.

Earl Sams had been a cop for nineteen years. He'd been born on this farm in 1917 and made it into adulthood with farm-yard kinbaku and wide experience of the milieu of farmers' wives, country festivals, "cutesy songs and melodies and songs of the land," hunting a pig in a Louisiana cotton field, a hiker frolicking with a deer in the Ozark hills, living in two houses together, raising and maintaining eighty-seven kids, three cars, two planes, and a private railroad and a utility truck. ("It wasn't very good farming and it wasn't very good trucking, but it was only ten miles from downtown Atlanta," Earl Sams had once said.) Then there was the acreage; old-school equipment, all sorts of "pedestrian carts," tailgates and fountains of soft drinks, canned goods, canned food, that dreaded kimchi corn.

When Earl Sams came along, the town was failing slowly. the factory and mill district around St. Martinville had been neglected, falling on hard times after the corn companies had shut their doors. It had two hospitals and a sheriff's office, but the infant would have to wait, it would have to wait. Nobody wanted to move into the county to take advantage of the increased area that Sams had planned to buy with the money he had just received in his current job. The road was lined with ranch houses and pulled up to many of them by the local back roads, where in the summer, with the grass growing deep and fast, they could have gravel runways that stretched over miles of dirt. There were a couple of old county airfields which now had strip lights and power lines just to the north. The county used to have a train depot in the 1920s, but the local farmer's crew did not like to travel from Kansas City to Hickman or Chapel Hill and start their shift, so in order to stay in business, the depot had been razed, leaving nothing but a fence and a ditto house.

AWS P3 output sample:

When Oedipa got home she cried, didn't know how much, for many long minutes, and then came up with that often enough, two people's wanton moments being too much. Wendell lay across the back seat of her car and was sitting up when she arrived home. She started holding him more carefully, so as not to bump his head on the window, and almost drifted off until she could think of little else. "In other words, I won't remember my husband's name, Wendell," she told herself.

"Or maybe it won't be him, and I'll know who died." She opened her window and sat down with her arms on the steering wheel and turned the key. "I won't be able to take it, won't be able to get there."

She lost two hours, wasn't able to drive. She was sad. And maybe this wasn't the time to get angry. Oedipa had something of a position of privilege; she had just come through a dozen solid months with the murder of one man and a whole quarter of a century of acquaintance with that same man, when the only memory she could derive was of his unholy ghost that seemed to hide away as often as it was borne up to meet it. Was that at all real, her itchy sense that somebody was out there who wasn't quite supposed to be there, trailing slowly across the sun-kissed fields of lowlands and behind the straight and narrow lanes of what appeared to be an English village?

What happened in the fall of 1966 and early 1967 wasn't the best of times, but if she had to go by only the state's case and two sworn affidavits, it was bad, she thought, with things festering under her. I think it's worse here than when I came here, she said herself, and then shifted her viewpoint; people are really bad here, people are all over the map, and yet she sees them all the same. It could have all been better, she thought. "He was even worse than he was before," she thought, "and I'm the mother of his child." I really wish that he were here, she said, and felt a rumbling in her, remembering and disbelieving this last sentence, old hat.

By the time she finished cooking breakfast the next morning, she felt a familiar course of fatigue (she knew this feeling, had felt it before and gone through it) but not quite because of anything she'd done. It was the same stuff of the weeks gone by. By three o'clock in the afternoon she was feverishly studying the latest installment in the The Winnipeg Free Press and wondering, if she was going to have another baby, what time they were going to have the baby. Later, after Wendell had gone to bed, and she had fallen into a restless, frenzied sleep, it became clear that those thoughts had been heading away from the child, toward her father. Her husband was supposed to come out and hug her one more time. And something strange happened: no hug.

Part 1

Fifteen-thirty on that sunny October day in the early nineteen-seventies in Lake of the Woods, an hour's drive from Indiana, was a normal day. It was a typical county-issue late afternoon: a burst of snow, mostly covering the ground at eight or nine o'clock; an overweight man riding a first-class brown ticket train, in cotton dress and gold plate badge, who was carrying a sapphire metallic briefcase into St. Martinville, Oklahoma City; he stood before the crescent-shaped office of the Mulberry County Auditor; one of those large stainless steel doors opened and a ruggedly handsome man walked out in his tan, easy-looking suit, without his outer appearance a warning sign to any observer. The man was Wendell Sams, chief of police of Mulberry County, and his name was Earl Sams.

Earl Sams had been a cop for nineteen years. He'd been born on this farm in 1917 and made it into adulthood with farm-yard kinbaku and wide experience of the milieu of farmers' wives, country festivals, "cutesy songs and melodies and songs of the land," hunting the dawn and the sunset, among a horticultural and industrial past that, through mites and lies, was dressed up as simple piety, tradition, or familial environment. He was in his fifties, so that he looked little of his twenty-nine years and how well he matched the image of old-time independence he had been given. He'd been a lieutenant in the Corps of Coroners of Pulaski County, Arkansas, and then-run and more or less ad-hoc detective lieutenant on Davenport Road, for a short time in Lake County. The Mulberry County office seemed to be an unchanged place, although new signs had gone up informing the new chief that there would be a transfer. And yet Earl Sams liked it here, liked the cabin, the lake and the horses, and when the announcement was made that it would be his desk, he could just have stayed.

His first words as chief were flat, indifferent. "To whom it may concern, sir," he said, "my name is Earl Sams."

He had a wife, Rosalie, and they had four children; four boys and two girls. The boys were all little, and a couple of them were to be would-be sons-in-law. The eldest of them was fourteen and half; he was usually called Denton, for something and something else; and one of the girls was sixteen, a girl who had held her own in the boys' club, although now he could't really remember any one there. The boys had always liked him, laughed at him when he played catch with them, had found him amusing to be around. He liked them all, too, although when they were adults he didn't bother to find out their names, their names, their ages or their ages' dates. They were all small and stocky, tall but skinny, and bright. They were working people, but they kept his life in the background and seemed neither to get for him the unnecessary attention nor to get annoyed when he took time out to buy the place a new color and put up new signs. He bought the land himself, with luck, in an estate in Manitoba or Kansas or Tennessee, and he kept busy with old times, with horses and the crew, mending camp and clearing the woods and digging the drift and sharpening the bow, or tracking down grubs and other insects. The crew consisted of nothing but horses: two black from Arkansas, two Indian, one old pit bull. They tended the corn and lifted pigs and other sheep, took care of the cows, mopped up the mud, raised the chickens and pulled in the fertilizer. The younger brother was kind, handy with tools, and the father was a man with a truck and a radio. He used it and offered as much service as he could. He was tall, lean and clean-cut, with the deeply cool eyes of an oil-field man or two and a shrewdly-willed attitude about his job. As Chief Sams he was even a little hard to like, not unlike the older residents of Mulberry, but it wasn't his fault. Old men make mistakes, and it was no fault of theirs.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.