DirectML error when input is longer than half of the maximum context length

[jackylu0124]

I am running the Phi-3-mini-4k-instruct-onnx model on DirectML with the example phi3-qa.py script (https://github.com/microsoft/onnxruntime-genai/blob/main/examples/python/phi3-qa.py), and the only changes that I have made to the script is that I commented out the if-block setting search_options['max_length'] = 2048 so that it uses the maximum context length supported by the model (4096 in this case) by default. I encountered the issue when the input is longer than 2048 tokens, and more specifically, the following is my input prompt, and the error logs:

Input Prompt:

My sister, Mrs. Joe Gargery, was more than twenty years older than I, and had established a great reputation with herself and the neighbours because she had brought me up “by hand.” Having at that time to find out for myself what the expression meant, and knowing her to have a hard and heavy hand, and to be much in the habit of laying it upon her husband as well as upon me, I supposed that Joe Gargery and I were both brought up by hand.\n\nShe was not a good-looking woman, my sister; and I had a general impression that she must have made Joe Gargery marry her by hand. Joe was a fair man, with curls of flaxen hair on each side of his smooth face, and with eyes of such a very undecided blue that they seemed to have somehow got mixed with their own whites. He was a mild, good-natured, sweet-tempered, easy-going, foolish, dear fellow,—a sort of Hercules in strength, and also in weakness.\n\nMy sister, Mrs. Joe, with black hair and eyes, had such a prevailing redness of skin that I sometimes used to wonder whether it was possible she washed herself with a nutmeg-grater instead of soap. She was tall and bony, and almost always wore a coarse apron, fastened over her figure behind with two loops, and having a square impregnable bib in front, that was stuck full of pins and needles. She made it a powerful merit in herself, and a strong reproach against Joe, that she wore this apron so much. Though I really see no reason why she should have worn it at all; or why, if she did wear it at all, she should not have taken it off, every day of her life.\n\nJoe’s forge adjoined our house, which was a wooden house, as many of the dwellings in our country were,—most of them, at that time. When I ran home from the churchyard, the forge was shut up, and Joe was sitting alone in the kitchen. Joe and I being fellow-sufferers, and having confidences as such, Joe imparted a confidence to me, the moment I raised the latch of the door and peeped in at him opposite to it, sitting in the chimney corner.\n\n“Mrs. Joe has been out a dozen times, looking for you, Pip. And she’s out now, making it a baker’s dozen.”\n\n“Is she?”\n\n“Yes, Pip,” said Joe; “and what’s worse, she’s got Tickler with her.”\n\nAt this dismal intelligence, I twisted the only button on my waistcoat round and round, and looked in great depression at the fire. Tickler was a wax-ended piece of cane, worn smooth by collision with my tickled frame.\n\n“She sot down,” said Joe, “and she got up, and she made a grab at Tickler, and she Ram-paged out. That’s what she did,” said Joe, slowly clearing the fire between the lower bars with the poker, and looking at it; “she Ram-paged out, Pip.”\n\n“Has she been gone long, Joe?” I always treated him as a larger species of child, and as no more than my equal.\n\n“Well,” said Joe, glancing up at the Dutch clock, “she’s been on the Ram-page, this last spell, about five minutes, Pip. She’s a-coming! Get behind the door, old chap, and have the jack-towel betwixt you.”\n\nI took the advice. My sister, Mrs. Joe, throwing the door wide open, and finding an obstruction behind it, immediately divined the cause, and applied Tickler to its further investigation. She concluded by throwing me—I often served as a connubial missile—at Joe, who, glad to get hold of me on any terms, passed me on into the chimney and quietly fenced me up there with his great leg.\n\n“Where have you been, you young monkey?” said Mrs. Joe, stamping her foot. “Tell me directly what you’ve been doing to wear me away with fret and fright and worrit, or I’d have you out of that corner if you was fifty Pips, and he was five hundred Gargerys.”\n\n“I have only been to the churchyard,” said I, from my stool, crying and rubbing myself.\n\n“Churchyard!” repeated my sister. “If it warn’t for me you’d have been to the churchyard long ago, and stayed there. Who brought you up by hand?”\n\n“You did,” said I.\n\n“And why did I do it, I should like to know?” exclaimed my sister.\n\nI whimpered, “I don’t know.”\n\n“I don’t!” said my sister. “I’d never do it again! I know that. I may truly say I’ve never had this apron of mine off since born you were. It’s bad enough to be a blacksmith’s wife (and him a Gargery) without being your mother.”\n\nMy thoughts strayed from that question as I looked disconsolately at the fire. For the fugitive out on the marshes with the ironed leg, the mysterious young man, the file, the food, and the dreadful pledge I was under to commit a larceny on those sheltering premises, rose before me in the avenging coals.\n\n“Hah!” said Mrs. Joe, restoring Tickler to his station. “Churchyard, indeed! You may well say churchyard, you two.” One of us, by the by, had not said it at all. “You’ll drive me to the churchyard betwixt you, one of these days, and O, a pr-r-recious pair you’d be without me!”\n\nAs she applied herself to set the tea-things, Joe peeped down at me over his leg, as if he were mentally casting me and himself up, and calculating what kind of pair we practically should make, under the grievous circumstances foreshadowed. After that, he sat feeling his right-side flaxen curls and whisker, and following Mrs. Joe about with his blue eyes, as his manner always was at squally times.\n\nMy sister had a trenchant way of cutting our bread and butter for us, that never varied. First, with her left hand she jammed the loaf hard and fast against her bib,—where it sometimes got a pin into it, and sometimes a needle, which we afterwards got into our mouths. Then she took some butter (not too much) on a knife and spread it on the loaf, in an apothecary kind of way, as if she were making a plaster,—using both sides of the knife with a slapping dexterity, and trimming and moulding the butter off round the crust. Then, she gave the knife a final smart wipe on the edge of the plaster, and then sawed a very thick round off the loaf: which she finally, before separating from the loaf, hewed into two halves, of which Joe got one, and I the other.\n\nOn the present occasion, though I was hungry, I dared not eat my slice. I felt that I must have something in reserve for my dreadful acquaintance, and his ally the still more dreadful young man. I knew Mrs. Joe’s housekeeping to be of the strictest kind, and that my larcenous researches might find nothing available in the safe. Therefore I resolved to put my hunk of bread and butter down the leg of my trousers.\n\nThe effort of resolution necessary to the achievement of this purpose I found to be quite awful. It was as if I had to make up my mind to leap from the top of a high house, or plunge into a great depth of water. And it was made the more difficult by the unconscious Joe. In our already-mentioned freemasonry as fellow-sufferers, and in his good-natured companionship with me, it was our evening habit to compare the way we bit through our slices, by silently holding them up to each other’s admiration now and then,—which stimulated us to new exertions. To-night, Joe several times invited me, by the display of his fast diminishing slice, to enter upon our usual friendly competition; but he found me, each time, with my yellow mug of tea on one knee, and my untouched bread and butter on the other. At last, I desperately considered that the thing I contemplated must be done, and that it had best be done in the least improbable manner consistent with the circumstances. I took advantage of a moment when Joe had just looked at me, and got my bread and butter down my leg.\n\nJoe was evidently made uncomfortable by what he supposed to be my loss of appetite, and took a thoughtful bite out of his slice, which he didn’t seem to enjoy. He turned it about in his mouth much longer than usual, pondering over it a good deal, and after all gulped it down like a pill. He was about to take another bite, and had just got his head on one side for a good purchase on it, when his eye fell on me, and he saw that my bread and butter was gone.\n\nThe wonder and consternation with which Joe stopped on the threshold of his bite and stared at me, were too evident to escape my sister’s observation.\n\n“What’s the matter now?” said she, smartly, as she put down her cup.\n\n“I say, you know!” muttered Joe, shaking his head at me in very serious remonstrance. “Pip, old chap! You’ll do yourself a mischief. It’ll stick somewhere. You can’t have chawed it, Pip.”\n\n“What’s the matter now?” repeated my sister, more sharply than before.\n\n“If you can cough any trifle on it up, Pip, I’d recommend you to do it,” said Joe, all aghast. “Manners is manners, but still your elth’s your elth.”\n\nBy this time, my sister was quite desperate, so she pounced on Joe, and, taking him by the two whiskers, knocked his head for a little while against the wall behind him, while I sat in the corner, looking guiltily on.\n\n“Now, perhaps you’ll mention what’s the matter,” said my sister, out of breath, “you staring great stuck pig.”\n\nJoe looked at her in a helpless way, then took a helpless bite, and looked at me again.\n\n“You know, Pip,” said Joe, solemnly, with his last bite in his cheek, and speaking in a confidential voice, as if we two were quite alone, “you and me is always friends, and I’d be the last to tell upon you, any time. But such a—” he moved his chair and looked about the floor between us, and then again at me—“such a most oncommon Bolt as that!”\n\n“Been bolting his food, has he?” cried my sister.\n\n“You know, old chap,” said Joe, looking at me, and not at Mrs. Joe, with his bite still in his cheek, “I Bolted, myself, when I was your age—frequent—and as a boy I’ve been among a many Bolters; but I never see your Bolting equal yet, Pip, and it’s a mercy you ain’t Bolted dead.”\n

Error Logs (parts of the exact paths are replaced with xxx for privacy reasons):

PowerShell 7.4.2
PS D:\xxx\xxx\xxx\onnx_test> python phi3-qa.py -m .\Phi-3-mini-4k-instruct-onnx\directml\directml-int4-awq-block-128
Input: My sister, Mrs. Joe Gargery, was more than twenty years older than I, and had established a great reputation with herself and the neighbours because she had brought me up “by hand.” Having at that time to find out for myself what the expression meant, and knowing her to have a hard and heavy hand, and to be much in the habit of laying it upon her husband as well as upon me, I supposed that Joe Gargery and I were both brought up by hand.\n\nShe was not a good-looking woman, my sister; and I had a general impression that she must have made Joe Gargery marry her by hand. Joe was a fair man, with curls of flaxen hair on each side of his smooth face, and with eyes of such a very undecided blue that they seemed to have somehow got mixed with their own whites. He was a mild, good-natured, sweet-tempered, easy-going, foolish, dear fellow,—a sort of Hercules in strength, and also in weakness.\n\nMy sister, Mrs. Joe, with black hair and eyes, had such a prevailing redness of skin that I sometimes used to wonder whether it was possible she washed herself with a nutmeg-grater instead of soap. She was tall and bony, and almost always wore a coarse apron, fastened over her figure behind with two loops, and having a square impregnable bib in front, that was stuck full of pins and needles. She made it a powerful merit in herself, and a strong reproach against Joe, that she wore this apron so much. Though I really see no reason why she should have worn it at all; or why, if she did wear it at all, she should not have taken it off, every day of her life.\n\nJoe’s forge adjoined our house, which was a wooden house, as many of the dwellings in our country were,—most of them, at that time. When I ran home from the churchyard, the forge was shut up, and Joe was sitting alone in the kitchen. Joe and I being fellow-sufferers, and having confidences as such, Joe imparted a confidence to me, the moment I raised the latch of the door and peeped in at him opposite to it, sitting in the chimney corner.\n\n“Mrs. Joe has been out a dozen times, looking for you, Pip. And she’s out now, making it a baker’s dozen.”\n\n“Is she?”\n\n“Yes, Pip,” said Joe; “and what’s worse, she’s got Tickler with her.”\n\nAt this dismal intelligence, I twisted the only button on my waistcoat round and round, and looked in great depression at the fire. Tickler was a wax-ended piece of cane, worn smooth by collision with my tickled frame.\n\n“She sot down,” said Joe, “and she got up, and she made a grab at Tickler, and she Ram-paged out. That’s what she did,” said Joe, slowly clearing the fire between the lower bars with the poker, and looking at it; “she Ram-paged out, Pip.”\n\n“Has she been gone long, Joe?” I always treated him as a larger species of child, and as no more than my equal.\n\n“Well,” said Joe, glancing up at the Dutch clock, “she’s been on the Ram-page, this last spell, about five minutes, Pip. She’s a-coming! Get behind the door, old chap, and have the jack-towel betwixt you.”\n\nI took the advice. My sister, Mrs. Joe, throwing the door wide open, and finding an obstruction behind it, immediately divined the cause, and applied Tickler to its further investigation. She concluded by throwing me—I often served as a connubial missile—at Joe, who, glad to get hold of me on any terms, passed me on into the chimney and quietly fenced me up there with his great leg.\n\n“Where have you been, you young monkey?” said Mrs. Joe, stamping her foot. “Tell me directly what you’ve been doing to wear me away with fret and fright and worrit, or I’d have you out of that corner if you was fifty Pips, and he was five hundred Gargerys.”\n\n“I have only been to the churchyard,” said I, from my stool, crying and rubbing myself.\n\n“Churchyard!” repeated my sister. “If it warn’t for me you’d have been to the churchyard long ago, and stayed there. Who brought you up by hand?”\n\n“You did,” said I.\n\n“And why did I do it, I should like to know?” exclaimed my sister.\n\nI whimpered, “I don’t know.”\n\n“I don’t!” said my sister. “I’d never do it again! I know that. I may truly say I’ve never had this apron of mine off since born you were. It’s bad enough to be a blacksmith’s wife (and him a Gargery) without being your mother.”\n\nMy thoughts strayed from that question as I looked disconsolately at the fire. For the fugitive out on the marshes with the ironed leg, the mysterious young man, the file, the food, and the dreadful pledge I was under to commit a larceny on those sheltering premises, rose before me in the avenging coals.\n\n“Hah!” said Mrs. Joe, restoring Tickler to his station. “Churchyard, indeed! You may well say churchyard, you two.” One of us, by the by, had not said it at all. “You’ll drive me to the churchyard betwixt you, one of these days, and O, a pr-r-recious pair you’d be without me!”\n\nAs she applied herself to set the tea-things, Joe peeped down at me over his leg, as if he were mentally casting me and himself up, and calculating what kind of pair we practically should make, under the grievous circumstances foreshadowed. After that, he sat feeling his right-side flaxen curls and whisker, and following Mrs. Joe about with his blue eyes, as his manner always was at squally times.\n\nMy sister had a trenchant way of cutting our bread and butter for us, that never varied. First, with her left hand she jammed the loaf hard and fast against her bib,—where it sometimes got a pin into it, and sometimes a needle, which we afterwards got into our mouths. Then she took some butter (not too much) on a knife and spread it on the loaf, in an apothecary kind of way, as if she were making a plaster,—using both sides of the knife with a slapping dexterity, and trimming and moulding the butter off round the crust. Then, she gave the knife a final smart wipe on the edge of the plaster, and then sawed a very thick round off the loaf: which she finally, before separating from the loaf, hewed into two halves, of which Joe got one, and I the other.\n\nOn the present occasion, though I was hungry, I dared not eat my slice. I felt that I must have something in reserve for my dreadful acquaintance, and his ally the still more dreadful young man. I knew Mrs. Joe’s housekeeping to be of the strictest kind, and that my larcenous researches might find nothing available in the safe. Therefore I resolved to put my hunk of bread and butter down the leg of my trousers.\n\nThe effort of resolution necessary to the achievement of this purpose I found to be quite awful. It was as if I had to make up my mind to leap from the top of a high house, or plunge into a great depth of water. And it was made the more difficult by the unconscious Joe. In our already-mentioned freemasonry as fellow-sufferers, and in his good-natured companionship with me, it was our evening habit to compare the way we bit through our slices, by silently holding them up to each other’s admiration now and then,—which stimulated us to new exertions. To-night, Joe several times invited me, by the display of his fast diminishing slice, to enter upon our usual friendly competition; but he found me, each time, with my yellow mug of tea on one knee, and my untouched bread and butter on the other. At last, I desperately considered that the thing I contemplated must be done, and that it had best be done in the least improbable manner consistent with the circumstances. I took advantage of a moment when Joe had just looked at me, and got my bread and butter down my leg.\n\nJoe was evidently made uncomfortable by what he supposed to be my loss of appetite, and took a thoughtful bite out of his slice, which he didn’t seem to enjoy. He turned it about in his mouth much longer than usual, pondering over it a good deal, and after all gulped it down like a pill. He was about to take another bite, and had just got his head on one side for a good purchase on it, when his eye fell on me, and he saw that my bread and butter was gone.\n\nThe wonder and consternation with which Joe stopped on the threshold of his bite and stared at me, were too evident to escape my sister’s observation.\n\n“What’s the matter now?” said she, smartly, as she put down her cup.\n\n“I say, you know!” muttered Joe, shaking his head at me in very serious remonstrance. “Pip, old chap! You’ll do yourself a mischief. It’ll stick somewhere. You can’t have chawed it, Pip.”\n\n“What’s the matter now?” repeated my sister, more sharply than before.\n\n“If you can cough any trifle on it up, Pip, I’d recommend you to do it,” said Joe, all aghast. “Manners is manners, but still your elth’s your elth.”\n\nBy this time, my sister was quite desperate, so she pounced on Joe, and, taking him by the two whiskers, knocked his head for a little while against the wall behind him, while I sat in the corner, looking guiltily on.\n\n“Now, perhaps you’ll mention what’s the matter,” said my sister, out of breath, “you staring great stuck pig.”\n\nJoe looked at her in a helpless way, then took a helpless bite, and looked at me again.\n\n“You know, Pip,” said Joe, solemnly, with his last bite in his cheek, and speaking in a confidential voice, as if we two were quite alone, “you and me is always friends, and I’d be the last to tell upon you, any time. But such a—” he moved his chair and looked about the floor between us, and then again at me—“such a most oncommon Bolt as that!”\n\n“Been bolting his food, has he?” cried my sister.\n\n“You know, old chap,” said Joe, looking at me, and not at Mrs. Joe, with his bite still in his cheek, “I Bolted, myself, when I was your age—frequent—and as a boy I’ve been among a many Bolters; but I never see your Bolting equal yet, Pip, and it’s a mercy you ain’t Bolted dead.”\n

Output: 2024-05-30 12:36:11.9182626 [E:onnxruntime:onnxruntime-genai, sequential_executor.cc:516 onnxruntime::ExecuteKernel] Non-zero status code returned while running DmlFusedNode_0_0 node. Name:'DmlFusedNode_0_0' Status Message: D:\a\_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\DmlGraphFusionHelper.cpp(1060)\onnxruntime.dll!00007FFB2A8BA0B1: (caller: 00007FFB2A948F0B) Exception(2) tid(2e8c) 887A0001 The application made a call that is invalid. Either the parameters of the call or the state of some object was incorrect.
Enable the D3D debug layer in order to see details via debug messages.

Traceback (most recent call last):
  File "D:\xxx\xxx\xxx\onnx_test\phi3-qa.py", line 94, in <module>
    main(args)
  File "D:\xxx\xxx\xxx\onnx_test\phi3-qa.py", line 57, in main
    generator.compute_logits()
onnxruntime_genai.onnxruntime_genai.OrtException: Non-zero status code returned while running DmlFusedNode_0_0 node. Name:'DmlFusedNode_0_0' Status Message: D:\a\_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\DmlGraphFusionHelper.cpp(1060)\onnxruntime.dll!00007FFB2A8BA0B1: (caller: 00007FFB2A948F0B) Exception(2) tid(2e8c) 887A0001 The application made a call that is invalid. Either the parameters of the call or the state of some object was incorrect.
Enable the D3D debug layer in order to see details via debug messages.

I also checked these two issues, but they seem be different from the error that I am facing: https://github.com/microsoft/onnxruntime-genai/issues/424 https://github.com/microsoft/onnxruntime-genai/issues/521

Package Version: onnxruntime-genai-directml 0.3.0rc2 GPU: RTX 3090

[baijumeswani]

This issue should be resolved with the 0.3.0 release. I'll close this issue now. But please let us know if you still see the issue and we will re-open the investigation.

[jackylu0124]

I confirm that this issue is now resolved with the 0.3.0 release.