llama13b model generates abnormal results under multi-GPU multi-batch-size in tensorrt-llm 0.7.1

NVIDIA / TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.

Apache License 2.0

8.11k stars 896 forks source link

output_beams_list2: [[' How much money does Janet make from selling duck eggs at the'], [' A robe']] output_id: [2585, 1753, 3220, 1558, 53665, 1281, 504, 11236, 35985, 18805, 518, 279, 151855, 151855, ..., 151855] output_id: [362, 62619, 151855, 151855, ..., 151855, 151643, ..., 151643] #151643 is end_id and pad_id, 151855 is the last token id output_beams_list2: [[' Man'], [' How']]

output_beams_list2: [[" How much money does Janet make from selling eggs at the farmers' market?\nJanet eats 3 eggs every day and bakes 4 muffins every day so she uses 3 + 4 = <<3+4=7>>7 eggs every day.\nShe lays 16 eggs and uses 7 eggs so she has 16 - 7 = <<16-7=9>>9 eggs left over every day.\nShe sells 9 eggs every day at the farmers' market and they are $2 each so she makes 9 * 2 = $<<9*2=18>>18 per day.\n The answer is 18."]] output_beams_list2: [[' A robe requires 2 blue and 1/2 white, so 3 total bolts.\nIt takes 2 blue + 1/2 white = 3/2 = 1 1/2 total bolts. The answer is 1.5.']] output_beams_list2: [[' Man buys house for $80,000 and repairs it for $50,000. The value of the house increased by 150% after the repairs. How much profit did he make?\nThe value of the house increased by 150/100*$80,000=$<<150/100*80000=120000>>120,000.\nThe profit was $120,000-$80,000=$<<120000-80000=40000>>40,000.\n The answer is 40,000.']] output_beams_list2: [[" How many meters does James run in a week?\nHe runs 3 sprints a week so that's 3*60 = <<3*60=180>>180 meters\nHe runs 180 meters each time he sprints so that's 180*3 = <<180*3=540>>540 meters\nHe runs 540 meters a week so that's 540*7 = <<540*7=3780>>3780 meters\n The answer is 3780."]]

export LLAMA_MODEL=/home/scratch.trt_llm_data/llm-models/llama-models-v2/llama-v2-13b-hf/ python build.py --model_dir ${LLAMA_MODEL} --dtype float16 --use_gemm_plugin float16 --use_gpt_attention_plugin float16 --enable_context_fmha --remove_input_padding --output_dir /tmp/llama/llama-13b_tllm --world_size 2 --tp_size 2 --max_batch_size 4 --max_input_len 1024 --max_output_len 1024 mpirun -n 2 python ../summarize.py --test_trt_llm --hf_model_dir ${LLAMA_MODEL} --data_type fp16 --engine_dir /tmp/llama/llama-13b_tllm --batch_size 2 --log_level verbose

[01/15/2024-03:33:03] [TRT-LLM] [V] run data_point 38 ~ 40 [01/15/2024-03:33:03] [TRT-LLM] [V] The max_attention_window_size is not set, we will use max_seq_length by default. [01/15/2024-03:33:03] [TRT-LLM] [V] The max_attention_window_size is not set, we will use max_seq_length by default. [01/15/2024-03:33:04] [TRT-LLM] [V] ---------------------------------------------------------------------------------------------------- [01/15/2024-03:33:04] [TRT-LLM] [V] Input : ['Istanbul, Turkey (CNN)The questions Turks asked on Tuesday were tinged with fear. "What\'s going on? What happened? Why can\'t I get into the subway?" asked an elderly woman in a white headscarf with several shopping bags as she stood outside the barricaded entrance to one of Istanbul\'s busiest subway stations. She was one of millions of Turks left confused and concerned by the worst power outage to grip the country in more than a decade. Dozens of cities across Turkey lost power for hours on Tuesday. Millions of people were affected, including passengers stranded on paralyzed trains and subways. Municipal workers were forced to evacuate Istanbul\'s Marmaray Tunnel, where the black-out left commuters trapped deep beneath the rushing waters of the Bosphorus Strait. More than 24 hours later, Turkish officials were still at pains to explain the power outage. The energy minister suggested a possible failure in transmission lines. The prime minister did not rule out the possibility of a terror attack. The mysterious collapse of much of the country\'s energy grid triggered a burst of wild conspiracy theories across social media. Some Twitter users went so far as to suggest the black-outs were a warm up for elections scheduled to take place in June. There is fertile ground for rumor-mongering in Turkey. Over the last five years, security forces have arrested hundreds of army generals, journalists, prosecutors, civil society activists and police commanders and accused them of being members of assorted plots aimed at toppling the government. The government\'s increasingly heavy-handed repression of public dissent, combined with overt censorship of the media and the internet, have also contributed to a hyper-polarized and deeply mistrustful political atmosphere. Even Turkey\'s veteran deputy prime minister, Bulent Arinc, recently observed that opposition supporters now look at him "with hatred" rather than the grudging respect he enjoyed when his political party first swept to power in elections in 2002. Meanwhile, mysterious black-outs are a sore spot for some Turks, after a surreal 2014 incident on election night -- allegedly involving a feline saboteur. That is -- a cat that allegedly wandered into a power transformer. That was almost exactly a year ago, a smaller series of power outages affected some polling stations during nationwide municipal elections, prompting unsubstantiated accusations of vote rigging. The ruling Justice and Development Party ended up winning by a comfortable margin, but few Turks were reassured by the energy minister\'s explanation that the voting day black-outs were caused by a cat getting lost. As electricity was just starting to come back on in Istanbul on Tuesday, a second crisis erupted. Websites linked to an extremist leftist militant group known as the DHKP-C began publishing chilling photos of a masked man holding a pistol to the head of a hostage in front of communist flags. Two gunmen had somehow infiltrated the Palace of Justice, the monolithic court house in the center of Istanbul. There they took hostage Mehmet Selim Kiraz, the prosecutor in charge of one of the most politically sensitive trials in the country. The gunmen demanded the confessions of police officers accused of shooting a tear gas canister at Berkin Elvan, a 15-year-old boy who was critically wounded during anti-government protests that raged across Istanbul in 2013. The boy\'s death after months in a medically-induced coma triggered a fresh burst of protests and riots against the government. On Tuesday, in the midst of the hostage crisis at the court house, the Turkish government imposed a gag order banning broadcasters from reporting on the Palace of Justice siege. The broadcast ban is a measure that the Turkish government has repeatedly used in recent years to stifle reporting on deadly terrorist attacks. The government also famously shut down Twitter and YouTube in an effort to kill highly embarrassing political scandals involving corruption. Ultimately, Tuesday\'s court house siege ended in a deadly hail of bullets that left both gunmen dead and the prosecutor mortally wounded. Turkish officials say special forces raided the court house only after the militants began shooting. Online and in the streets, some Turks began linking the massive electricity blackouts to the hostage-taking inside one of Turkey\'s best-protected buildings, even though there is little to suggest the two incidents are connected. Just hours after the shooting, tensions exploded yet again at the court house. Istanbul\'s police chief had called for a press conference. As journalists jostled their way through security at the entrance to the largely deserted courthouse, some bystanders began chanting "government thieves." Just hours after a devastating lapse of security at the Palace of Justice, police began detaining the demonstrators hurling abuse at their elected government. The reactions to Tuesday\'s bewildering series of events revealed several truths about contemporary Turkey. The country is tense and confused after years of back-to-back political crises. Heavy-handed censorship has left the mainstream media widely distrusted and discredited by broad segments of society. And the absence of a common, credible space for sharing information has pushed critics of the government to the fringes of social media. Amid the burst of optimism and civil society activism in the early heady days of the Arab Spring in 2011, Turkey was often cited as a possible democratic model for countries in the Middle East. Many of those Arab countries have since descended into conflict, repression and instability. In the meantime, Turkey feels increasingly vulnerable to demons of its own making.', "Sanaa, Yemen (CNN)Al Qaeda fighters attacked a prison in the coastal Yemeni city of Al Mukallah early Thursday, freeing at least 270 prisoners, a third of whom have al Qaeda links, a senior Defense Ministry official has told CNN. Khaled Batarfi, a senior al Qaeda figure, was among the escapees, officials said. Dozens of attackers took control of government buildings, including the city's Central Prison, Central Bank and radio station during the assault early Thursday, according to officials. Government troops arrived early Thursday and clashed with the al Qaeda fighters, and most of the militants fled, the officials said. Last month, hundreds of inmates escaped from Al Mansoorah Central Prison in Aden after clashes between Shiite Houthi rebels and forces loyal to ousted Sunni President Abdu Rabu Mansour Hadi. Yemen has been descending into chaos in the weeks since Shiite Houthi rebels removed Hadi, a Sunni, from power. The sectarian nature of the conflict is drawing in regional rivals Saudi Arabia, which is predominately Sunni -- and is the country to which Hadi ultimately fled -- and Iran, which is predominately Shiite and supports the Houthi rebels. Because of that, the conflict in Yemen risks becoming a proxy war in the struggle between the Iranians and the Saudis for preeminence in the Middle East. The Saudis have conducted airstrikes against the Houthi rebels and could send in ground troops. But little is simple in the Middle East. And while the conflict between the Houthis and forces loyal to Hadi rages in the western part of the country, where it has caused hundreds of civilian deaths, al Qaeda in the Arabian Peninsula, or AQAP, controls parts of eastern Yemen. AQAP is considered one of the most ruthless branches of the terrorist organization."] [01/15/2024-03:33:04] [TRT-LLM] [V] TensorRT-LLM Output: [["Turkey is a mess.\nThe Turkish government has banned Twitter and YouTube after a recording of a top security meeting was leaked online. The ban was imposed after a recording of a top security meeting was leaked online. The recording, which was allegedly made by the head of Turkey's intelligence agency, was published on YouTube. It purportedly shows the head of Turkey's intelligence agency, Hakan Fidan, telling the foreign minister, Ahmet Davut"], ["Al Qaeda fighters attacked a prison in the coastal Yemeni city of Al Mukallah early Thursday, freeing at least 270 prisoners, a third of whom have al Qaeda links, a senior Defense Ministry official has told CNN. Khaled Batarfi, a senior al Qaeda figure, was among the escapees, officials said. Dozens of attackers took control of government buildings, including the city's"]] [01/15/2024-03:33:04] [TRT-LLM] [V] Reference : ['This week, Turkey was gripped by a massive power outage and a deadly hostage crisis .\nReactions reveal contemporary Turkey is tense and confused after years of political crises .\nCensorship has pushed critics to fringes in country cited as democratic model for Mideast .', 'Al Qaeda fighters attack a prison and other government buildings, freeing many prisoners .\nGovernment troops clash with the fighters, most of whom flee .\nYemen is descending into chaos as a Shia-Sunni conflict draws in regional rivals Saudi Arabia and Iran .'] [01/15/2024-03:33:04] [TRT-LLM] [I] TensorRT-LLM (total latency: 24.902254819869995 sec) [01/15/2024-03:33:04] [TRT-LLM] [I] TensorRT-LLM beam 0 result [01/15/2024-03:33:04] [TRT-LLM] [I] rouge1 : 18.639209970478824 [01/15/2024-03:33:04] [TRT-LLM] [I] rouge2 : 4.486236665518625 [01/15/2024-03:33:04] [TRT-LLM] [I] rougeL : 13.817401267734514 [01/15/2024-03:33:04] [TRT-LLM] [I] rougeLsum : 16.46296159716107

commit 80bc07510ac4ddf13c0d76ad295cdb2b75614618 (HEAD, tag: v0.7.1) Author: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com> Date: Tue Dec 26 19:42:17 2023 +0800 Update TensorRT-LLM Release branch (#745) * Update TensorRT-LLM --------- Co-authored-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

[01/15/2024-14:16:42] [TRT-LLM] [V] run data_point 2 ~ 4 [01/15/2024-14:16:42] [TRT-LLM] [V] The max_attention_window_size is not set, we will use max_seq_length by default. [01/15/2024-14:16:42] [TRT-LLM] [V] The max_attention_window_size is not set, we will use max_seq_length by default. [01/15/2024-14:17:00] [TRT-LLM] [V] ---------------------------------------------------------------------------------------------------- [01/15/2024-14:17:00] [TRT-LLM] [V] Input : ['(CNN)President Barack Obama took part in a roundtable discussion this week on climate change, refocusing on the issue from a public health vantage point. After the event at Washington\'s Howard University on Tuesday, Obama sat down with me for a one-on-one interview. I asked him about the science behind climate change and public health and the message he wants the average American to take away, as well as how enforceable his action plan is. Here are five things I learned: . The President enrolled at Occidental College in Los Angeles in 1979 (he transferred to Columbia University his junior year). While in L.A., he said, the air was so bad that it prevented him from running outside. He remembers the air quality alerts and how people with respiratory problems had to stay inside. He credits the Clean Air Act with making Americans "a lot" healthier, in addition to being able to "see the mountains in the background because they aren\'t covered in smog." Obama also said the instances of asthma and other respiratory diseases went down after these measures were taken. Peer-reviewed Environmental Protection Agency studies say that the Clean Air Act and subsequent amendments have reduced early deaths associated with exposure to ambient fine particle pollution and ozone, and reduced illnesses such as chronic bronchitis and acute myocardial infarction. The EPA estimates that, between 1970 and 2010, the act and its amendments prevented 365,000 early deaths from particulate matter alone. "No challenge poses more of a public threat than climate change," the President told me. When I asked about the strength of the science supporting the direct relationship between climate change and public health, he said, "We know as temperatures rise, insect-borne diseases potentially start shifting up. We know, in a very straight-forward fashion, that heatstroke and other heat-related illnesses and deaths potentially increase, and so what we\'re doing here is to make sure that in addition to public awareness around the potential for big storms like Hurricane Sandy or big wildfires or droughts, that people recognize there\'s a very personal, potential impact in climate change, and the good news is we can do something about it." In many ways, Obama is attempting to reframe the discussion around climate change as a public health issue that affects all of us, while conceding that we don\'t fully understand the magnitude of the correlation between rising temperatures and impact on human health. When asked what the average American can do about all this, the President encouraged ordinary citizens, doctors and nurses to start putting some pressure on elected officials "to try and make something happen to reduce the impacts of climate change." He also issued a presidential proclamation declaring April 6-12 as National Public Health Week "to better understand, communicate and reduce the health impacts of climate change on our communities." The average American can also do their part to reduce their own carbon footprint, including: . • Change your incandescent light bulbs to compact fluorescent lights. One CFL can reduce up to 1,300 pounds of carbon dioxide pollution during its lifetime. If every house in the U.S. switched its bulbs, we could reduce the electricity spent on lighting by half. • Unplug your gadgets and chargers when not in use. According to the U.S. Department of Energy, this practice can save $100 a year on your energy bill. • Use a laptop instead of a desktop. Laptops are designed to be energy-efficient, because battery life is a major factor in their design. According to Energy Star, a laptop can be up to 80% more energy-efficient than a desktop. • Filter your own water. Beyond the environmental toll of plastic waste, consider just how far your water was transported before you bought it at the grocery store. • Adjust your curtains and thermostats. If you keep your house 2 degrees warmer in the summer and 2 degrees colder in the winter, you can save big bucks on your energy bill. The Department of Energy estimates you can save up to 15% on your bill by turning off your thermostat when you\'re not at home. Obama did not appear particularly concerned about the current Supreme Court challenge to the Affordable Care Act. He said he believes the statute is "clear and straightforward." He said, "I am not anticipating the Supreme Court would make such a bad decision." At issue is the 32 states that did not set up their own health care exchanges and left it to the federal government to do so. The plaintiffs in the lawsuit contend that the language of the Affordable Care Act does not allow for tax subsidies in those states (without state-based exchanges), possibly creating a situation, for example, in which people in Massachusetts would receive a tax credit, but people living in Texas would not. Obama did tell me that if the Supreme Court challenge is upheld, however, there is no Plan B. "Millions of people would lose their health insurance. They would no longer be able to afford the health insurance that\'s being provided out there." Obama went on to say, "I think this is the last gasp of folks who have been fighting against [the Affordable Care Act] for ideological reasons." He told me that he "gets letters every day from people who say, \'you know what, the Affordable Care Act saved my life or saved my kid\'s life because I got insurance.\' \'I thought I was healthy; turns out I had a tumor, but because I went and got a checkup, it was removed in time, and I\'m now cancer-free.\' " He added, "I think stories like that will be factored in when the Supreme Court takes a look at this case." CNN\'s Ben Tinker contributed to this report.', 'Moscow (CNN)A Russian TV channel aired Hillary Clinton\'s first campaign video with a rating stamp that means it\'s for mature audiences, because of fears it might run afoul of the country\'s anti-gay propaganda law. A clip of the video, which features a gay couple holding hands, got the 18+ rating from the independent TV Rain channel in Russia on Monday. The channel told CNN that it didn\'t want to break the controversial law, which bans "propaganda of nontraditional sexual relations around minors" and bars public discussion of gay rights and relationships within earshot of children. "There are no legal precedents for this law, so we just don\'t know what comes under this law and (what) doesn\'t," a TV Rain spokesperson told CNN. "Therefore, fearing to break the law -- especially given the high attention to TV Rain from the supervising authorities -- we decided to put a marker (on the video)." Clinton\'s video was released over the weekend to announce the start of her 2016 U.S. presidential campaign. It features about five seconds of two men holding hands. One of the men says, "I\'m getting married this summer to someone I really care about." The former senator and first lady first declared her support for same-sex marriage in early 2013, saying that "gay rights are human rights, and human rights are gay rights." Russia\'s controversial law caused an international outcry after it was passed by the Russian Parliament and signed by President Vladimir Putin in July 2013. Human Rights Watch described the anti-gay propaganda law as "a profoundly discriminatory and dangerous bill that is bound to worsen homophobia in Russia." Rights campaigners called for a boycott of the 2014 Winter Olympics in Sochi, and a number of bars around the world stopped serving Russian vodka in protest. U.S. President Barack Obama -- Clinton\'s former boss -- said at the time that he found the legislation offensive. "I have no patience for countries that try to treat gays or lesbians or transgendered persons in ways that intimidate them or harmful to them," Obama told Jay Leno in 2013. Putin defended the law, noting that unlike other countries, Russia decriminalized homosexual relationships (in 1993). "We don\'t outlaw anything and don\'t nab anyone," he said before the 2014 Games. "That\'s why you can feel safe and free here," he added, "but please leave our children in peace." The rights group ILGA-Europe said in a May 2014 report that Russia was the worst place in Europe (out of 49 countries) for LGBTI people to live. READ MORE: Social media react to Hillary Clinton logo .'] [01/15/2024-14:17:00] [TRT-LLM] [V] TensorRT-LLM Output: [['Ob给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给'], ['Hill给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给']] [01/15/2024-14:17:00] [TRT-LLM] [V] Reference : ['"No challenge poses more of a public threat than climate change," the President says .\nHe credits the Clean Air Act with making Americans "a lot" healthier .', "Presidential hopeful's video, featuring gay couple, gets mature rating in Russia .\nRussian TV channel feared airing it would break the country's anti-gay propaganda law .\nClinton announced her support for same-sex marriage in 2013 ."]

I did not use the Llama-2-13b-hf model to reproduce the problem because the Llama-2-13b-hf model with output_len 1024 has a duplicate generation problem on this dataset. Now I have tried again and found that the same problem can be replicated for the Lama-2-13B-HF model with multiple batches.

the build cmd line: python build.py --model_dir /Llama-2-13b-hf --dtype float16 --use_gemm_plugin float16 --use_gpt_attention_plugin float16 --enable_context_fmha --remove_input_padding --output_dir /Llama-2-13b-hf_tp2 --world_size 2 --tp_size 2 --max_batch_size 4 --max_input_len 1024 --max_output_len 1024

the run cmd line: mpirun -n 2 --allow-run-as-root python ../summarize.py --test_trt_llm --hf_model_dir /Llama-2-13b-hf --data_type fp16 --engine_dir /Llama-2-13b-hf_tp2 --batch_size 2 --output_len 1024 --log_level verbose

here is the abnormal generate results with '--batch_size 2':

[01/17/2024-09:43:07] [TRT-LLM] [V] run data_point 2 ~ 4
[01/17/2024-09:43:07] [TRT-LLM] [V] The max_attention_window_size is not set, we will use max_seq_length by default.
[01/17/2024-09:43:25] [TRT-LLM] [V] ----------------------------------------------------------------------------------------------------
[01/17/2024-09:43:25] [TRT-LLM] [V] Input : ['(CNN)President Barack Obama took part in a roundtable discussion this week on climate change, refocusing on the issue from a public health vantage point. After the event at Washington\'s Howard University on Tuesday, Obama sat down with me for a one-on-one interview. I asked him about the science behind climate change and public health and the message he wants the average American to take away, as well as how enforceable his action plan is. Here are five things I learned: . The President enrolled at Occidental College in Los Angeles in 1979 (he transferred to Columbia University his junior year). While in L.A., he said, the air was so bad that it prevented him from running outside. He remembers the air quality alerts and how people with respiratory problems had to stay inside. He credits the Clean Air Act with making Americans "a lot" healthier, in addition to being able to "see the mountains in the background because they aren\'t covered in smog." Obama also said the instances of asthma and other respiratory diseases went down after these measures were taken. Peer-reviewed Environmental Protection Agency studies say that the Clean Air Act and subsequent amendments have reduced early deaths associated with exposure to ambient fine particle pollution and ozone, and reduced illnesses such as chronic bronchitis and acute myocardial infarction. The EPA estimates that, between 1970 and 2010, the act and its amendments prevented 365,000 early deaths from particulate matter alone. "No challenge poses more of a public threat than climate change," the President told me. When I asked about the strength of the science supporting the direct relationship between climate change and public health, he said, "We know as temperatures rise, insect-borne diseases potentially start shifting up. We know, in a very straight-forward fashion, that heatstroke and other heat-related illnesses and deaths potentially increase, and so what we\'re doing here is to make sure that in addition to public awareness around the potential for big storms like Hurricane Sandy or big wildfires or droughts, that people recognize there\'s a very personal, potential impact in climate change, and the good news is we can do something about it." In many ways, Obama is attempting to reframe the discussion around climate change as a public health issue that affects all of us, while conceding that we don\'t fully understand the magnitude of the correlation between rising temperatures and impact on human health. When asked what the average American can do about all this, the President encouraged ordinary citizens, doctors and nurses to start putting some pressure on elected officials "to try and make something happen to reduce the impacts of climate change." He also issued a presidential proclamation declaring April 6-12 as National Public Health Week "to better understand, communicate and reduce the health impacts of climate change on our communities." The average American can also do their part to reduce their own carbon footprint, including: . • Change your incandescent light bulbs to compact fluorescent lights. One CFL can reduce up to 1,300 pounds of carbon dioxide pollution during its lifetime. If every house in the U.S. switched its bulbs, we could reduce the electricity spent on lighting by half. • Unplug your gadgets and chargers when not in use. According to the U.S. Department of Energy, this practice can save $100 a year on your energy bill. • Use a laptop instead of a desktop. Laptops are designed to be energy-efficient, because battery life is a major factor in their design. According to Energy Star, a laptop can be up to 80% more energy-efficient than a desktop. • Filter your own water. Beyond the environmental toll of plastic waste, consider just how far your water was transported before you bought it at the grocery store. • Adjust your curtains and thermostats. If you keep your house 2 degrees warmer in the summer and 2 degrees colder in the winter, you can save big bucks on your energy bill. The Department of Energy estimates you can save up to 15% on your bill by turning off your thermostat when you\'re not at home. Obama did not appear particularly concerned about the current Supreme Court challenge to the Affordable Care Act. He  said he believes the statute is "clear and straightforward." He said, "I am not anticipating the Supreme Court would make such a bad decision." At issue is the 32 states that did not set up their own health care exchanges and left it to the federal government to do so. The plaintiffs in the lawsuit contend that the language of the Affordable Care Act does not allow for tax subsidies in those states (without state-based exchanges), possibly creating a situation, for example, in which people in Massachusetts would receive a tax credit, but people living in Texas would not. Obama did tell me that if the Supreme Court challenge is upheld, however, there is no Plan B. "Millions of people would lose their health insurance. They would no longer be able to afford the health insurance that\'s being provided out there." Obama went on to say, "I think this is the last gasp of folks who have been fighting against [the Affordable Care Act] for ideological reasons." He told me that he "gets letters every day from people who say, \'you know what, the Affordable Care Act saved my life or saved my kid\'s life because I got insurance.\' \'I thought I was healthy; turns out I had a tumor, but because I went and got a checkup, it was removed in time, and I\'m now cancer-free.\' " He added, "I think stories like that will be factored in when the Supreme Court takes a look at this case." CNN\'s Ben Tinker contributed to this report.', 'Moscow (CNN)A Russian TV channel aired Hillary Clinton\'s first campaign video with a rating stamp that means it\'s for mature audiences, because of fears it might run afoul of the country\'s anti-gay propaganda law. A clip of the video, which features a gay couple holding hands, got the 18+ rating from the independent TV Rain channel in Russia on Monday. The channel told CNN that it didn\'t want to break the controversial law, which bans "propaganda of nontraditional sexual relations around minors" and bars public discussion of gay rights and relationships within earshot of children. "There are no legal precedents for this law, so we just don\'t know what comes under this law and (what) doesn\'t," a TV Rain spokesperson told CNN. "Therefore, fearing to break the law -- especially given the high attention to TV Rain from the supervising authorities -- we decided to put a marker (on the video)." Clinton\'s video was released over the weekend to announce the start of her 2016 U.S. presidential campaign. It features about five seconds of two men holding hands. One of the men says, "I\'m getting married this summer to someone I really care about." The former senator and first lady first declared her support for same-sex marriage in early 2013, saying that "gay rights are human rights, and human rights are gay rights." Russia\'s controversial law caused an international outcry after it was passed by the Russian Parliament and signed by President Vladimir Putin in July 2013. Human Rights Watch described the anti-gay propaganda law as "a profoundly discriminatory and dangerous bill that is bound to worsen homophobia in Russia." Rights campaigners called for a boycott of the 2014 Winter Olympics in Sochi, and a number of bars around the world stopped serving Russian vodka in protest. U.S. President Barack Obama -- Clinton\'s former boss -- said at the time that he found the legislation offensive. "I have no patience for countries that try to treat gays or lesbians or transgendered persons in ways that intimidate them or harmful to them," Obama told Jay Leno in 2013. Putin defended the law, noting that unlike other countries, Russia decriminalized homosexual relationships (in 1993). "We don\'t outlaw anything and don\'t nab anyone," he said before the 2014 Games. "That\'s why you can feel safe and free here," he added, "but please leave our children in peace." The rights group ILGA-Europe said in a May 2014 report that Russia was the worst place in Europe (out of 49 countries) for LGBTI people to live. READ MORE: Social media react to Hillary Clinton logo .']
[01/17/2024-09:43:25] [TRT-LLM] [V] TensorRT-LLM Output: [['Ob给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给'], ['Hill给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给']]
[01/17/2024-09:43:25] [TRT-LLM] [V] Reference : ['"No challenge poses more of a public threat than climate change," the President says .\nHe credits the Clean Air Act with making Americans "a lot" healthier .', "Presidential hopeful's video, featuring gay couple, gets mature rating in Russia .\nRussian TV channel feared airing it would break the country's anti-gay propaganda law .\nClinton announced her support for same-sex marriage in 2013 ."]

here is the normal generate results with '--batch_size 1':

[01/17/2024-09:50:11] [TRT-LLM] [V] run data_point 2 ~ 3
[01/17/2024-09:50:11] [TRT-LLM] [V] The max_attention_window_size is not set, we will use max_seq_length by default.
[01/17/2024-09:50:11] [TRT-LLM] [V] The max_attention_window_size is not set, we will use max_seq_length by default.
[01/17/2024-09:50:28] [TRT-LLM] [V] ----------------------------------------------------------------------------------------------------
[01/17/2024-09:50:28] [TRT-LLM] [V] Input : ['(CNN)President Barack Obama took part in a roundtable discussion this week on climate change, refocusing on the issue from a public health vantage point. After the event at Washington\'s Howard University on Tuesday, Obama sat down with me for a one-on-one interview. I asked him about the science behind climate change and public health and the message he wants the average American to take away, as well as how enforceable his action plan is. Here are five things I learned: . The President enrolled at Occidental College in Los Angeles in 1979 (he transferred to Columbia University his junior year). While in L.A., he said, the air was so bad that it prevented him from running outside. He remembers the air quality alerts and how people with respiratory problems had to stay inside. He credits the Clean Air Act with making Americans "a lot" healthier, in addition to being able to "see the mountains in the background because they aren\'t covered in smog." Obama also said the instances of asthma and other respiratory diseases went down after these measures were taken. Peer-reviewed Environmental Protection Agency studies say that the Clean Air Act and subsequent amendments have reduced early deaths associated with exposure to ambient fine particle pollution and ozone, and reduced illnesses such as chronic bronchitis and acute myocardial infarction. The EPA estimates that, between 1970 and 2010, the act and its amendments prevented 365,000 early deaths from particulate matter alone. "No challenge poses more of a public threat than climate change," the President told me. When I asked about the strength of the science supporting the direct relationship between climate change and public health, he said, "We know as temperatures rise, insect-borne diseases potentially start shifting up. We know, in a very straight-forward fashion, that heatstroke and other heat-related illnesses and deaths potentially increase, and so what we\'re doing here is to make sure that in addition to public awareness around the potential for big storms like Hurricane Sandy or big wildfires or droughts, that people recognize there\'s a very personal, potential impact in climate change, and the good news is we can do something about it." In many ways, Obama is attempting to reframe the discussion around climate change as a public health issue that affects all of us, while conceding that we don\'t fully understand the magnitude of the correlation between rising temperatures and impact on human health. When asked what the average American can do about all this, the President encouraged ordinary citizens, doctors and nurses to start putting some pressure on elected officials "to try and make something happen to reduce the impacts of climate change." He also issued a presidential proclamation declaring April 6-12 as National Public Health Week "to better understand, communicate and reduce the health impacts of climate change on our communities." The average American can also do their part to reduce their own carbon footprint, including: . • Change your incandescent light bulbs to compact fluorescent lights. One CFL can reduce up to 1,300 pounds of carbon dioxide pollution during its lifetime. If every house in the U.S. switched its bulbs, we could reduce the electricity spent on lighting by half. • Unplug your gadgets and chargers when not in use. According to the U.S. Department of Energy, this practice can save $100 a year on your energy bill. • Use a laptop instead of a desktop. Laptops are designed to be energy-efficient, because battery life is a major factor in their design. According to Energy Star, a laptop can be up to 80% more energy-efficient than a desktop. • Filter your own water. Beyond the environmental toll of plastic waste, consider just how far your water was transported before you bought it at the grocery store. • Adjust your curtains and thermostats. If you keep your house 2 degrees warmer in the summer and 2 degrees colder in the winter, you can save big bucks on your energy bill. The Department of Energy estimates you can save up to 15% on your bill by turning off your thermostat when you\'re not at home. Obama did not appear particularly concerned about the current Supreme Court challenge to the Affordable Care Act. He  said he believes the statute is "clear and straightforward." He said, "I am not anticipating the Supreme Court would make such a bad decision." At issue is the 32 states that did not set up their own health care exchanges and left it to the federal government to do so. The plaintiffs in the lawsuit contend that the language of the Affordable Care Act does not allow for tax subsidies in those states (without state-based exchanges), possibly creating a situation, for example, in which people in Massachusetts would receive a tax credit, but people living in Texas would not. Obama did tell me that if the Supreme Court challenge is upheld, however, there is no Plan B. "Millions of people would lose their health insurance. They would no longer be able to afford the health insurance that\'s being provided out there." Obama went on to say, "I think this is the last gasp of folks who have been fighting against [the Affordable Care Act] for ideological reasons." He told me that he "gets letters every day from people who say, \'you know what, the Affordable Care Act saved my life or saved my kid\'s life because I got insurance.\' \'I thought I was healthy; turns out I had a tumor, but because I went and got a checkup, it was removed in time, and I\'m now cancer-free.\' " He added, "I think stories like that will be factored in when the Supreme Court takes a look at this case." CNN\'s Ben Tinker contributed to this report.']
[01/17/2024-09:50:28] [TRT-LLM] [V] TensorRT-LLM Output: [['Obama is a big fan of the Affordable Care Act.\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html?hpt=hp_t1\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html?hpt=hp_t1#\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html?hpt=hp_t1#.\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html?hpt=hp_t1#.VRX44_lViko\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html?hpt=hp_t1#.VRX44_lViko.\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html?hpt=hp_t1#.VRX44_lViko.\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html?hpt=hp_t1#.VRX44_lViko.\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html?hpt=hp_t1#.VRX44_lViko.\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html?hpt=hp_t1#.VRX44_lViko.\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html?hpt=hp_t1#.VRX44_lViko.\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html?hpt=hp_t1#.VRX44_lViko.\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html?hpt=hp_t1#.VRX44_lViko.\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html?hpt=hp_t1#.VRX44_lViko.\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html?hpt=hp_t1#.VRX44_lViko.\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html?hpt=hp_t1#.VRX44_lViko.\nhttp://www.cnn.com/2015/04/06/politics/']]
[01/17/2024-09:50:28] [TRT-LLM] [V] Reference : ['"No challenge poses more of a public threat than climate change," the President says .\nHe credits the Clean Air Act with making Americans "a lot" healthier .']
[01/17/2024-09:50:28] [TRT-LLM] [V] run data_point 3 ~ 4
[01/17/2024-09:50:28] [TRT-LLM] [V] The max_attention_window_size is not set, we will use max_seq_length by default.
[01/17/2024-09:50:28] [TRT-LLM] [V] The max_attention_window_size is not set, we will use max_seq_length by default.
[01/17/2024-09:50:44] [TRT-LLM] [V] ----------------------------------------------------------------------------------------------------
[01/17/2024-09:50:44] [TRT-LLM] [V] Input : ['Moscow (CNN)A Russian TV channel aired Hillary Clinton\'s first campaign video with a rating stamp that means it\'s for mature audiences, because of fears it might run afoul of the country\'s anti-gay propaganda law. A clip of the video, which features a gay couple holding hands, got the 18+ rating from the independent TV Rain channel in Russia on Monday. The channel told CNN that it didn\'t want to break the controversial law, which bans "propaganda of nontraditional sexual relations around minors" and bars public discussion of gay rights and relationships within earshot of children. "There are no legal precedents for this law, so we just don\'t know what comes under this law and (what) doesn\'t," a TV Rain spokesperson told CNN. "Therefore, fearing to break the law -- especially given the high attention to TV Rain from the supervising authorities -- we decided to put a marker (on the video)." Clinton\'s video was released over the weekend to announce the start of her 2016 U.S. presidential campaign. It features about five seconds of two men holding hands. One of the men says, "I\'m getting married this summer to someone I really care about." The former senator and first lady first declared her support for same-sex marriage in early 2013, saying that "gay rights are human rights, and human rights are gay rights." Russia\'s controversial law caused an international outcry after it was passed by the Russian Parliament and signed by President Vladimir Putin in July 2013. Human Rights Watch described the anti-gay propaganda law as "a profoundly discriminatory and dangerous bill that is bound to worsen homophobia in Russia." Rights campaigners called for a boycott of the 2014 Winter Olympics in Sochi, and a number of bars around the world stopped serving Russian vodka in protest. U.S. President Barack Obama -- Clinton\'s former boss -- said at the time that he found the legislation offensive. "I have no patience for countries that try to treat gays or lesbians or transgendered persons in ways that intimidate them or harmful to them," Obama told Jay Leno in 2013. Putin defended the law, noting that unlike other countries, Russia decriminalized homosexual relationships (in 1993). "We don\'t outlaw anything and don\'t nab anyone," he said before the 2014 Games. "That\'s why you can feel safe and free here," he added, "but please leave our children in peace." The rights group ILGA-Europe said in a May 2014 report that Russia was the worst place in Europe (out of 49 countries) for LGBTI people to live. READ MORE: Social media react to Hillary Clinton logo .']
[01/17/2024-09:50:44] [TRT-LLM] [V] TensorRT-LLM Output: [["Hillary Clinton's campaign logo is a big deal.\nHillary Clinton's campaign logo is a big deal.\nHillary Clinton's campaign logo is a big deal.\nHillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton"]]
[01/17/2024-09:50:44] [TRT-LLM] [V] Reference : ["Presidential hopeful's video, featuring gay couple, gets mature rating in Russia .\nRussian TV channel feared airing it would break the country's anti-gay propaganda law .\nClinton announced her support for same-sex marriage in 2013 ."]

NVIDIA / TensorRT-LLM

llama13b model generates abnormal results under multi-GPU multi-batch-size in tensorrt-llm 0.7.1 #869