Open cocoking99 opened 7 months ago
the dataset is:
elif args.eval_task == 'summarize':
dataset_name = "gsm8k"
dataset_revision = "main"
dataset_input_key = 'question'
dataset_output_key = 'answer'
dataset_split = 'test'
Could you reproduce your issue on public llama-13b model?
@byshiue I reproduced this problem on the above dataset using the public llama13b model(meta-llama/Llama-2-13b-chat-hf). if i run with '--batch_size 2', the results is abnormal, the output id may contain many 31999(the last token id) but i run with '--batch_size 1', the results is normal, the output id does not have 31999.
I am trying to reproduce your issue but fail. That's my scripts:
export LLAMA_MODEL=/home/scratch.trt_llm_data/llm-models/llama-models-v2/llama-v2-13b-hf/
python build.py --model_dir ${LLAMA_MODEL} --dtype float16 --use_gemm_plugin float16 --use_gpt_attention_plugin float16 --enable_context_fmha --remove_input_padding --output_dir /tmp/llama/llama-13b_tllm --world_size 2 --tp_size 2 --max_batch_size 4 --max_input_len 1024 --max_output_len 1024
mpirun -n 2 python ../summarize.py --test_trt_llm --hf_model_dir ${LLAMA_MODEL} --data_type fp16 --engine_dir /tmp/llama/llama-13b_tllm --batch_size 2 --log_level verbose
and the results of last two tokens and scores:
[01/15/2024-03:33:03] [TRT-LLM] [V] run data_point 38 ~ 40
[01/15/2024-03:33:03] [TRT-LLM] [V] The max_attention_window_size is not set, we will use max_seq_length by default.
[01/15/2024-03:33:03] [TRT-LLM] [V] The max_attention_window_size is not set, we will use max_seq_length by default.
[01/15/2024-03:33:04] [TRT-LLM] [V] ----------------------------------------------------------------------------------------------------
[01/15/2024-03:33:04] [TRT-LLM] [V] Input : ['Istanbul, Turkey (CNN)The questions Turks asked on Tuesday were tinged with fear. "What\'s going on? What happened? Why can\'t I get into the subway?" asked an elderly woman in a white headscarf with several shopping bags as she stood outside the barricaded entrance to one of Istanbul\'s busiest subway stations. She was one of millions of Turks left confused and concerned by the worst power outage to grip the country in more than a decade. Dozens of cities across Turkey lost power for hours on Tuesday. Millions of people were affected, including passengers stranded on paralyzed trains and subways. Municipal workers were forced to evacuate Istanbul\'s Marmaray Tunnel, where the black-out left commuters trapped deep beneath the rushing waters of the Bosphorus Strait. More than 24 hours later, Turkish officials were still at pains to explain the power outage. The energy minister suggested a possible failure in transmission lines. The prime minister did not rule out the possibility of a terror attack. The mysterious collapse of much of the country\'s energy grid triggered a burst of wild conspiracy theories across social media. Some Twitter users went so far as to suggest the black-outs were a warm up for elections scheduled to take place in June. There is fertile ground for rumor-mongering in Turkey. Over the last five years, security forces have arrested hundreds of army generals, journalists, prosecutors, civil society activists and police commanders and accused them of being members of assorted plots aimed at toppling the government. The government\'s increasingly heavy-handed repression of public dissent, combined with overt censorship of the media and the internet, have also contributed to a hyper-polarized and deeply mistrustful political atmosphere. Even Turkey\'s veteran deputy prime minister, Bulent Arinc, recently observed that opposition supporters now look at him "with hatred" rather than the grudging respect he enjoyed when his political party first swept to power in elections in 2002. Meanwhile, mysterious black-outs are a sore spot for some Turks, after a surreal 2014 incident on election night -- allegedly involving a feline saboteur. That is -- a cat that allegedly wandered into a power transformer. That was almost exactly a year ago, a smaller series of power outages affected some polling stations during nationwide municipal elections, prompting unsubstantiated accusations of vote rigging. The ruling Justice and Development Party ended up winning by a comfortable margin, but few Turks were reassured by the energy minister\'s explanation that the voting day black-outs were caused by a cat getting lost. As electricity was just starting to come back on in Istanbul on Tuesday, a second crisis erupted. Websites linked to an extremist leftist militant group known as the DHKP-C began publishing chilling photos of a masked man holding a pistol to the head of a hostage in front of communist flags. Two gunmen had somehow infiltrated the Palace of Justice, the monolithic court house in the center of Istanbul. There they took hostage Mehmet Selim Kiraz, the prosecutor in charge of one of the most politically sensitive trials in the country. The gunmen demanded the confessions of police officers accused of shooting a tear gas canister at Berkin Elvan, a 15-year-old boy who was critically wounded during anti-government protests that raged across Istanbul in 2013. The boy\'s death after months in a medically-induced coma triggered a fresh burst of protests and riots against the government. On Tuesday, in the midst of the hostage crisis at the court house, the Turkish government imposed a gag order banning broadcasters from reporting on the Palace of Justice siege. The broadcast ban is a measure that the Turkish government has repeatedly used in recent years to stifle reporting on deadly terrorist attacks. The government also famously shut down Twitter and YouTube in an effort to kill highly embarrassing political scandals involving corruption. Ultimately, Tuesday\'s court house siege ended in a deadly hail of bullets that left both gunmen dead and the prosecutor mortally wounded. Turkish officials say special forces raided the court house only after the militants began shooting. Online and in the streets, some Turks began linking the massive electricity blackouts to the hostage-taking inside one of Turkey\'s best-protected buildings, even though there is little to suggest the two incidents are connected. Just hours after the shooting, tensions exploded yet again at the court house. Istanbul\'s police chief had called for a press conference. As journalists jostled their way through security at the entrance to the largely deserted courthouse, some bystanders began chanting "government thieves." Just hours after a devastating lapse of security at the Palace of Justice, police began detaining the demonstrators hurling abuse at their elected government. The reactions to Tuesday\'s bewildering series of events revealed several truths about contemporary Turkey. The country is tense and confused after years of back-to-back political crises. Heavy-handed censorship has left the mainstream media widely distrusted and discredited by broad segments of society. And the absence of a common, credible space for sharing information has pushed critics of the government to the fringes of social media. Amid the burst of optimism and civil society activism in the early heady days of the Arab Spring in 2011, Turkey was often cited as a possible democratic model for countries in the Middle East. Many of those Arab countries have since descended into conflict, repression and instability. In the meantime, Turkey feels increasingly vulnerable to demons of its own making.', "Sanaa, Yemen (CNN)Al Qaeda fighters attacked a prison in the coastal Yemeni city of Al Mukallah early Thursday, freeing at least 270 prisoners, a third of whom have al Qaeda links, a senior Defense Ministry official has told CNN. Khaled Batarfi, a senior al Qaeda figure, was among the escapees, officials said. Dozens of attackers took control of government buildings, including the city's Central Prison, Central Bank and radio station during the assault early Thursday, according to officials. Government troops arrived early Thursday and clashed with the al Qaeda fighters, and most of the militants fled, the officials said. Last month, hundreds of inmates escaped from Al Mansoorah Central Prison in Aden after clashes between Shiite Houthi rebels and forces loyal to ousted Sunni President Abdu Rabu Mansour Hadi. Yemen has been descending into chaos in the weeks since Shiite Houthi rebels removed Hadi, a Sunni, from power. The sectarian nature of the conflict is drawing in regional rivals Saudi Arabia, which is predominately Sunni -- and is the country to which Hadi ultimately fled -- and Iran, which is predominately Shiite and supports the Houthi rebels. Because of that, the conflict in Yemen risks becoming a proxy war in the struggle between the Iranians and the Saudis for preeminence in the Middle East. The Saudis have conducted airstrikes against the Houthi rebels and could send in ground troops. But little is simple in the Middle East. And while the conflict between the Houthis and forces loyal to Hadi rages in the western part of the country, where it has caused hundreds of civilian deaths, al Qaeda in the Arabian Peninsula, or AQAP, controls parts of eastern Yemen. AQAP is considered one of the most ruthless branches of the terrorist organization."]
[01/15/2024-03:33:04] [TRT-LLM] [V] TensorRT-LLM Output: [["Turkey is a mess.\nThe Turkish government has banned Twitter and YouTube after a recording of a top security meeting was leaked online. The ban was imposed after a recording of a top security meeting was leaked online. The recording, which was allegedly made by the head of Turkey's intelligence agency, was published on YouTube. It purportedly shows the head of Turkey's intelligence agency, Hakan Fidan, telling the foreign minister, Ahmet Davut"], ["Al Qaeda fighters attacked a prison in the coastal Yemeni city of Al Mukallah early Thursday, freeing at least 270 prisoners, a third of whom have al Qaeda links, a senior Defense Ministry official has told CNN. Khaled Batarfi, a senior al Qaeda figure, was among the escapees, officials said. Dozens of attackers took control of government buildings, including the city's"]]
[01/15/2024-03:33:04] [TRT-LLM] [V] Reference : ['This week, Turkey was gripped by a massive power outage and a deadly hostage crisis .\nReactions reveal contemporary Turkey is tense and confused after years of political crises .\nCensorship has pushed critics to fringes in country cited as democratic model for Mideast .', 'Al Qaeda fighters attack a prison and other government buildings, freeing many prisoners .\nGovernment troops clash with the fighters, most of whom flee .\nYemen is descending into chaos as a Shia-Sunni conflict draws in regional rivals Saudi Arabia and Iran .']
[01/15/2024-03:33:04] [TRT-LLM] [I] TensorRT-LLM (total latency: 24.902254819869995 sec)
[01/15/2024-03:33:04] [TRT-LLM] [I] TensorRT-LLM beam 0 result
[01/15/2024-03:33:04] [TRT-LLM] [I] rouge1 : 18.639209970478824
[01/15/2024-03:33:04] [TRT-LLM] [I] rouge2 : 4.486236665518625
[01/15/2024-03:33:04] [TRT-LLM] [I] rougeL : 13.817401267734514
[01/15/2024-03:33:04] [TRT-LLM] [I] rougeLsum : 16.46296159716107
Info of git log
:
commit 80bc07510ac4ddf13c0d76ad295cdb2b75614618 (HEAD, tag: v0.7.1)
Author: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>
Date: Tue Dec 26 19:42:17 2023 +0800
Update TensorRT-LLM Release branch (#745)
* Update TensorRT-LLM
---------
Co-authored-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>
I am not sure what's the summarize_new.py
, but could you reproduce the accuracy issue by the scripts I share above? The scripts are almost same to yours, but use the files of TRT-LLM directly.
Sorry, I missed a reproduction condition (--output_len 1024) in my description, you can try to reproduce it again, there is no difference in the inference flow between summarize_new.py and summarize.py.
the run cmd line:
mpirun -n 2 python ../summarize.py --test_trt_llm --hf_model_dir ${LLAMA_MODEL} --data_type fp16 --engine_dir /tmp/llama/llama-13b_tllm --batch_size 2 --log_level verbose --output_len 1024
here is the abnormal generate results:
[01/15/2024-14:16:42] [TRT-LLM] [V] run data_point 2 ~ 4
[01/15/2024-14:16:42] [TRT-LLM] [V] The max_attention_window_size is not set, we will use max_seq_length by default.
[01/15/2024-14:16:42] [TRT-LLM] [V] The max_attention_window_size is not set, we will use max_seq_length by default.
[01/15/2024-14:17:00] [TRT-LLM] [V] ----------------------------------------------------------------------------------------------------
[01/15/2024-14:17:00] [TRT-LLM] [V] Input : ['(CNN)President Barack Obama took part in a roundtable discussion this week on climate change, refocusing on the issue from a public health vantage point. After the event at Washington\'s Howard University on Tuesday, Obama sat down with me for a one-on-one interview. I asked him about the science behind climate change and public health and the message he wants the average American to take away, as well as how enforceable his action plan is. Here are five things I learned: . The President enrolled at Occidental College in Los Angeles in 1979 (he transferred to Columbia University his junior year). While in L.A., he said, the air was so bad that it prevented him from running outside. He remembers the air quality alerts and how people with respiratory problems had to stay inside. He credits the Clean Air Act with making Americans "a lot" healthier, in addition to being able to "see the mountains in the background because they aren\'t covered in smog." Obama also said the instances of asthma and other respiratory diseases went down after these measures were taken. Peer-reviewed Environmental Protection Agency studies say that the Clean Air Act and subsequent amendments have reduced early deaths associated with exposure to ambient fine particle pollution and ozone, and reduced illnesses such as chronic bronchitis and acute myocardial infarction. The EPA estimates that, between 1970 and 2010, the act and its amendments prevented 365,000 early deaths from particulate matter alone. "No challenge poses more of a public threat than climate change," the President told me. When I asked about the strength of the science supporting the direct relationship between climate change and public health, he said, "We know as temperatures rise, insect-borne diseases potentially start shifting up. We know, in a very straight-forward fashion, that heatstroke and other heat-related illnesses and deaths potentially increase, and so what we\'re doing here is to make sure that in addition to public awareness around the potential for big storms like Hurricane Sandy or big wildfires or droughts, that people recognize there\'s a very personal, potential impact in climate change, and the good news is we can do something about it." In many ways, Obama is attempting to reframe the discussion around climate change as a public health issue that affects all of us, while conceding that we don\'t fully understand the magnitude of the correlation between rising temperatures and impact on human health. When asked what the average American can do about all this, the President encouraged ordinary citizens, doctors and nurses to start putting some pressure on elected officials "to try and make something happen to reduce the impacts of climate change." He also issued a presidential proclamation declaring April 6-12 as National Public Health Week "to better understand, communicate and reduce the health impacts of climate change on our communities." The average American can also do their part to reduce their own carbon footprint, including: . • Change your incandescent light bulbs to compact fluorescent lights. One CFL can reduce up to 1,300 pounds of carbon dioxide pollution during its lifetime. If every house in the U.S. switched its bulbs, we could reduce the electricity spent on lighting by half. • Unplug your gadgets and chargers when not in use. According to the U.S. Department of Energy, this practice can save $100 a year on your energy bill. • Use a laptop instead of a desktop. Laptops are designed to be energy-efficient, because battery life is a major factor in their design. According to Energy Star, a laptop can be up to 80% more energy-efficient than a desktop. • Filter your own water. Beyond the environmental toll of plastic waste, consider just how far your water was transported before you bought it at the grocery store. • Adjust your curtains and thermostats. If you keep your house 2 degrees warmer in the summer and 2 degrees colder in the winter, you can save big bucks on your energy bill. The Department of Energy estimates you can save up to 15% on your bill by turning off your thermostat when you\'re not at home. Obama did not appear particularly concerned about the current Supreme Court challenge to the Affordable Care Act. He said he believes the statute is "clear and straightforward." He said, "I am not anticipating the Supreme Court would make such a bad decision." At issue is the 32 states that did not set up their own health care exchanges and left it to the federal government to do so. The plaintiffs in the lawsuit contend that the language of the Affordable Care Act does not allow for tax subsidies in those states (without state-based exchanges), possibly creating a situation, for example, in which people in Massachusetts would receive a tax credit, but people living in Texas would not. Obama did tell me that if the Supreme Court challenge is upheld, however, there is no Plan B. "Millions of people would lose their health insurance. They would no longer be able to afford the health insurance that\'s being provided out there." Obama went on to say, "I think this is the last gasp of folks who have been fighting against [the Affordable Care Act] for ideological reasons." He told me that he "gets letters every day from people who say, \'you know what, the Affordable Care Act saved my life or saved my kid\'s life because I got insurance.\' \'I thought I was healthy; turns out I had a tumor, but because I went and got a checkup, it was removed in time, and I\'m now cancer-free.\' " He added, "I think stories like that will be factored in when the Supreme Court takes a look at this case." CNN\'s Ben Tinker contributed to this report.', 'Moscow (CNN)A Russian TV channel aired Hillary Clinton\'s first campaign video with a rating stamp that means it\'s for mature audiences, because of fears it might run afoul of the country\'s anti-gay propaganda law. A clip of the video, which features a gay couple holding hands, got the 18+ rating from the independent TV Rain channel in Russia on Monday. The channel told CNN that it didn\'t want to break the controversial law, which bans "propaganda of nontraditional sexual relations around minors" and bars public discussion of gay rights and relationships within earshot of children. "There are no legal precedents for this law, so we just don\'t know what comes under this law and (what) doesn\'t," a TV Rain spokesperson told CNN. "Therefore, fearing to break the law -- especially given the high attention to TV Rain from the supervising authorities -- we decided to put a marker (on the video)." Clinton\'s video was released over the weekend to announce the start of her 2016 U.S. presidential campaign. It features about five seconds of two men holding hands. One of the men says, "I\'m getting married this summer to someone I really care about." The former senator and first lady first declared her support for same-sex marriage in early 2013, saying that "gay rights are human rights, and human rights are gay rights." Russia\'s controversial law caused an international outcry after it was passed by the Russian Parliament and signed by President Vladimir Putin in July 2013. Human Rights Watch described the anti-gay propaganda law as "a profoundly discriminatory and dangerous bill that is bound to worsen homophobia in Russia." Rights campaigners called for a boycott of the 2014 Winter Olympics in Sochi, and a number of bars around the world stopped serving Russian vodka in protest. U.S. President Barack Obama -- Clinton\'s former boss -- said at the time that he found the legislation offensive. "I have no patience for countries that try to treat gays or lesbians or transgendered persons in ways that intimidate them or harmful to them," Obama told Jay Leno in 2013. Putin defended the law, noting that unlike other countries, Russia decriminalized homosexual relationships (in 1993). "We don\'t outlaw anything and don\'t nab anyone," he said before the 2014 Games. "That\'s why you can feel safe and free here," he added, "but please leave our children in peace." The rights group ILGA-Europe said in a May 2014 report that Russia was the worst place in Europe (out of 49 countries) for LGBTI people to live. READ MORE: Social media react to Hillary Clinton logo .']
[01/15/2024-14:17:00] [TRT-LLM] [V] TensorRT-LLM Output: [['Ob给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给'], ['Hill给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给']]
[01/15/2024-14:17:00] [TRT-LLM] [V] Reference : ['"No challenge poses more of a public threat than climate change," the President says .\nHe credits the Clean Air Act with making Americans "a lot" healthier .', "Presidential hopeful's video, featuring gay couple, gets mature rating in Russia .\nRussian TV channel feared airing it would break the country's anti-gay propaganda law .\nClinton announced her support for same-sex marriage in 2013 ."]
Since I don't have permission to access meta-llama/Llama-2-13b-chat-hf
now, could you try meta-llama/Llama-2-13b-hf? I want to align the reproduce environment first because I cannot reproduce such issue from my side.
I did not use the Llama-2-13b-hf model to reproduce the problem because the Llama-2-13b-hf model with output_len 1024 has a duplicate generation problem on this dataset. Now I have tried again and found that the same problem can be replicated for the Lama-2-13B-HF model with multiple batches.
the build cmd line:
python build.py --model_dir /Llama-2-13b-hf --dtype float16 --use_gemm_plugin float16 --use_gpt_attention_plugin float16 --enable_context_fmha --remove_input_padding --output_dir /Llama-2-13b-hf_tp2 --world_size 2 --tp_size 2 --max_batch_size 4 --max_input_len 1024 --max_output_len 1024
the run cmd line:
mpirun -n 2 --allow-run-as-root python ../summarize.py --test_trt_llm --hf_model_dir /Llama-2-13b-hf --data_type fp16 --engine_dir /Llama-2-13b-hf_tp2 --batch_size 2 --output_len 1024 --log_level verbose
here is the abnormal generate results with '--batch_size 2':
[01/17/2024-09:43:07] [TRT-LLM] [V] run data_point 2 ~ 4
[01/17/2024-09:43:07] [TRT-LLM] [V] The max_attention_window_size is not set, we will use max_seq_length by default.
[01/17/2024-09:43:25] [TRT-LLM] [V] ----------------------------------------------------------------------------------------------------
[01/17/2024-09:43:25] [TRT-LLM] [V] Input : ['(CNN)President Barack Obama took part in a roundtable discussion this week on climate change, refocusing on the issue from a public health vantage point. After the event at Washington\'s Howard University on Tuesday, Obama sat down with me for a one-on-one interview. I asked him about the science behind climate change and public health and the message he wants the average American to take away, as well as how enforceable his action plan is. Here are five things I learned: . The President enrolled at Occidental College in Los Angeles in 1979 (he transferred to Columbia University his junior year). While in L.A., he said, the air was so bad that it prevented him from running outside. He remembers the air quality alerts and how people with respiratory problems had to stay inside. He credits the Clean Air Act with making Americans "a lot" healthier, in addition to being able to "see the mountains in the background because they aren\'t covered in smog." Obama also said the instances of asthma and other respiratory diseases went down after these measures were taken. Peer-reviewed Environmental Protection Agency studies say that the Clean Air Act and subsequent amendments have reduced early deaths associated with exposure to ambient fine particle pollution and ozone, and reduced illnesses such as chronic bronchitis and acute myocardial infarction. The EPA estimates that, between 1970 and 2010, the act and its amendments prevented 365,000 early deaths from particulate matter alone. "No challenge poses more of a public threat than climate change," the President told me. When I asked about the strength of the science supporting the direct relationship between climate change and public health, he said, "We know as temperatures rise, insect-borne diseases potentially start shifting up. We know, in a very straight-forward fashion, that heatstroke and other heat-related illnesses and deaths potentially increase, and so what we\'re doing here is to make sure that in addition to public awareness around the potential for big storms like Hurricane Sandy or big wildfires or droughts, that people recognize there\'s a very personal, potential impact in climate change, and the good news is we can do something about it." In many ways, Obama is attempting to reframe the discussion around climate change as a public health issue that affects all of us, while conceding that we don\'t fully understand the magnitude of the correlation between rising temperatures and impact on human health. When asked what the average American can do about all this, the President encouraged ordinary citizens, doctors and nurses to start putting some pressure on elected officials "to try and make something happen to reduce the impacts of climate change." He also issued a presidential proclamation declaring April 6-12 as National Public Health Week "to better understand, communicate and reduce the health impacts of climate change on our communities." The average American can also do their part to reduce their own carbon footprint, including: . • Change your incandescent light bulbs to compact fluorescent lights. One CFL can reduce up to 1,300 pounds of carbon dioxide pollution during its lifetime. If every house in the U.S. switched its bulbs, we could reduce the electricity spent on lighting by half. • Unplug your gadgets and chargers when not in use. According to the U.S. Department of Energy, this practice can save $100 a year on your energy bill. • Use a laptop instead of a desktop. Laptops are designed to be energy-efficient, because battery life is a major factor in their design. According to Energy Star, a laptop can be up to 80% more energy-efficient than a desktop. • Filter your own water. Beyond the environmental toll of plastic waste, consider just how far your water was transported before you bought it at the grocery store. • Adjust your curtains and thermostats. If you keep your house 2 degrees warmer in the summer and 2 degrees colder in the winter, you can save big bucks on your energy bill. The Department of Energy estimates you can save up to 15% on your bill by turning off your thermostat when you\'re not at home. Obama did not appear particularly concerned about the current Supreme Court challenge to the Affordable Care Act. He said he believes the statute is "clear and straightforward." He said, "I am not anticipating the Supreme Court would make such a bad decision." At issue is the 32 states that did not set up their own health care exchanges and left it to the federal government to do so. The plaintiffs in the lawsuit contend that the language of the Affordable Care Act does not allow for tax subsidies in those states (without state-based exchanges), possibly creating a situation, for example, in which people in Massachusetts would receive a tax credit, but people living in Texas would not. Obama did tell me that if the Supreme Court challenge is upheld, however, there is no Plan B. "Millions of people would lose their health insurance. They would no longer be able to afford the health insurance that\'s being provided out there." Obama went on to say, "I think this is the last gasp of folks who have been fighting against [the Affordable Care Act] for ideological reasons." He told me that he "gets letters every day from people who say, \'you know what, the Affordable Care Act saved my life or saved my kid\'s life because I got insurance.\' \'I thought I was healthy; turns out I had a tumor, but because I went and got a checkup, it was removed in time, and I\'m now cancer-free.\' " He added, "I think stories like that will be factored in when the Supreme Court takes a look at this case." CNN\'s Ben Tinker contributed to this report.', 'Moscow (CNN)A Russian TV channel aired Hillary Clinton\'s first campaign video with a rating stamp that means it\'s for mature audiences, because of fears it might run afoul of the country\'s anti-gay propaganda law. A clip of the video, which features a gay couple holding hands, got the 18+ rating from the independent TV Rain channel in Russia on Monday. The channel told CNN that it didn\'t want to break the controversial law, which bans "propaganda of nontraditional sexual relations around minors" and bars public discussion of gay rights and relationships within earshot of children. "There are no legal precedents for this law, so we just don\'t know what comes under this law and (what) doesn\'t," a TV Rain spokesperson told CNN. "Therefore, fearing to break the law -- especially given the high attention to TV Rain from the supervising authorities -- we decided to put a marker (on the video)." Clinton\'s video was released over the weekend to announce the start of her 2016 U.S. presidential campaign. It features about five seconds of two men holding hands. One of the men says, "I\'m getting married this summer to someone I really care about." The former senator and first lady first declared her support for same-sex marriage in early 2013, saying that "gay rights are human rights, and human rights are gay rights." Russia\'s controversial law caused an international outcry after it was passed by the Russian Parliament and signed by President Vladimir Putin in July 2013. Human Rights Watch described the anti-gay propaganda law as "a profoundly discriminatory and dangerous bill that is bound to worsen homophobia in Russia." Rights campaigners called for a boycott of the 2014 Winter Olympics in Sochi, and a number of bars around the world stopped serving Russian vodka in protest. U.S. President Barack Obama -- Clinton\'s former boss -- said at the time that he found the legislation offensive. "I have no patience for countries that try to treat gays or lesbians or transgendered persons in ways that intimidate them or harmful to them," Obama told Jay Leno in 2013. Putin defended the law, noting that unlike other countries, Russia decriminalized homosexual relationships (in 1993). "We don\'t outlaw anything and don\'t nab anyone," he said before the 2014 Games. "That\'s why you can feel safe and free here," he added, "but please leave our children in peace." The rights group ILGA-Europe said in a May 2014 report that Russia was the worst place in Europe (out of 49 countries) for LGBTI people to live. READ MORE: Social media react to Hillary Clinton logo .']
[01/17/2024-09:43:25] [TRT-LLM] [V] TensorRT-LLM Output: [['Ob给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给'], ['Hill给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给给']]
[01/17/2024-09:43:25] [TRT-LLM] [V] Reference : ['"No challenge poses more of a public threat than climate change," the President says .\nHe credits the Clean Air Act with making Americans "a lot" healthier .', "Presidential hopeful's video, featuring gay couple, gets mature rating in Russia .\nRussian TV channel feared airing it would break the country's anti-gay propaganda law .\nClinton announced her support for same-sex marriage in 2013 ."]
here is the normal generate results with '--batch_size 1':
[01/17/2024-09:50:11] [TRT-LLM] [V] run data_point 2 ~ 3
[01/17/2024-09:50:11] [TRT-LLM] [V] The max_attention_window_size is not set, we will use max_seq_length by default.
[01/17/2024-09:50:11] [TRT-LLM] [V] The max_attention_window_size is not set, we will use max_seq_length by default.
[01/17/2024-09:50:28] [TRT-LLM] [V] ----------------------------------------------------------------------------------------------------
[01/17/2024-09:50:28] [TRT-LLM] [V] Input : ['(CNN)President Barack Obama took part in a roundtable discussion this week on climate change, refocusing on the issue from a public health vantage point. After the event at Washington\'s Howard University on Tuesday, Obama sat down with me for a one-on-one interview. I asked him about the science behind climate change and public health and the message he wants the average American to take away, as well as how enforceable his action plan is. Here are five things I learned: . The President enrolled at Occidental College in Los Angeles in 1979 (he transferred to Columbia University his junior year). While in L.A., he said, the air was so bad that it prevented him from running outside. He remembers the air quality alerts and how people with respiratory problems had to stay inside. He credits the Clean Air Act with making Americans "a lot" healthier, in addition to being able to "see the mountains in the background because they aren\'t covered in smog." Obama also said the instances of asthma and other respiratory diseases went down after these measures were taken. Peer-reviewed Environmental Protection Agency studies say that the Clean Air Act and subsequent amendments have reduced early deaths associated with exposure to ambient fine particle pollution and ozone, and reduced illnesses such as chronic bronchitis and acute myocardial infarction. The EPA estimates that, between 1970 and 2010, the act and its amendments prevented 365,000 early deaths from particulate matter alone. "No challenge poses more of a public threat than climate change," the President told me. When I asked about the strength of the science supporting the direct relationship between climate change and public health, he said, "We know as temperatures rise, insect-borne diseases potentially start shifting up. We know, in a very straight-forward fashion, that heatstroke and other heat-related illnesses and deaths potentially increase, and so what we\'re doing here is to make sure that in addition to public awareness around the potential for big storms like Hurricane Sandy or big wildfires or droughts, that people recognize there\'s a very personal, potential impact in climate change, and the good news is we can do something about it." In many ways, Obama is attempting to reframe the discussion around climate change as a public health issue that affects all of us, while conceding that we don\'t fully understand the magnitude of the correlation between rising temperatures and impact on human health. When asked what the average American can do about all this, the President encouraged ordinary citizens, doctors and nurses to start putting some pressure on elected officials "to try and make something happen to reduce the impacts of climate change." He also issued a presidential proclamation declaring April 6-12 as National Public Health Week "to better understand, communicate and reduce the health impacts of climate change on our communities." The average American can also do their part to reduce their own carbon footprint, including: . • Change your incandescent light bulbs to compact fluorescent lights. One CFL can reduce up to 1,300 pounds of carbon dioxide pollution during its lifetime. If every house in the U.S. switched its bulbs, we could reduce the electricity spent on lighting by half. • Unplug your gadgets and chargers when not in use. According to the U.S. Department of Energy, this practice can save $100 a year on your energy bill. • Use a laptop instead of a desktop. Laptops are designed to be energy-efficient, because battery life is a major factor in their design. According to Energy Star, a laptop can be up to 80% more energy-efficient than a desktop. • Filter your own water. Beyond the environmental toll of plastic waste, consider just how far your water was transported before you bought it at the grocery store. • Adjust your curtains and thermostats. If you keep your house 2 degrees warmer in the summer and 2 degrees colder in the winter, you can save big bucks on your energy bill. The Department of Energy estimates you can save up to 15% on your bill by turning off your thermostat when you\'re not at home. Obama did not appear particularly concerned about the current Supreme Court challenge to the Affordable Care Act. He said he believes the statute is "clear and straightforward." He said, "I am not anticipating the Supreme Court would make such a bad decision." At issue is the 32 states that did not set up their own health care exchanges and left it to the federal government to do so. The plaintiffs in the lawsuit contend that the language of the Affordable Care Act does not allow for tax subsidies in those states (without state-based exchanges), possibly creating a situation, for example, in which people in Massachusetts would receive a tax credit, but people living in Texas would not. Obama did tell me that if the Supreme Court challenge is upheld, however, there is no Plan B. "Millions of people would lose their health insurance. They would no longer be able to afford the health insurance that\'s being provided out there." Obama went on to say, "I think this is the last gasp of folks who have been fighting against [the Affordable Care Act] for ideological reasons." He told me that he "gets letters every day from people who say, \'you know what, the Affordable Care Act saved my life or saved my kid\'s life because I got insurance.\' \'I thought I was healthy; turns out I had a tumor, but because I went and got a checkup, it was removed in time, and I\'m now cancer-free.\' " He added, "I think stories like that will be factored in when the Supreme Court takes a look at this case." CNN\'s Ben Tinker contributed to this report.']
[01/17/2024-09:50:28] [TRT-LLM] [V] TensorRT-LLM Output: [['Obama is a big fan of the Affordable Care Act.\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html?hpt=hp_t1\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html?hpt=hp_t1#\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html?hpt=hp_t1#.\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html?hpt=hp_t1#.VRX44_lViko\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html?hpt=hp_t1#.VRX44_lViko.\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html?hpt=hp_t1#.VRX44_lViko.\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html?hpt=hp_t1#.VRX44_lViko.\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html?hpt=hp_t1#.VRX44_lViko.\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html?hpt=hp_t1#.VRX44_lViko.\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html?hpt=hp_t1#.VRX44_lViko.\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html?hpt=hp_t1#.VRX44_lViko.\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html?hpt=hp_t1#.VRX44_lViko.\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html?hpt=hp_t1#.VRX44_lViko.\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html?hpt=hp_t1#.VRX44_lViko.\nhttp://www.cnn.com/2015/04/06/politics/obama-climate-change-health-care-supreme-court/index.html?hpt=hp_t1#.VRX44_lViko.\nhttp://www.cnn.com/2015/04/06/politics/']]
[01/17/2024-09:50:28] [TRT-LLM] [V] Reference : ['"No challenge poses more of a public threat than climate change," the President says .\nHe credits the Clean Air Act with making Americans "a lot" healthier .']
[01/17/2024-09:50:28] [TRT-LLM] [V] run data_point 3 ~ 4
[01/17/2024-09:50:28] [TRT-LLM] [V] The max_attention_window_size is not set, we will use max_seq_length by default.
[01/17/2024-09:50:28] [TRT-LLM] [V] The max_attention_window_size is not set, we will use max_seq_length by default.
[01/17/2024-09:50:44] [TRT-LLM] [V] ----------------------------------------------------------------------------------------------------
[01/17/2024-09:50:44] [TRT-LLM] [V] Input : ['Moscow (CNN)A Russian TV channel aired Hillary Clinton\'s first campaign video with a rating stamp that means it\'s for mature audiences, because of fears it might run afoul of the country\'s anti-gay propaganda law. A clip of the video, which features a gay couple holding hands, got the 18+ rating from the independent TV Rain channel in Russia on Monday. The channel told CNN that it didn\'t want to break the controversial law, which bans "propaganda of nontraditional sexual relations around minors" and bars public discussion of gay rights and relationships within earshot of children. "There are no legal precedents for this law, so we just don\'t know what comes under this law and (what) doesn\'t," a TV Rain spokesperson told CNN. "Therefore, fearing to break the law -- especially given the high attention to TV Rain from the supervising authorities -- we decided to put a marker (on the video)." Clinton\'s video was released over the weekend to announce the start of her 2016 U.S. presidential campaign. It features about five seconds of two men holding hands. One of the men says, "I\'m getting married this summer to someone I really care about." The former senator and first lady first declared her support for same-sex marriage in early 2013, saying that "gay rights are human rights, and human rights are gay rights." Russia\'s controversial law caused an international outcry after it was passed by the Russian Parliament and signed by President Vladimir Putin in July 2013. Human Rights Watch described the anti-gay propaganda law as "a profoundly discriminatory and dangerous bill that is bound to worsen homophobia in Russia." Rights campaigners called for a boycott of the 2014 Winter Olympics in Sochi, and a number of bars around the world stopped serving Russian vodka in protest. U.S. President Barack Obama -- Clinton\'s former boss -- said at the time that he found the legislation offensive. "I have no patience for countries that try to treat gays or lesbians or transgendered persons in ways that intimidate them or harmful to them," Obama told Jay Leno in 2013. Putin defended the law, noting that unlike other countries, Russia decriminalized homosexual relationships (in 1993). "We don\'t outlaw anything and don\'t nab anyone," he said before the 2014 Games. "That\'s why you can feel safe and free here," he added, "but please leave our children in peace." The rights group ILGA-Europe said in a May 2014 report that Russia was the worst place in Europe (out of 49 countries) for LGBTI people to live. READ MORE: Social media react to Hillary Clinton logo .']
[01/17/2024-09:50:44] [TRT-LLM] [V] TensorRT-LLM Output: [["Hillary Clinton's campaign logo is a big deal.\nHillary Clinton's campaign logo is a big deal.\nHillary Clinton's campaign logo is a big deal.\nHillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton's campaign logo is a big deal. Hillary Clinton"]]
[01/17/2024-09:50:44] [TRT-LLM] [V] Reference : ["Presidential hopeful's video, featuring gay couple, gets mature rating in Russia .\nRussian TV channel feared airing it would break the country's anti-gay propaganda law .\nClinton announced her support for same-sex marriage in 2013 ."]
Is it reproducable on single GPU?
Single GPU does not have this problem. As I first described, I use 4090 GPU and llama13b_fp16 can only do inference with multiple cards. When I used llama13b_int8 for single-card inference, this problem did not reproduce.
Unfortunately, I try on main branch and v0.7.1 again and I am still not able to reproduce your issue on 2-way tensor parallelism. You might need to investigate on your side and we will try our best to provide helps. You could try printing some intermediate results (here is document) to find where does the issue happen.
You could compare the results of TP2-FP16 and TP1-INT8 like results of layernorm, qkv GEMM, attention to find where does the big gap happen.
I used the self-trained llama13b model in tensorrt-llm 0.7.1 environment, using 2 4090 gpu, running multiple batches of the dataset, generating abnormal results. However, when running a single batch in the same environment, the results were fine.
here is the build cmd line:
python build.py --model_dir /home/TensorRT-LLM_071/examples/llama/llama-13b --dtype float16 --use_gemm_plugin float16 --use_gpt_attention_plugin float16 --enable_context_fmha --remove_input_padding --output_dir /home/TensorRT-LLM_071/examples/llama/llama-13b_tllm --world_size 2 --tp_size 2 --max_batch_size 4 --max_input_len 1024 --max_output_len 1024
and the run cmd line:
python ../summarize_new.py --test_trt_llm --hf_model_dir /home/TensorRT-LLM_071/examples/llama/llama-13b --data_type fp16 --engine_dir /home/TensorRT-LLM_071/examples/llama/llama-13b_tllm --batch_size 2
here is the abnormal generate results:
but if i run with '--batch_size 1', the results is normal:
In addition, if the model is quantized with int8 using the parameters --use_weight_only --weight_only_precision int8 --world_size 1 --tp_size 1 and runs on a single card, even multi-batch-size will not cause the above problems.
Finally, I tried to run the model and test script to version 0.6.1 without any problems. How to solve it?