Open dukesun99 opened 3 years ago
Thank you for letting me know and for providing an input. Let me poke around and see. Bootleg was trained on individual sentences --- geared more towards short text NED like for QA. So it might be worth trying to take the document and split in to sentences (nltk has a sentence splitter you can try here). I'll test and see. This is great feedback!
@lorr1 Thank you for your answer! Yes, I currently use nltk to split the doc into sentences as a workaround. Not sure if there exists such a dataset, but maybe if the model would be able to take in long documents, the extra context might also benefit the performance? Just some random idea haha
When I apply the following code on long articles
The results significantly get worse when the article is longer. Is it expected? You may use the below text to test:
And yet, 100 years from now, Obama’s presidency will be hailed as the most transformative of our lifetimes, and Donald Trump’s will be viewed with the same scorn that followed the Dred Scott decision. Like that pre-Civil War Supreme Court case, Trump will forever be condemned as a racial reprobate whose words and actions inspired white supremacists and neo-Nazis.\nLast week’s slaughter of Muslims in New Zealand was allegedly committed by a fascist who claimed to draw inspiration from President Trump, among others. It was the latest in a long line of tragedies that our president failed to clearly condemn. After the 2017 riots in Charlottesville, Trump proclaimed a moral equivalence between neo-Nazis and their opponents. Following the killings in Christchurch, the president dismissed the threat of white supremacy while claiming the rising tide of violence coming from the far right was limited to a few troublemakers with “very serious problems.”\nTrump’s acting chief of staff appeared on national television to declare that “the president is not a white supremacist.” White House counselor Kellyanne Conway once again shamed herself by dismissing the fascist mass murderer from Australia as an “eco-terrorist.” The president’s apologists denied that the current commander in chief was inspiring right-wing violence. But the Center for Strategic and International Studies reported in November that far-right attacks rose in Europe by 43 percent since 2016, while right-wing terrorist attacks have quadrupled in the United States over the same time. Hate crimes rose 17 percent in 2017.\nThis troubling chapter in U.S. history has one author — and his name is Donald Trump. This sputtering reality star has created a political identity and corrupt presidency inspired by the wave of racism that followed Obama’s. The Manhattan multimillionaire’s 2016 calls for a Muslim ban and creation of a Muslim registry; the claim of ignorance toward former Ku Klux Klan leader David Duke; the attack on a Hispanic judge’s integrity; the callousness shown toward a Muslim Gold Star family; and the anti-Semitic tweet featuring a Star of David and piles of $100 bills next to Hillary Clinton’s face. These are just a few of the racially charged offenses that Trump committed even before Americans elected him president.\nThe shocking conclusion to the 2016 campaign made millions of Americans, including me, look foolish for believing that Obama’s victories in 2008 and 2012 had proved that the United States had emerged from the scourge of racism infecting it for more than four centuries. I remain shocked that this strain of bigotry still fuels the political careers of Trump and his enablers on Capitol Hill.\nThe Rev. Martin Luther King Jr. often said that “the arc of the moral universe is long, but it bends toward justice.” That is but one reason why the rise in bigotry shown to Muslims, Jews, Hispanics, blacks and “others” has been so discouraging in the age of Trump. Like those who believed these racists were relics from a bygone age, I had also convinced myself that my Republican colleagues were so repelled by racism that they would never support a leader who provided inspiration to neo-Nazis and white supremacists; the manifesto of the New Zealand killer and the words of David Duke after Charlottesville showed just how wrong I was.\nThat’s why any policy differences I had with Obama now seem so insignificant. Americans who still have faith in the upward arc of King’s moral universe should be grateful for Obama’s presidency and the way his election exposed the white racism that is still at large in our land. If changing the Constitution and reelecting Obama two more times would break the fever that now ravages Trump’s Washington, I would cheerfully champion the passage of that constitutional amendment, slap a “Hope and Change” sticker on my shirt, and race to the nearest voting booth to support the man historians will remember as the most significant president since Abraham Lincoln.\nRead more: