llama.cpp + Final Jeopardy

[ggerganov]

I was browsing reddit and saw this post:

https://www.reddit.com/r/LocalLLaMA/comments/12xkm9v/alpaca_vs_final_jeopardy/

If anyone is interested, it would be great to add such evaluation as an example to llama.cpp and add instructions for running it with different models: LLaMA, Alpaca, Vicuna, etc. and different quantizations.

Here is the original work by @aigoopy which can be a good starting point:

https://github.com/aigoopy/llm-jeopardy

[CRD716]

Going to attempt this, should it be in a separate folder (ex. examples/jeopardy) or should it be thrown directly in examples?

[ggerganov]

In examples/jeopardy, with a README with instructions, question files, eval scripts, plot scripts, etc. would be nice

[SlyEcho]

I extracted the data from the spreadsheet. I don't know if it is necessary to reverse the Jeopardy "answer in the form of a question". I think LLMs should be able to emulate it.

JSON

```json [ { "category": "The Oscars", "answer": "Born in 1932 & the son of a percussionist in the CBS radio orchestra, he's been nominated for 53 Oscars", "question": "Who is John Williams?" }, { "category": "English Literature", "answer": "It says, \"The mind is its own place, & in itself can make a heaven of hell, a hell of heaven. What matter where, if I be still the same\"", "question": "What is Paradise Lost?" }, { "category": "Writers' Lesser-Known Works", "answer": "Known for more philosophical works, he wrote the play \"La Mandragola\", in which Florentines are rewarded for immoral actions", "question": "Who is Niccolò Machiavelli?" }, { "category": "Exploration", "answer": "James Cook's account of a 1774 visit here records an object \"near 27 feet long, and upwards of 8 feet over the breast or shoulders\"", "question": "What is Easter Island (Rapa Nui)?" }, { "category": "The Bill of Rights", "answer": "England's \"Bloody Assizes\" & a 1685 life sentence for perjury were 2 main origins of this amendment to the U.S. Constitution", "question": "What is the Eighth Amendment?" }, { "category": "Nobel Peace Prize Winners", "answer": "At times they each lived on Vilakazi St. in Soweto, so it claims to be the world's only street home to 2 Nobel Peace Prize winners", "question": "Who are Nelson Mandela & Desmond Tutu?" }, { "category": "Famous Names", "answer": "In 1966, the year of his death, he shared plans for an experimental prototype community in Florida", "question": "Who is Walt Disney?" }, { "category": "Geography", "answer": "Of the 13 nations through which the Equator passes, it's the only one whose coastline borders the Caribbean Sea", "question": "What is Colombia?" }, { "category": "Fashion History", "answer": "These decorative items get their name from their origin in the port city of Strasbourg, on the border of France & Germany", "question": "What are rhinestones?" }, { "category": "Movies of the '80s", "answer": "Based on an off-Broadway play with just 3 characters, it won the Best Picture Oscar & the actors in all 3 roles were nominated", "question": "What is Driving Miss Daisy?" }, { "category": "Novelists", "answer": "A 2012 book review noted subjects that \"sparked his ire\": capital punishment, big tobacco & \"the plight of the unjustly convicted\"", "question": "Who is John Grisham?" }, { "category": "20th Century Eponyms", "answer": "A 1940 headline about this included \"failure\", \"liability when it came to offense\" & \"stout hearts no match for tanks\"", "question": "What is the Maginot Line?" }, { "category": "City History", "answer": "Over 700 years after its traditional 1252 founding date, this port city became associated with a psychological response", "question": "What is Stockholm?" }, { "category": "Brand Names", "answer": "The success of this brand has its roots with a hydrotherapy pump its cofounder created for his son, who had arthritis", "question": "What is Jacuzzi?" }, { "category": "American Authors", "answer": "In a periodical in 1807, he called New York City \"Gotham, Gotham! Most enlightened of cities\"", "question": "Who is Washington Irving?" }, { "category": "Symbols", "answer": "In math, it's a rotated V; in society, it's a feeling of some marginalized or underrepresented people", "question": "What is \"less than\"?" }, { "category": "Movie Theme Songs", "answer": "Monty Norman, the composer of this character's theme, said the staccato riff conveyed sexiness, mystery & ruthlessness", "question": "Who is James Bond?" }, { "category": "American Novelists", "answer": "He served with an airman named Yohannan in World War II & despite what readers might think, he said he enjoyed his service", "question": "Who is Joseph Heller?" }, { "category": "Medieval Places", "answer": "One of the participants in an 1170 event at this place said, \"Let us away, knights; he will rise no more\"", "question": "What is Canterbury, England? (Canterbury Cathedral)" }, { "category": "Countries of Africa", "answer": "At one time a province of the Roman Empire, this kingdom is known to Arabic scholars as Al-Maghrib Al-Aqsa, \"the far west\"", "question": "What is Morocco?" }, { "category": "Statehood", "answer": "Congress relented in 1890 after this prospective state said it would wait 100 years rather than come in without the women", "question": "What is Wyoming?" }, { "category": "1980s Movies", "answer": "A writer & producer of this movie said he wanted it to be like a Western or James Bond film, \"only it takes place in the '30s\"", "question": "What is Raiders of the Lost Ark?" }, { "category": "Art Exhibitions", "answer": "In 1898 what's been called the first blockbuster art show was devoted to him & put on for Queen Wilhelmina's coronation", "question": "Who is Rembrandt?" }, { "category": "Countries of the World", "answer": "Part of the largest contiguous land empire during the 1200s & 1300s, today it's the world's second-largest landlocked country", "question": "What is Mongolia?" }, { "category": "Literature", "answer": "A 2006 book was titled \"The Poem That Changed America:\" this \"Fifty Years Later\"", "question": "What is \"Howl\"?" }, { "category": "Invasions", "answer": "Backed by 14,000 troops, he invaded England to restore, in his words, its \"religion, laws, and liberties\"", "question": "Who is William of Orange?" }, { "category": "Landmarks", "answer": "After its completion in the late 19th c., it was called \"a truly tragic street lamp\" & a \"high & skinny pyramid of iron ladders\"", "question": "What is the Eiffel Tower?" }, { "category": "Geographic Name's the Same", "answer": "The busiest passenger port in the U.K., it shares its name with a capital of one of the original 13 states", "question": "What is Dover?" }, { "category": "Names in the Bookstore", "answer": "This man made lists, perhaps to cope with depression; a set of lists he published in 1852 made his name synonymous with a type of book", "question": "Who is Peter Mark Roget?" }, { "category": "U.S. History", "answer": "An 1869 presidential pardon was granted to this man, due in part to a plea by the Medical Society of Harford County, Maryland", "question": "Who is Dr. Samuel Mudd?" }, { "category": "American Literature", "answer": "Letters, pocket knives, C rations & steel helmets are among the tangible items referred to in the title of this modern war classic", "question": "What is The Things They Carried?" }, { "category": "Nonfiction", "answer": "It has the line, \"The discovery of America…opened up fresh ground for the rising bourgeoisie\"", "question": "What is The Communist Manifesto" }, { "category": "Laws in U.S. History", "answer": "A radical Republican championed this 1875 act but the Supreme Court struck it down in 1883; a new version was passed 81 years later", "question": "What is the Civil Rights Act?" }, { "category": "Names of Myth", "answer": "Her brothers, Castor & Pollux, saved her after Theseus stole her away as a kid; a larger force would seek her later in life", "question": "Who is Helen of Troy?" }, { "category": "African Countries", "answer": "Once Africa's largest country in area, it dropped to third in 2011 when a portion of it declared independence", "question": "What is Sudan?" }, { "category": "The Ancient World", "answer": "The ancient writer Galen said books on ships arriving to this city's port were seized, originals kept & copies returned", "question": "What is Alexandria?" }, { "category": "Famous Names", "answer": "For a special 1970s cookbook, he provided one simple recipe -- a can of Campbell's tomato soup & 2 cans of milk", "question": "Who is Andy Warhol?" }, { "category": "People & Places", "answer": "Thought to descend from people of Southeast Asia, the Chamorro make up this U.S. territory's largest ethnic group", "question": "What is Guam?" }, { "category": "Current World Leaders", "answer": "In office from 2022, the president of this country has taken so many foreign trips a play on his name is \"Ferdinand Magellan Jr.\"", "question": "What is the Philippines?" }, { "category": "Writers & The South", "answer": "In 1939 he lived on Toulouse Street in the French Quarter & chose the professional name that bonded him to the South", "question": "Who is Tennessee Williams?" }, { "category": "National Parks", "answer": "It's named for a river indigenous people called Mi tse a-da-zi, translated by French-speaking trappers as \"Pierre Jaune\"", "question": "What is Yellowstone?" }, { "category": "Sports", "answer": "In 2010 they introduced the 4-point shot, 35 feet from the basket", "question": "Who are the Harlem Globetrotters?" }, { "category": "The U.S. Military", "answer": "Losses over Asia in the 1960s led to the establishment of the program known as this at a San Diego naval base in 1969", "question": "What is \"Top Gun\"?" }, { "category": "Art & Science", "answer": "A craft that visited it was named for Giotto, based on the story that 680 years earlier, the painter depicted it as the Star of Bethlehem", "question": "What is Halley's Comet?" }, { "category": "Words From World War I", "answer": "\"Cistern\" & \"reservoir\" were suggested names for a secret invention, but the British preferred this less clumsy monosyllable", "question": "What is \"tank\"?" }, { "category": "European History", "answer": "Until 1806, some German nobles included among their honors the title of \"Elector\" for their role in selecting this personage", "question": "What is Holy Roman Emperor?" }, { "category": "Theater History", "answer": "In 1904, wearing a harness, actress Nina Boucicault became the first to play this character onstage", "question": "Who is Peter Pan?" }, { "category": "European Cities", "answer": "Alphabetically the first German city in encyclopedias, it was also the first one taken by the Allies in World War II", "question": "What is Aachen?" }, { "category": "Word Origins", "answer": "This Sanskrit word referring to a spoken word or phrase comes from a word for \"to think\"", "question": "What is mantra?" }, { "category": "Inventions", "answer": "1917's \"Elements of Trench Warfare\" said this Old West item was \"difficult to destroy\" & \"difficult to get through\"", "question": "What is barbed wire?" }, { "category": "World War II", "answer": "Mimi Reinhard, who never learned to type using more than 2 fingers, produced this with 1,100 names, including hers", "question": "What is Schindler's list?" }, { "category": "Mythology", "answer": "Poseidon carried off the maiden Theophane & turned her into a ewe; their offspring was the source of this mythical object", "question": "What is the Golden Fleece?" }, { "category": "Literature", "answer": "Published in 2011, P.D. James' final novel, \"Death Comes to Pemberley\", was a sequel to this novel from 200 years earlier", "question": "What is Pride and Prejudice?" }, { "category": "U.S. State Names", "answer": "5 U.S. states have 6-letter names; only these 2 west of the Mississippi River border each other", "question": "What are Oregon & Nevada?" }, { "category": "Word Origins", "answer": "Originally relating to a story of suffering, this word now more commonly refers to strong emotion of any kind", "question": "What is passion?" }, { "category": "World Cinema", "answer": "The 2007 biopic called \"La Môme\" in France, meaning \"The Kid\", was released in the U.S. under this other French title", "question": "What is La Vie en Rose?" }, { "category": "History", "answer": "Returning home in 1493, Columbus stopped in the Azores at an island with this name, also something he'd lost off the Haiti coast", "question": "What is Santa Maria?" }, { "category": "Landmarks", "answer": "Pskov & Nizhny Novgorod are 2 of the cities that have a fortress called this", "question": "What is a kremlin?" }, { "category": "Foreign-Born Authors", "answer": "In the 1950s the New York Times said this author \"is writing about all lust\" & his lecherous narrator \"is all of us\"", "question": "Who is Vladimir Nabokov?" }, { "category": "Astronomy & Geography", "answer": "At the winter solstice, the sun is in Sagittarius; it once appeared in this constellation, giving a geographic feature its name", "question": "What is Capricorn?" }, { "category": "Television", "answer": "Mike Post combined the sound of a slamming jail door, an anvil & 100 men stomping on a floor for this series that debuted in 1990", "question": "What is Law & Order?" }, { "category": "British Landmarks", "answer": "Like Sir Thomas More, 3 16th century English queens are buried at this location", "question": "What is the Tower of London?" }, { "category": "Early American History", "answer": "In 1692 Increase Mather wrote, \"It were better that ten suspected\" these \"escape, than that one innocent person … be condemned\"", "question": "What are witches?" }, { "category": "Geography Mnemonics", "answer": "Mimal, sometimes said to be the silhouette of a chef or elf, stands for Minnesota, Iowa, Missouri, and these 2 states", "question": "What are Arkansas and Louisiana?" }, { "category": "Business Milestones", "answer": "These were first sold in 1908, at a price equivalent to about $27,000 today", "question": "What is the Ford Model T?" }, { "category": "In The Bookstore", "answer": "The name of this author dead since 2013 now appears on books written by a former U.S. marshal & a former Apache helicopter pilot", "question": "Who is Tom Clancy?" }, { "category": "Historic Art", "answer": "The artwork once known in France as \"la tapisserie de la Reine Mathilde\" is better known as this", "question": "What is the Bayeux Tapestry?" }, { "category": "Pop Stars", "answer": "In 2022 she became the first woman to have a Billboard Top 10 album in 5 decades starting with the 1980s", "question": "Who is Madonna?" }, { "category": "Classic Tale Characters", "answer": "In one 19th century translation, she \"perceived the dawn of day and ceased\" speaking nearly 1,000 times", "question": "Who is Scheherazade?" }, { "category": "USA", "answer": "Ironically, though this company founded in the 1860s is Moore County, Tennessee's largest employer, Moore is a dry county", "question": "What is Jack Daniel's?" }, { "category": "Historic People", "answer": "After a 1789 event, he wrote, \"My first determination was to seek a supply of…water at Tofoa, & afterwards to sail for Tongataboo\"", "question": "Who was William Bligh?" }, { "category": "The Movies", "answer": "Laurence Olivier & Ernest Borgnine were considered for the lead role & Sergio Leone to direct for this film that turned 50 in 2022", "question": "What is The Godfather?" }, { "category": "Continental Geography", "answer": "Until a 1903 secession, this country's contiguous territory spanned 2 continents", "question": "What is Colombia?" }, { "category": "Foreign-Born Authors", "answer": "Early in her career she translated romance novels into Spanish, often changing the dialogue to make the heroines smarter", "question": "Who is Isabel Allende?" }, { "category": "Historic Crimes", "answer": "Saying it was stolen by Napoleon, self-styled Italian patriot Vincenzo Peruggia took it in 1911", "question": "What is the Mona Lisa?" }, { "category": "U.S. Bodies of Water", "answer": "Continuing a downward trend, in July 2022 it was at 27% capacity, its lowest level since 1937 when it was first being filled", "question": "What is Lake Mead?" }, { "category": "Gods & Goddesses", "answer": "Each morning she began her ride in her chariot across the sky ahead of her brother Sol, or Helios", "question": "Who is Aurora (or Eos)?" }, { "category": "America At War", "answer": "Until the Civil War, the Jan. 8 date of this battle of dubious military importance but big morale value was a national holiday", "question": "What is the Battle of New Orleans?" }, { "category": "Children's Books", "answer": "Its title character is told \"By the time you are real, most of your hair has been loved off…your eyes drop out & you get…shabby\"", "question": "What is The Velveteen Rabbit?" }, { "category": "TV Finales", "answer": "In a reunion over 40 years in the making, Dolly Parton appeared as an angel named Agnes in the final episode of this comedy in 2022", "question": "What is Grace and Frankie?" }, { "category": "American Poems", "answer": "In an 1847 poem this character sees her town of Grand-Pré burned, but finally reunites with her beau for a kiss before his death", "question": "Who is Evangeline?" }, { "category": "Famous Names", "answer": "In 2001 he published a book called \"Banging Your Head Against a Brick Wall\"; in 2002, \"Existencilism\"", "question": "Who is Banksy?" }, { "category": "Children's Lit", "answer": "The title object of this book \"never looked more beautiful…each strand held dozens of bright drops of early morning dew\"", "question": "What is Charlotte's Web?" }, { "category": "Classic Songs", "answer": "The shouts of excited children at a 1946 holiday parade are said to have inspired this perennial favorite", "question": "What is \"Here Comes Santa Claus\"?" }, { "category": "Brand Names", "answer": "Unable to make these candies perfectly round, the confectioner embraced this flawed name for the product", "question": "What are Milk Duds?" }, { "category": "Countries of the World", "answer": "It's home to 58 UNESCO World Heritage Sites, more than any other country; the sites include a volcano & a lagoon", "question": "What is Italy?" }, { "category": "Action Movies", "answer": "Its last line is \"If this is their idea of Christmas, I gotta be here for New Year's\"", "question": "What is Die Hard?" }, { "category": "Presidential Facts", "answer": "Only 3 presidents have married while in office— John Tyler was the first & he was the last", "question": "Who is Woodrow Wilson?" }, { "category": "19th Century Americans", "answer": "Demonstrating the dignity & humanity of Black Americans, he sat for 160 known photographs, the most of any American in the 19th century", "question": "Who is Frederick Douglass?" }, { "category": "Latin Phrases", "answer": "Originally, this 3-word phrase referred to when a doctor or apothecary substituted one medicine for another", "question": "What is \"quid pro quo\"?" }, { "category": "1970s Movies", "answer": "A 1975 premiere of this comedy advertised free coconuts for the first thousand in the audience", "question": "What is Monty Python and the Holy Grail?" }, { "category": "Name's The Same", "answer": "A cocktail, an island & a WWII venture originally called \"Development of Substitute Materials\" all bear this name", "question": "What is Manhattan?" }, { "category": "U.S. Presidents", "answer": "He was sworn in twice as President within 2 years, first by his father & then later by a former U.S. President", "question": "Who is Calvin Coolidge?" }, { "category": "Plays", "answer": "A 1609 story in which an exiled king of Bulgaria creates a sea palace with his magic may have inspired the plot of this play", "question": "What is The Tempest?" }, { "category": "Landmarks", "answer": "In 2009, during a 20th anniversary celebration, it was called \"an edifice of fear. On Nov. 9, it became a place of joy\"", "question": "What is the Berlin Wall?" }, { "category": "World Capitals", "answer": "Among its nicknames are the \"City of Classical Music\" &, possibly in honor of a famous resident from 1860 to 1938, the \"City of Dreams\"", "question": "What is Vienna, Austria?" }, { "category": "Language & Its Meanings", "answer": "Now meaning someone with nocturnal habits, it catches a sleeping dove in Shakespeare's \"…Lucrece\"", "question": "What is a night owl?" }, { "category": "Flags of Our Hemisphere", "answer": "The stars on this country's flag represent states, 26 of them; unlike the USA's, its \"federal district\" gets its own 27th star", "question": "What is Brazil?" }, { "category": "Names in U.S. History", "answer": "This father was the only man among the 13 plaintiffs in a class-action case filed in 1951", "question": "Who is Oliver Brown?" }, { "category": "Children's Authors", "answer": "Reversing the story of this heroine she created, Patricia Maclachlan was born on the prairie but spent much of her life in New England", "question": "Who is Sarah? (from Sarah, Plain and Tall)" } ] ```

[SlyEcho]

I got a notebook here where I use Langchain to create prompts to evaluate the quiz data.

It looks like I was right in that the models can just play Jeopardy without needing to change the data.

https://gist.github.com/SlyEcho/a1e6ac9e44eb48a6769b44b61050b635

[CRD716]

1168 is in a usable state now.