trial with wizardcoder-python-34b-v1.0.Q5_K_M.llamafile on substrate/NUC

[yoshikiohshima]

I ran wizardcoder-python-34b-v1.0.Q5_K_M.llamafile on substrate/NUC and ran the benchmark program in experiments/timeline-benchmark(with one line modification to APP_URL to access an LLM on network.

 # node benchmark.js less-ambigious.txt less-ambigious.txt 

Interestingly it has very strong "no" bias and out of 121 trials, where 11 of those should be yes, it gave me only one "yes":

no, 1965: Butler Lampson's thesis., 1965: Butler Lampson's thesis.
no, 1965: Butler Lampson's thesis., 1966: Simula introduced by Nygaard and Dahl.
no, 1965: Butler Lampson's thesis., 1967: Case Western Reserve University founded.
no, 1965: Butler Lampson's thesis., 1969: ARPANET started running.
no, 1965: Butler Lampson's thesis., 1970: Xerox PARC founded.
no, 1965: Butler Lampson's thesis., 1972: Smalltalk created.
no, 1965: Butler Lampson's thesis., 1979: C++ development started by Bjarne Stroustrup.
no, 1965: Butler Lampson's thesis., 1981: MS-DOS introduced.
no, 1965: Butler Lampson's thesis., 1984: X Windows introduced.
no, 1965: Butler Lampson's thesis., 1989: HTML on the Internet's foundational technology proposed by Tim Berners-Lee.
no, 1965: Butler Lampson's thesis., 1995: Java programming language and Java VM first released by Sun Microsystems.
no, 1966: Simula introduced by Nygaard and Dahl., 1965: Butler Lampson's thesis.
no, 1966: Simula introduced by Nygaard and Dahl., 1966: Simula introduced by Nygaard and Dahl.
yes, 1966: Simula introduced by Nygaard and Dahl., 1967: Case Western Reserve University founded.
no, 1966: Simula introduced by Nygaard and Dahl., 1969: ARPANET started running.
no, 1966: Simula introduced by Nygaard and Dahl., 1970: Xerox PARC founded.
no, 1966: Simula introduced by Nygaard and Dahl., 1972: Smalltalk created.
no, 1966: Simula introduced by Nygaard and Dahl., 1979: C++ development started by Bjarne Stroustrup.
no, 1966: Simula introduced by Nygaard and Dahl., 1981: MS-DOS introduced.
no, 1966: Simula introduced by Nygaard and Dahl., 1984: X Windows introduced.
no, 1966: Simula introduced by Nygaard and Dahl., 1989: HTML on the Internet's foundational technology proposed by Tim Berners-Lee.
no, 1966: Simula introduced by Nygaard and Dahl., 1995: Java programming language and Java VM first released by Sun Microsystems.
no, 1967: Case Western Reserve University founded., 1965: Butler Lampson's thesis.
no, 1967: Case Western Reserve University founded., 1966: Simula introduced by Nygaard and Dahl.
no, 1967: Case Western Reserve University founded., 1967: Case Western Reserve University founded.
no, 1967: Case Western Reserve University founded., 1969: ARPANET started running.
no, 1967: Case Western Reserve University founded., 1970: Xerox PARC founded.
no, 1967: Case Western Reserve University founded., 1972: Smalltalk created.
no, 1967: Case Western Reserve University founded., 1979: C++ development started by Bjarne Stroustrup.
no, 1967: Case Western Reserve University founded., 1981: MS-DOS introduced.
no, 1967: Case Western Reserve University founded., 1984: X Windows introduced.
no, 1967: Case Western Reserve University founded., 1989: HTML on the Internet's foundational technology proposed by Tim Berners-Lee.
no, 1967: Case Western Reserve University founded., 1995: Java programming language and Java VM first released by Sun Microsystems.
no, 1969: ARPANET started running., 1965: Butler Lampson's thesis.
no, 1969: ARPANET started running., 1966: Simula introduced by Nygaard and Dahl.
no, 1969: ARPANET started running., 1967: Case Western Reserve University founded.
no, 1969: ARPANET started running., 1969: ARPANET started running.
no, 1969: ARPANET started running., 1970: Xerox PARC founded.
no, 1969: ARPANET started running., 1972: Smalltalk created.
no, 1969: ARPANET started running., 1979: C++ development started by Bjarne Stroustrup.
no, 1969: ARPANET started running., 1981: MS-DOS introduced.
no, 1969: ARPANET started running., 1984: X Windows introduced.
no, 1969: ARPANET started running., 1989: HTML on the Internet's foundational technology proposed by Tim Berners-Lee.
no, 1969: ARPANET started running., 1995: Java programming language and Java VM first released by Sun Microsystems.
no, 1970: Xerox PARC founded., 1965: Butler Lampson's thesis.
no, 1970: Xerox PARC founded., 1966: Simula introduced by Nygaard and Dahl.
no, 1970: Xerox PARC founded., 1967: Case Western Reserve University founded.
no, 1970: Xerox PARC founded., 1969: ARPANET started running.
no, 1970: Xerox PARC founded., 1970: Xerox PARC founded.
no, 1970: Xerox PARC founded., 1972: Smalltalk created.
no, 1970: Xerox PARC founded., 1979: C++ development started by Bjarne Stroustrup.
no, 1970: Xerox PARC founded., 1981: MS-DOS introduced.
no, 1970: Xerox PARC founded., 1984: X Windows introduced.
no, 1970: Xerox PARC founded., 1989: HTML on the Internet's foundational technology proposed by Tim Berners-Lee.
no, 1970: Xerox PARC founded., 1995: Java programming language and Java VM first released by Sun Microsystems.
no, 1972: Smalltalk created., 1965: Butler Lampson's thesis.
no, 1972: Smalltalk created., 1966: Simula introduced by Nygaard and Dahl.
no, 1972: Smalltalk created., 1967: Case Western Reserve University founded.
no, 1972: Smalltalk created., 1969: ARPANET started running.
no, 1972: Smalltalk created., 1970: Xerox PARC founded.
no, 1972: Smalltalk created., 1972: Smalltalk created.
no, 1972: Smalltalk created., 1979: C++ development started by Bjarne Stroustrup.
no, 1972: Smalltalk created., 1981: MS-DOS introduced.
no, 1972: Smalltalk created., 1984: X Windows introduced.
no, 1972: Smalltalk created., 1989: HTML on the Internet's foundational technology proposed by Tim Berners-Lee.
no, 1972: Smalltalk created., 1995: Java programming language and Java VM first released by Sun Microsystems.
no, 1979: C++ development started by Bjarne Stroustrup., 1965: Butler Lampson's thesis.
no, 1979: C++ development started by Bjarne Stroustrup., 1966: Simula introduced by Nygaard and Dahl.
no, 1979: C++ development started by Bjarne Stroustrup., 1967: Case Western Reserve University founded.
no, 1979: C++ development started by Bjarne Stroustrup., 1969: ARPANET started running.
no, 1979: C++ development started by Bjarne Stroustrup., 1970: Xerox PARC founded.
no, 1979: C++ development started by Bjarne Stroustrup., 1972: Smalltalk created.
no, 1979: C++ development started by Bjarne Stroustrup., 1979: C++ development started by Bjarne Stroustrup.
no, 1979: C++ development started by Bjarne Stroustrup., 1981: MS-DOS introduced.
no, 1979: C++ development started by Bjarne Stroustrup., 1984: X Windows introduced.
no, 1979: C++ development started by Bjarne Stroustrup., 1989: HTML on the Internet's foundational technology proposed by Tim Berners-Lee.
no, 1979: C++ development started by Bjarne Stroustrup., 1995: Java programming language and Java VM first released by Sun Microsystems.
no, 1981: MS-DOS introduced., 1965: Butler Lampson's thesis.
no, 1981: MS-DOS introduced., 1966: Simula introduced by Nygaard and Dahl.
no, 1981: MS-DOS introduced., 1967: Case Western Reserve University founded.
no, 1981: MS-DOS introduced., 1969: ARPANET started running.
no, 1981: MS-DOS introduced., 1970: Xerox PARC founded.
no, 1981: MS-DOS introduced., 1972: Smalltalk created.
no, 1981: MS-DOS introduced., 1979: C++ development started by Bjarne Stroustrup.
no, 1981: MS-DOS introduced., 1981: MS-DOS introduced.
no, 1981: MS-DOS introduced., 1984: X Windows introduced.
no, 1981: MS-DOS introduced., 1989: HTML on the Internet's foundational technology proposed by Tim Berners-Lee.
no, 1981: MS-DOS introduced., 1995: Java programming language and Java VM first released by Sun Microsystems.
no, 1984: X Windows introduced., 1965: Butler Lampson's thesis.
no, 1984: X Windows introduced., 1966: Simula introduced by Nygaard and Dahl.
no, 1984: X Windows introduced., 1967: Case Western Reserve University founded.
no, 1984: X Windows introduced., 1969: ARPANET started running.
no, 1984: X Windows introduced., 1970: Xerox PARC founded.
no, 1984: X Windows introduced., 1972: Smalltalk created.
no, 1984: X Windows introduced., 1979: C++ development started by Bjarne Stroustrup.
no, 1984: X Windows introduced., 1981: MS-DOS introduced.
no, 1984: X Windows introduced., 1984: X Windows introduced.
no, 1984: X Windows introduced., 1989: HTML on the Internet's foundational technology proposed by Tim Berners-Lee.
no, 1984: X Windows introduced., 1995: Java programming language and Java VM first released by Sun Microsystems.
no, 1989: HTML on the Internet's foundational technology proposed by Tim Berners-Lee., 1965: Butler Lampson's thesis.
no, 1989: HTML on the Internet's foundational technology proposed by Tim Berners-Lee., 1966: Simula introduced by Nygaard and Dahl.
no, 1989: HTML on the Internet's foundational technology proposed by Tim Berners-Lee., 1967: Case Western Reserve University founded.
no, 1989: HTML on the Internet's foundational technology proposed by Tim Berners-Lee., 1969: ARPANET started running.
no, 1989: HTML on the Internet's foundational technology proposed by Tim Berners-Lee., 1970: Xerox PARC founded.
no, 1989: HTML on the Internet's foundational technology proposed by Tim Berners-Lee., 1972: Smalltalk created.
no, 1989: HTML on the Internet's foundational technology proposed by Tim Berners-Lee., 1979: C++ development started by Bjarne Stroustrup.
no, 1989: HTML on the Internet's foundational technology proposed by Tim Berners-Lee., 1981: MS-DOS introduced.
no, 1989: HTML on the Internet's foundational technology proposed by Tim Berners-Lee., 1984: X Windows introduced.
no, 1989: HTML on the Internet's foundational technology proposed by Tim Berners-Lee., 1989: HTML on the Internet's foundational technology proposed by Tim Berners-Lee.
no, 1989: HTML on the Internet's foundational technology proposed by Tim Berners-Lee., 1995: Java programming language and Java VM first released by Sun Microsystems.
no, 1995: Java programming language and Java VM first released by Sun Microsystems., 1965: Butler Lampson's thesis.
no, 1995: Java programming language and Java VM first released by Sun Microsystems., 1966: Simula introduced by Nygaard and Dahl.
no, 1995: Java programming language and Java VM first released by Sun Microsystems., 1967: Case Western Reserve University founded.
no, 1995: Java programming language and Java VM first released by Sun Microsystems., 1969: ARPANET started running.
no, 1995: Java programming language and Java VM first released by Sun Microsystems., 1970: Xerox PARC founded.
no, 1995: Java programming language and Java VM first released by Sun Microsystems., 1972: Smalltalk created.
no, 1995: Java programming language and Java VM first released by Sun Microsystems., 1979: C++ development started by Bjarne Stroustrup.
no, 1995: Java programming language and Java VM first released by Sun Microsystems., 1981: MS-DOS introduced.
no, 1995: Java programming language and Java VM first released by Sun Microsystems., 1984: X Windows introduced.
no, 1995: Java programming language and Java VM first released by Sun Microsystems., 1989: HTML on the Internet's foundational technology proposed by Tim Berners-Lee.
no, 1995: Java programming language and Java VM first released by Sun Microsystems., 1995: Java programming language and Java VM first released by Sun Microsystems.
{
  hit: {
    '0': 0,
    '1': 1,
    '2': 0,
    '3': 0,
    '4': 0,
    '5': 0,
    '6': 0,
    '7': 0,
    '8': 0,
    '9': 0,
    '10': 0
  },
  notMissed: { '2': true }
}
total = 11, hit = 1, missed = 10

[yoshikiohshima]

Come to think of it, the side that checks the result is true or false might be confused with the result it gives.

[yoshikiohshima]

Yes, that was the case, it cannot follow the instruction to only produce "yes" and "no". But also it gets it wrong:

result Yes, these two entries describe the same event.
yes, 1965: Butler Lampson's thesis., 1967: Case Western Reserve University founded.