Open Viren6 opened 1 year ago
worker output with fishtest cutechess-cli
v1.2.0
make[1]: Leaving directory '/home/usr00/_git/__test_folder00/fishtest/worker/tmprbf1h1wp/Viren6-Stockfish-1565b94/src'
strip stockfish
Obtaining EvalFile of stockfish_1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c ...
Obtaining EvalFile of stockfish_1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c ...
Verifying signature of stockfish_1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c ...
Verifying signature of stockfish_1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c ...
CPU factor : 1.1675572331082744 - tc adjusted to 70.053+0.701
CPU factor : 1.1675572331082744 - tc adjusted to 140.107+1.401
CPU factor : 1.1675572331082744 - tc adjusted to 140.107+1.401
Post request https://dfts-0.pigazzini.it:443/api/request_spsa handled in 457.91ms (server: 10.23ms)
TC limit 11208.549437839432 End time: 2023-09-02 16:23:26.362651+00:00
Indexing opening suite...
Warning: Engine Base-1565b948c3(1): no response to ping
Terminating process of engine Base-1565b948c3(1)
Warning: Engine Base-1565b948c3(2): no response to ping
Terminating process of engine Base-1565b948c3(2)
Warning: Unexpected result claim from Base-1565b948c3: 1/2-1/2 {Draw by stalled connection}
Warning: Engine New-1565b948c3(3): no response to ping
Terminating process of engine New-1565b948c3(3)
Finished game 2 (Base-1565b948c3 vs New-1565b948c3): * {No result}
Score of New-1565b948c3 vs Base-1565b948c3: 0 - 0 - 0 [nan] 0
Warning: Unexpected result claim from Base-1565b948c3: 1/2-1/2 {Draw by stalled connection}
Warning: Engine New-1565b948c3(0): no response to ping
Terminating process of engine New-1565b948c3(0)
Finished game 1 (New-1565b948c3 vs Base-1565b948c3): * {No result}
Score of New-1565b948c3 vs Base-1565b948c3: 0 - 0 - 0 [nan] 0
Elo difference: nan +/- nan, LOS: nan %, DrawRatio: nan %
Finished match
Waiting for cutechess-cli to finish ... done
Exception running games:
Finished match uncleanly
Informing the server
Post request https://dfts-0.pigazzini.it:443/api/failed_task handled in 171.60ms (server: 1.33ms)
Task exited
error using a brand new cutechess-cli
v1.3.1:
Obtaining EvalFile of stockfish_1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c ...
Obtaining EvalFile of stockfish_1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c ...
Verifying signature of stockfish_1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c ...
Verifying signature of stockfish_1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c ...
CPU factor : 1.7885788064590051 - tc adjusted to 107.315+1.073
CPU factor : 1.7885788064590051 - tc adjusted to 214.629+2.146
CPU factor : 1.7885788064590051 - tc adjusted to 214.629+2.146
Post request https://dfts-0.pigazzini.it:443/api/request_spsa handled in 599.47ms (server: 8.27ms)
TC limit 17170.356542006448 End time: 2023-09-02 18:18:56.569698+00:00
Indexing opening suite...
Warning: Engine Base-1565b948c3(1): no response to ping
Terminating process of engine Base-1565b948c3(1)
Warning: Engine New-1565b948c3(4): no response to ping
Terminating process of engine New-1565b948c3(4)
Warning: Engine New-1565b948c3(3): no response to ping
Terminating process of engine New-1565b948c3(3)
Warning: Engine Base-1565b948c3(6): no response to ping
Terminating process of engine Base-1565b948c3(6)
Warning: Engine New-1565b948c3(0): no response to ping
Terminating process of engine New-1565b948c3(0)
Warning: Engine Base-1565b948c3(5): no response to ping
Terminating process of engine Base-1565b948c3(5)
Waiting for cutechess-cli to finish ... done
Task exited
Current time is 2023-09-02 13:33:22.257951+00:00 UTC (local offset: +02:00)
Verify worker version...
Post request https://dfts-0.pigazzini.it:443/api/request_version handled in 194.49ms (server: 0.13ms)
Remaining number of github api calls = 4998
Fetching task...
the worker get correctly all the parameters from the server:
print(cmd)
try:
with subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
universal_newlines=True,
bufsize=1,
# The next options are necessary to be able to send a CTRL_C_EVENT to this process.
# https://stackoverflow.com/questions/7085604/sending-c-to-python-subprocess-objects-on-windows
startupinfo=subprocess.STARTUPINFO(
dwFlags=subprocess.STARTF_USESHOWWINDOW, wShowWindow=subprocess.SW_HIDE
)
if IS_WINDOWS
else None,
creationflags=subprocess.CREATE_NEW_CONSOLE if IS_WINDOWS else 0,
close_fds=not IS_WINDOWS,
) as p:
File with the command line: cute-cli.txt
The brand new cutechess-cli
v1.3.1 segfault when given that command line
option.PolicyMap[63][56]=0 option.PolicyMap[63][57]=0 option.PolicyMap[63][58]=0 option.PolicyMap[63][59]=0 option.PolicyMap[63][60]=0 option.PolicyMap[63][61]=14297 option.PolicyMap[63][62]=14297 option.PolicyMap[63][63]=0 -each proto=uci option.Threads=1ap[63][55]=0
Indexing opening suite...
Warning: Engine New-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c(4): no response to ping
Terminating process of engine New-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c(4)
Warning: Engine Base-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c(2): no response to ping
Terminating process of engine Base-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c(2)
Warning: Engine Base-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c(1): no response to ping
Terminating process of engine Base-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c(1)
Warning: Engine Base-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c(6): no response to ping
Terminating process of engine Base-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c(6)
Warning: Engine New-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c(7): no response to ping
Terminating process of engine New-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c(7)
Warning: Engine New-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c(3): no response to ping
Terminating process of engine New-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c(3)
Segmentation fault
This is the output of the fishtest cutechess-cli
v1.2.0 when given that command line:
option.PolicyMap[63][60]=0 option.PolicyMap[63][61]=14297 option.PolicyMap[63][62]=14297 option.PolicyMap[63][63]=0 -each proto=uci option.Threads=1
Indexing opening suite...
Warning: Engine New-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c(0): no response to ping
Terminating process of engine New-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c(0)
Warning: Engine Base-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c(2): no response to ping
Terminating process of engine Base-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c(2)
Warning: Engine New-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c(7): no response to ping
Terminating process of engine New-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c(7)
Warning: Engine Base-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c(5): no response to ping
Terminating process of engine Base-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c(5)
Warning: Unexpected result claim from Base-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c: 1/2-1/2 {Draw by stalled connection}
Warning: Engine New-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c(3): no response to ping
Terminating process of engine New-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c(3)
Finished game 2 (Base-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c vs New-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c): * {No result}
Score of New-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c vs Base-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c: 0 - 0 - 0 [nan] 0
Warning: Unexpected result claim from Base-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c: 1/2-1/2 {Draw by stalled connection}
Warning: Engine New-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c(4): no response to ping
Finished game 3 (New-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c vs Base-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c): * {No result}
Terminating process of engine New-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c(4)
Score of New-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c vs Base-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c: 0 - 0 - 0 [nan] 0
Warning: Unexpected result claim from New-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c: 1/2-1/2 {Draw by stalled connection}
Warning: Engine Base-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c(1): no response to ping
Terminating process of engine Base-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c(1)
Finished game 1 (New-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c vs Base-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c): * {No result}
Score of New-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c vs Base-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c: 0 - 0 - 0 [nan] 0
Warning: Unexpected result claim from New-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c: 1/2-1/2 {Draw by stalled connection}
Warning: Engine Base-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c(6): no response to pingFinished game 4 (Base-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c vs New-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c): * {No result}
Terminating process of engine Base-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c(6)
Score of New-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c vs Base-1565b948c3d3b2cc4cd0b3e757bfc906fb8a645c: 0 - 0 - 0 [nan] 0
Elo difference: nan +/- nan, LOS: nan %, DrawRatio: nan %
Finished match
IMO the only problem of fishtest is to no check the (unknown to me) or cutechess-cli
or Stockfish optional parameters limit.
Your runs do not show worker errors though. I am wondering where the
Exception AssertionError at games.py:819
Exception ValueError at games.py:900
come from that the OP is reporting.
This code is the culprit it seems https://github.com/official-stockfish/fishtest/blob/13297926d25a497d4ceefcc9c32bbdfa823e2971/worker/games.py#L895-L905
There are two different logical level in action:
cutechess-cli
or a Stockfish problem with the number of the optional parameterscutechess-cli
/Stockfish parameters limit, unexpected match results, cutechess-cli
segmentation fault etc.Yes. But 2. does not show in your runs. However it is reported in the event log.
It think sometimes the parsing of the "Score" lines fails. It would be nice to see an example of this.
As a stop gap measure we should protect this parsing with a try: except: construct.
I might add that windows doesnt even allow such long cli invocations... see https://tests.stockfishchess.org/actions?action=&user=mgrabiak&text= and I reproduced it locally too.
edit: the max length is 32767 characters, https://devblogs.microsoft.com/oldnewthing/20031210-00/?p=41553
Yes. But 2. does not show in your runs. However it is reported in the event log.
It think sometimes the parsing of the "Score" lines fails. It would be nice to see an example of this.
As a stop gap measure we should protect this parsing with a try: except: construct.
Check the cutechess-cli
v1.3.0 output, no lines with "Score", and cutechess-cli segfault as well.
edit: the max length is 32767 characters, https://devblogs.microsoft.com/oldnewthing/20031210-00/?p=41553
The command line is over 235k characters
I've run SPSAs with commands longer than that limit. I think even this currently running one is: https://tests.stockfishchess.org/tests/view/64f32673a0f9f22deb74cc61
@ppigazzini Check the cutechess-cli v1.3.0 output, no lines with "Score", and cutechess-cli segfault as well.
Yes but in your runs the worker seems to handle it gracefully. There are no uncaught exceptions.
@ppigazzini Check the cutechess-cli v1.3.0 output, no lines with "Score", and cutechess-cli segfault as well.
Yes but in your runs the worker seems to handle it gracefully. There are no uncaught exceptions.
Feel free to replicate with your PC.
In my local tests the real problem is in cutechess-cli/Stockfish, so we should no raise noise about this fact.
"In my local tests the real problem is in cutechess-cli/Stockfish, so we should no raise noise about this fact."
Yes but we should also understand unexpected worker behaviour (as reported by the OP). For example "Exception AssertionError at games.py:819" appears occasionally in the event log and it shouldn't (it is an assertion after all...). I have never been able to duplicate this.
I tried locally and the worker handles things gracefully (same output as yours) ...
I ran few times in WSL2 that long command line replacing cutechess-cli
with fast-chess
and the tournaments always finished, I given to Disservin (fast-chess
's author) the log of one tournament.
EDIT_000: the log seems fine, so the problem should be on cutechess-cli
side.
Attempting to run this SPSA tune (https://tests.stockfishchess.org/tests/view/64f2b8d3f5b0c54e3f04cd7e ) with 4096 parameters results in the following errors:
Exception AssertionError at games.py:819 Exception ValueError at games.py:900 Finished match uncleanly Unable to start cutechess. Error: [WinError 206] The filename or extension is too long
Event Log: