Open tissatussa opened 11 months ago
btw. i'm planning to build a Nim GUI with Owlkettle.
I think actually I never did it myself, and it is not mentioned in my book. But I read about it in the past. See
https://stackoverflow.com/questions/75712315/how-to-run-an-external-program
https://glenngillen.com/learning-nim/executing-external-commands/
https://stackoverflow.com/questions/71432607/how-to-get-what-is-on-the-terminal-in-nim-lang
And this is the answer from GPT-4. Please try yourself, it should be helpful, but GPT-4 is not free from errors:
https://chat.openai.com/share/3be72d94-e0f8-4017-abe9-0e9452bc9c54
https://glenngillen.com/learning-nim/executing-external-commands/
that link suits me the most : the last code part on that page should be able to feed the "uci" command to the stockfish
binary (my default OS test engine) but it fails : only the first Stockfish line it echoed, no UCI list, the script keeps waiting for exit .. here's my version of that code :
import osproc, strutils, streams
let process = startProcess("stockfish", args = [], options = {poInteractive, poUsePath})
let (FROMp, TOp) = (process.outputStream, process.inputStream)
TOp.write "uci\n"
TOp.flush
echo FROMp.readLine
discard process.waitForExit
process.close
i think i'm missing something obvious .. indeed i expect such in/out functionality can be done by just a few lines of Nim code, but it fails .. i tried several minor tweaks in such code, compared other documents & forum entries, too no avail .. i guess you could easily reproduce this with your stockfish binary ?
btw. it's strange to me : shouldn't waitForExit
be written as waitForExit()
in Nim ? Or are these syntax same in Nim ? Also the function write "uci\n"
: better write("uci\n")
?
i'm getting there ..! hold on..
newest working (!) code :
import osproc, streams
let process = startProcess( "stockfish", args = [], options = { poInteractive, poUsePath } )
let (fromP, toP) = ( process.outputStream , process.inputStream )
var s = ""
var TF = true
# echo the first Stockfish "welcome line"
echo fromP.readLine
fromP.flush()
toP.write("uci\n")
toP.flush
#echo all uci output lines
s = fromP.readLine
while TF:
echo s
fromP.flush()
try:
# after the last 'uciok' line, the process waits here forever : no error is catched
s = fromP.readLine
except CatchableError:
TF = false
# the script doesn't ever reach these lines
discard process.waitForExit
process.close
output :
roelof@roelof-HP-Elite-x2-1012-G2:~/Apps/NimRunExternal$ nim c ./test.nim
Hint: used config file '/home/roelof/.choosenim/toolchains/nim-1.6.12/config/nim.cfg' [Conf]
Hint: used config file '/home/roelof/.choosenim/toolchains/nim-1.6.12/config/config.nims' [Conf]
.................................................................................................
CC: test.nim
Hint: [Link]
Hint: gc: refc; opt: none (DEBUG BUILD, `-d:release` generates faster code)
54170 lines; 0.936s; 75.523MiB peakmem; proj: /home/roelof/Apps/NimRunExternal/test.nim; out: /home/roelof/Apps/NimRunExternal/test [SuccessX]
roelof@roelof-HP-Elite-x2-1012-G2:~/Apps/NimRunExternal$ ./test
Stockfish 14.1 by the Stockfish developers (see AUTHORS file)
id name Stockfish 14.1
id author the Stockfish developers (see AUTHORS file)
option name Debug Log File type string default
option name Threads type spin default 1 min 1 max 512
option name Hash type spin default 16 min 1 max 33554432
option name Clear Hash type button
option name Ponder type check default false
option name MultiPV type spin default 1 min 1 max 500
option name Skill Level type spin default 20 min 0 max 20
option name Move Overhead type spin default 10 min 0 max 5000
option name Slow Mover type spin default 100 min 10 max 1000
option name nodestime type spin default 0 min 0 max 10000
option name UCI_Chess960 type check default false
option name UCI_AnalyseMode type check default false
option name UCI_LimitStrength type check default false
option name UCI_Elo type spin default 1350 min 1350 max 2850
option name UCI_ShowWDL type check default false
option name SyzygyPath type string default <empty>
option name SyzygyProbeDepth type spin default 1 min 1 max 100
option name Syzygy50MoveRule type check default true
option name SyzygyProbeLimit type spin default 7 min 0 max 7
option name Use NNUE type check default true
option name EvalFile type string default nn-13406b1dcbe0.nnue
uciok
i must be almost there ..?
shouldn't waitForExit be written as waitForExit() in Nim ? Or are these syntax same in Nim ? Also the function write "uci\n" : better write("uci\n") ?
process.waitForExit is Nim's method call syntax for waitForExit(process). You can also write process.waitForExit() but the braces can be left out. write "uci\n" is the syntax with omitted braces which is used typically for the echo statement. I am using typically braces, and only leave them out for echo. All explained in much detail in my book.
after the last 'uciok' line, the process waits here forever : no error is catched
do you see / know why ? how can i check if all output is echoed, so my script can write() another UCI command, like "isready" or "go [..]" ? Does it hang or wait ?
that's why i'm wondering if threads are neede here : i imagine a threaded process runs "on its own" and its state can be read, so the main script can act upon this : read new output or feed new input .. am i wrong ?
I have never done this, but maybe I should add it to the book.
GPT-4 just suggested to use poParentStreams, see https://chat.openai.com/share/3be72d94-e0f8-4017-abe9-0e9452bc9c54
Generally, you could search how this communication is done in C langauge. For Nim it is then very similar. Generally, I have some doubts that such a communication gives enough freedom for user interaction, like forcing a move, give the user a hint, and all that. My impression was that such communication is typically done by network protocols like TCP.
Sorry, new GTP link does not work. Here is the text
User Can I use the input stream of the process to pass messages to it? ChatGPT
Yes, you can use the input stream of the process to send messages or data to it in Nim. This is particularly useful when you need to interact with an external process that reads from its standard input. Here's how you can do it:
Start the Process with Pipelines: When starting the process, ensure that you enable input streaming. This is done by using the poParentStreams option in startProcess.
Write to the Input Stream: Once the process is running, you can write data to its input stream. This is similar to typing input into the console when the program is running interactively.
Close the Input Stream: After sending all your messages or data, close the input stream. This often signals the external process that no more input will be provided.
Here’s an example demonstrating this:
nim
import osproc, os
let binaryPath = "/path/to/external/binary"
let proc = startProcess(binaryPath, options = {poParentStreams, poStdErrToStdOut})
proc.inputStream.write("Message to send\n") proc.inputStream.flush()
proc.inputStream.close()
let exitCode = proc.waitForExit()
let output = proc.outputStream.readAll()
echo "Process output: ", output
if exitCode == 0: echo "Process completed successfully" else: echo "Process failed with exit code: ", exitCode
proc.close()
In this code, write is used to send a string to the process's standard input. The \n at the end of the string is often necessary to signal the end of an input line, especially for console-based programs that read input line by line.
Remember to replace the message and the binary path with your specific requirements. Also, depending on the external program, you may need to format the input data differently.
Generally, I have some doubts that such a communication gives enough freedom for user interaction, like forcing a move, give the user a hint, and all that.
i only want to communicate with the engine by its normal UCI commands : "go .." will end with a best move .. before that, we can set some UCI options, the starting position etc. with isready (:readyOK) in between .. however, it would be nice to also be able to send a "stop" command which the engine will "hear" .. no need to force a move or give a hint yet ..
or maybe..
yes, i saw and studied many of those scripts, also non-Nim .. i do python and a little bash, but not C .. i really like a working Nim example ..
One more:
https://www.codeproject.com/Articles/1159746/StockChess
Sorry, I have no time for further investigations.
THANKS !!
If you don't get it working yourself, my advice would be to find a working Python, Rust, or C solution, and then ask in the Nim forum or at StackOverflow. When there is already a solution for some language, then people are typically motivated to do it in Nim as well. Or, mention that you will use Owekettle, that may motivate someone to help you. Bye.
[ i leave this Issue Open for now - i might post a final working solution here, soon ]
This Nim forum thread may be relevant. But reading delayed multi-line data seems to be difficult:
I think i solved the problem .. i will write about it tomorrow :) Your link seems to be good info .. i have to study it and compare it to my own findings .. it's a tricky thing .. indeed, probably we don't need a thread to communicate with an UCI engine .. my thinking was malformed - coding is fun .. finally.
It is difficult:
https://stackoverflow.com/questions/32301415/interfacing-with-stockfish-shell-using-c
Working with timeouts as in the Nim forum thread may work, but is not really nice.
For input, stockfish seems to wait for the quit command, but it is difficult to detect the end of stockfish's output.
Wait, I may be wrong: From https://chess.stackexchange.com/questions/28755/communication-with-stockfish-engine-via-python stockfish seems to terminate its output with message 'readyok'.
basically (and probably all i need) only 3 last output lines are possible and relevant :
uciok
is echoed by the engine after the command uci
is givenreadyok
is echoed by the engine after the command isready
is givenbestmove
is echoed by the engine after a go [..]
command is givenucinewgame
and position [..]
give no outputso, it's possible to intercept those (last) output lines and our script must respond accordingly.
it's late :) https://www.bitchute.com/video/FQHBdrSfonfg/
Have installed stockfish now, and run your code, and compared to the terminal output. Seems to be identical, which is fine. So stockfish seems to wait for new input:
$ stockfish uci
Stockfish 15 by the Stockfish developers (see AUTHORS file)
id name Stockfish 15
id author the Stockfish developers (see AUTHORS file)
option name Debug Log File type string default
option name Threads type spin default 1 min 1 max 512
option name Hash type spin default 16 min 1 max 33554432
option name Clear Hash type button
option name Ponder type check default false
option name MultiPV type spin default 1 min 1 max 500
option name Skill Level type spin default 20 min 0 max 20
option name Move Overhead type spin default 10 min 0 max 5000
option name Slow Mover type spin default 100 min 10 max 1000
option name nodestime type spin default 0 min 0 max 10000
option name UCI_Chess960 type check default false
option name UCI_AnalyseMode type check default false
option name UCI_LimitStrength type check default false
option name UCI_Elo type spin default 1350 min 1350 max 2850
option name UCI_ShowWDL type check default false
option name SyzygyPath type string default <empty>
option name SyzygyProbeDepth type spin default 1 min 1 max 100
option name Syzygy50MoveRule type check default true
option name SyzygyProbeLimit type spin default 7 min 0 max 7
option name Use NNUE type check default true
option name EvalFile type string default nn-6877cd24400e.nnue
uciok
i knew you'd be intrigued by this :) must be simple !? i should take a look at your books.
OK, this is my latest script .. it works :
import std/[streams, osproc, strutils]
let process = startProcess( "stockfish", args = [], options = { poInteractive, poUsePath } )
let (fromP, toP) = ( process.outputStream , process.inputStream )
var s = ""
var TF = true
toP.write("uci\n") # returns "uciok"
echo "<=- uci"
toP.flush
while TF:
s = fromP.readLine
echo "-=> " & s
if s == "uciok":
toP.write("ucinewgame\n") # returns nothing
toP.flush
echo "<=- ucinewgame"
toP.write("isready\n") # returns "readyok"
toP.flush
echo "<=- isready"
s = fromP.readLine
echo "-=> " & s # should be "readyok"
toP.write("position fen 3B4/1r2p3/r2p1p2/bkp1P1p1/1p1P1PPp/p1P1K2P/PPB5/8 w - - 1 1\n") # returns nothing
toP.flush
echo "<=- position fen 3B4/1r2p3/r2p1p2/bkp1P1p1/1p1P1PPp/p1P1K2P/PPB5/8 w - - 1 1"
toP.write("isready\n") # returns "readyok"
toP.flush
echo "<=- isready"
s = fromP.readLine
echo "-=> " & s # should be "readyok"
toP.write("go depth 12\n") # returns a lot of info lines, ending with "bestmove [...]"
toP.flush
echo "<=- go depth 12"
elif s == "readyok":
echo "<=- readyok"
elif s.startsWith("bestmove"):
TF = false # this will be the last echoed line
# this example script ends here,
# but we could input other UCI commands now (keeping TF true)
toP.write("quit\n")
toP.flush
discard process.waitForExit
process.close
editted .. tomorrow :)
Fine that it works for you. I can not really continue testing, because I do not know all the stockfish commands.
I hope you will have success with stockfish and owlkettle. The https://github.com/elcritch/figuro package seems to be still in an too early state to use it for a serious project.
I think I will convert my chess game to Rust this weekend, should be not that hard, as the code is short and only uses Nim concepts that are supported by Rust as well, that is no exceptions, no inheritance. I just tried a plain Gintro to GTK-rs translation, see https://discourse.gnome.org/t/fixing-rust-gtk4-code-with-gpt-4-works-really-well/18143. With some help of GPT-4 it was easy. But of course the Rust code looks not that clean as Nim or Python code, and is a bit longer.
I hope you will have success with stockfish and owlkettle
my script works now .. it was easy, at last .. i adjusted the previous example code to my latest version today .. it works with any (?) UCI chess engine binary AFAIK. Also read my comments in the code. The page https://nim-lang.org/docs/osproc.html gives all info.
Owlkettle seems one of few good tools to create a GUI with Nim .. i follow the PRs of this GitHub project, still lots of changes and new things.
btw. you should consider studying the UCI protocol, for various reasons .. it's not hard and well documented .. i also propose your newest engine should be UCI driven, eg. let it play in CuteChess against other engines. Good luck with the Rust adventure, i'm looking forward to the result !
looking for answers i found some .. and it's working now, more or less .. my latest discovery and question is at https://forum.nim-lang.org/t/10719
do you know how to set PRIORITY_HIGH_IDLE ?
FYI: here is my latest script, "unstable" though, as i wrote at the forum ..
import std/[streams, os, osproc, strutils, strformat]
import owlkettle
viewable App:
buffer: TextBuffer
monospace: bool = false
cursorVisible: bool = true
editable: bool = true
acceptsTab: bool = true
indent: int = 0
sensitive: bool = true
sizeRequest: tuple[x, y: int] = (-1, -1)
tooltip: string = ""
var MS = 1 # sleep [#] MilliSeconds
var DP = 18 # eval DePth
var TFecho = true
#var TFecho = false
when compileOption("threads"):
var thread: Thread[AppState]
proc chgBuffer(app: AppState, s: string) =
var abt = ""
var s2 = ""
if TFecho:
echo s
try:
abt = app.buffer.text
except Exception as e:
var error = getCurrentException()
echo "*** ERROR ABT 1"
try:
s2 = abt & s & "\n"
app.buffer.text = s2
#app.buffer.text = s & "\n"
#app.buffer.text = app.buffer.text & s & "\n"
except Exception as e:
var error = getCurrentException()
echo "*** ERROR ABT 2"
sleep(500)
try:
app.redrawFromThread()
except Exception as e:
var error = getCurrentException()
echo "*** ERROR ABT 3"
sleep(500)
#discard
proc threadProc(app: AppState) {.thread.} =
#let process = startProcess( "./examples/widgets/", args = [], options = { poInteractive, poUsePath } )
let process = startProcess( "./examples/widgets/Clubfoot_1.0.8e3b4da_compiled_HP", args = [], options = { poInteractive, poUsePath } )
#let process = startProcess( "./examples/widgets/amoeba_v3.4_compiled_fast", args = [], options = { poInteractive, poUsePath } )
#let process = startProcess( "./examples/widgets/camel_v1.2.0-PR90-compiled_HP", args = [], options = { poInteractive, poUsePath } ) # ???
let (fromP, toP) = ( process.outputStream , process.inputStream )
var s = ""
var TF = false
proc mySleep() =
sleep(MS)
#discard
toP.write("uci\n") # returns "uciok"
toP.flush
s = fromP.readLine # should return "uciok"
mySleep()
chgBuffer(app,s)
TF = true
while TF:
s = fromP.readLine
if not isNil(s) and len(s.strip()) > 0:
mySleep()
else:
echo "*** CONTINUE A"
continue
chgBuffer(app, "-=> " & s)
if s == "uciok":
toP.write("ucinewgame\n") # returns nothing
toP.flush
mySleep()
chgBuffer(app,"<=- ucinewgame")
toP.write("isready\n") # returns "readyok"
toP.flush
mySleep()
chgBuffer(app,"<=- isready")
s = fromP.readLine
if not isNil(s) and len(s.strip()) > 0:
mySleep()
else:
echo "*** CONTINUE B"
continue
# should be "readyok"
chgBuffer(app,"-=> " & s)
toP.write("position fen 3B4/1r2p3/r2p1p2/bkp1P1p1/1p1P1PPp/p1P1K2P/PPB5/8 w - - 1 1\n") # returns nothing
toP.flush
mySleep()
chgBuffer(app,"<=- position fen 3B4/1r2p3/r2p1p2/bkp1P1p1/1p1P1PPp/p1P1K2P/PPB5/8 w - - 1 1")
toP.write("isready\n") # returns "readyok"
toP.flush
mySleep()
chgBuffer(app,"<=- isready")
s = fromP.readLine
if not isNil(s) and len(s.strip()) > 0:
mySleep()
else:
echo "*** CONTINUE C"
continue
# should be "readyok"
chgBuffer(app,"-=> " & s)
# returns a lot of info lines, ending with "bestmove [...]"
toP.write(&"go depth {DP}\n") # NOTE: using fmt"..." does NOT work here !? : use the '&' syntax..
toP.flush
mySleep()
chgBuffer(app, fmt"<=- go depth {DP}")
elif s == "readyok":
try:
chgBuffer(app,"<=- readyok")
except Exception as e:
echo "FAILED READYOK ***********************"
elif s.startsWith("bestmove"):
TF = false # this will be the last echoed line
# this example script ends here,
# but we could input other UCI commands now (keeping TF true)
#sleep(1000)
toP.write("quit\n")
toP.flush
mySleep()
echo "<=- quit"
discard process.waitForExit
process.close
echo "MSG : process closed"
method view(app: AppState): Widget =
result = gui:
Window:
title = "*** TEST ***"
defaultSize = (1100, 600)
HeaderBar {.addTitlebar.}:
Button {.addLeft.}:
style = [ButtonFlat]
text = "GO"
proc clicked() =
when compileOption("threads"):
createThread(thread, threadProc, app)
ScrolledWindow:
TextView:
margin = 12
buffer = app.buffer
monospace = app.monospace
cursorVisible = app.cursorVisible
editable = app.editable
acceptsTab = app.acceptsTab
indent = app.indent
sensitive = app.sensitive
tooltip = app.tooltip
sizeRequest = app.sizeRequest
proc changed() = discard
#when compileOption("threads") and (defined(gcOrc) or defined(gcArc)):
#when compileOption("threads") and defined(mmArc):
when compileOption("threads"):
let buffer = newTextBuffer()
buffer.text = "first text line\n"
brew(gui(App(buffer = buffer)))
else:
#quit "Compile with --threads:on and --gc:orc to enable threads" # ORG
quit "Compile with --threads:on and --mm:arc to enable threads"
btw. as you see, i used try - except
, but i never caught any exception, nothing was ever echoed..
Sorry, for multi-threading with Owlkettle I can not help you. Multi-threading with GTK is generally not that easy. My GTK4 book may have some examples, or there may be at least some examples in the Gintro package. I think one was drawing a sine wave. But for Owlkettle all may be different. And while for Gintro typical program layout was close to C layout, so we always could ask at Gnome forum, for Owlkettle it may be very different. For threading and channel communication, I gave some examples in my Nim book.
I concentrate on learning Rust myself now, which is some fun. I guess that chances that I may ever return to Nim are not that big. Some days ago I found a Rust chess book, see https://rustic-chess.org/front_matter/about_book.html. Looks nice. Have you at least been able to draw a chessboard with Owlkettle? Should be not difficult, when Owlkettle has Cairo or other drawing support -- maybe I could add your board to my salewski-chess.
For threading and channel communication, I gave some examples in my Nim book.
today i learned about Channels in my forum post https://forum.nim-lang.org/t/10719 .. user @Isofruit delved into the matter, also according to @Araq, and posted a basic working Owlkettle example, which seems well made .. i'm beginning to understand these techniques .. now i must replace the fixed message (sent by button click) by output of the chess engine, which should probably run in another thread .. it would be great if you could also compile the current Isofruit script and give some feedback .. these days i will study this tech and then post an update .. i did GUI programming before, and rather extensively, but only with TKinter and Python3, which is well documented and complete .. threads are new to me, they can be "locked" by an event, as i understand correctly .. while creating the Nim script as i imagine it, many new terms arrive .. i'm delighted some programmers are willing to help .. running an external process in the GUI and managing its in/output must be rather common, but it was not easy for me to find a solution / examples .. i think my final script can be a good example for your book ? - i will certainly study the Chapters you mentioned
maybe I could add your board to my salewski-chess
i have some ideas to create a chess GUI .. presenting a board with pieces should not be too difficult indeed .. first i want to tackle the big bottleneck we're discussing : threads in Nim and how to interact with the binary : set another chess position, take back one move, let it think for x seconds or upto a certain "depth", etc.
the salewski-chess engine is GTK4 based, but did you create it with Owlkettle ? Could i have known ? I think the piece font could be better .. i know a lot of nice chess piece fonts exist, i used them, this will not be a problem.
ahhh .. Gintro
learning Rust seems a good idea .. many modern chess engines are built with it .. it also has complete modules with functions to create chess programs .. so, lots of example code for me !? I looked into Julia, because it should run fast, but i discovered not much faster than using PYPY with Python scripts .. and Julia can not be compiled, so to run it the user should have Julia installed .. also GO seems good, in many aspects, but it's all connected to Google, which i try to avoid, just like Microsoft stuff like .NET .. ahwell, i'm a Linux adapt ..
but did you create it with Owlkettle ?
I never really looked into Owlkettle, and think I newer will. I admire all the energy and work Can Lehman put into it. But all what is possible with Owlkettle, was possible with Gintro for years. It is just that Gintro uses the classical syntax, while Owlkettle rewrote it to a declarative notation. So one more layer to learn. IsoFruit, alias Phil, alias Philip M. Doehner, is someone who recently came to Owlkettle, and he seems to drive it development. But I think he already discovered that GTK is not that easy -- he is the one posting all the questions to StackOverflow.
For your mention of Julia and Go: Both interesting languages, but both use a GC and are no true systems programming languages. Rust has some restrictions: No inheritance, no method name overload, no exceptions, no subrange types, no default values for function parameters. Each function parameter needs a type annotation, so max(a, b: int) has to be written max(a: int, b: int). And global variables are nearly unsupported. And of course curly brackets and semicolons everywhere. Coming from Nim, it is hard to accept all these restrictions, but I guess I will manage it.
- this is not about your engine, but programming Nim in general -
i already wrote some words about it in #2 : how to create code in Nim which runs an external binary (on Linux), esp. a UCI chess engine (like SF), and feed UCI command strings and catch its output ? I'm stuck on this, can you help or point me to a working example ?
see also my recent Issue https://github.com/mdoege/nimTUROCHAMP/issues/4 and its links.
i guess threads are involved, but this subject is difficult for me .. i found some Nim modules / code for "async" and so, but i have no clue ..