Can't load engine binary on Linux (FileNotFoundError)

[Witek902]

Hi, I wanted to move my working bot from Windows PC to Linux machine (Raspberry Pi 4), but the bot is failing to load the engine. I'm sure that the file exists because I'm able to run the engine manually. I'm using the same config.yml file as on Windows (only had to remove .exe extension).

Here's full console output:

Lichess-bot version: 2023.5.2.1

[MarkZH]

Can you run your bot again and save the log to a file? Posting that will get us more information: python lichess-bot.py -v -l log.txt

Also, post the output of ls -l engines/.

[Witek902]

ls -l engines/ output:

(venv) ubuntu@ubuntu:~/lichess-bot$ ls -l engines/
total 248872
-rw-rw-r-- 1 ubuntu ubuntu 242975040 May  4 10:28 M11.2.bin
-rw-rw-r-- 1 ubuntu ubuntu        40 May  3 23:04 README.md
-rwxrwxr-x 1 ubuntu ubuntu    415208 May  3 23:04 caissa
-rw-rw-r-- 1 ubuntu ubuntu   2190912 May  3 23:17 eval-12.pnn
-rw-rw-r-- 1 ubuntu ubuntu   9250016 May  4 10:27 komodo.bin

Log file: log.txt

[MarkZH]

Let's check that the python-chess library works. Put the following script (adapted from here) in the lichess-bot folder and see if it runs without error.

import chess
import chess.engine

engine = chess.engine.SimpleEngine.popen_uci(["engines/caissa"])

board = chess.Board()
print(board.fen())
while not board.is_game_over():
    result = engine.play(board, chess.engine.Limit(time=0.1))
    board.push(result.move)
print(board.fen())

engine.quit()

The print statements are there to make sure a game was played.

[Witek902]

Seems to work:

ubuntu@ubuntu:~/lichess-bot$ source ./venv/bin/activate
(venv) ubuntu@ubuntu:~/lichess-bot$ python3 test.py
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1
k7/8/Q3K3/4B3/p7/P7/6P1/8 b - - 8 87
(venv) ubuntu@ubuntu:~/lichess-bot$

[MarkZH]

(This is getting weirder ...)

Try adding ./ to the beginning of the dir and working_dir entries in the engine section of your config file. The warning in the popen docs seems relevant.

engine:
  dir: ./engines/
  name: caissa
  working_dir: ./engines/

[MarkZH]

I was able to reproduce the error on Ubuntu. The problem is described in the Python docs link in my previous comment, namely:

Resolving the path of executable (or the first item of args) is platform dependent. For POSIX, see os.execvpe(), and note that when resolving or searching for the executable path, cwd overrides the current working directory and env can override the PATH environment variable.

In other words, since OP's config file has working_dir: engines/, lichess-bot effectively tries (through the python-chess library and ultimately through popen) to open the engine at ./engines/engines/caissa, which doesn't exist.

The quickest fix would be to explicitly call the absolute path to the engine executable on line 43 of engine_wrapper.py:

engine_path = os.path.abspath(os.path.join(cfg.dir, cfg.name))

However, there are still be problems if the engine wants to read files and needs to know the location of those files. If the paths in the user's config file were relative to the current working directory where lichess-bot was started, then having a working_dir config entry will cause these relative paths to be wrong. I just tested this with stockfish and the SyzygyPath option. I placed the syzygy files in the directory engines/3. Setting SyzygyPath: engines/3 in the uci_options section with no working_dir caused no problems and stockfish found the files. However, setting working_dir: engines/ causes stockfish to be unable to find the syzygy files.

@AttackingOrDefending I can think of two changes to address this:

1. Use the os.path.abspath() fix above.
2. Warn users that if working_dir is different from the current working directory then all paths (engine and engine-associated files) require absolute paths. I think this warning is only needed for linux and mac systems.

Do you agree? Can you think of any others? I don't think trying to fix all paths in in the user's config files is feasible since it would be impossible to figure out which entries contained paths.

[MarkZH]

@Witek902 While we are coming up with a solution for this, you can use one or both of these workarounds to get your bot working on lichess:

1. Use an absolute path in engine: dir:
2. Delete engine: working_dir:

[AttackingOrDefending]

I prefer the first option. The cwd is for the engine, so that the engine knows where to read/write files from, while dir and engine is for lichess-bot. I think that we should keep dir and engine independent from working_dir and engine options.

On your test, if you set SyzygyPath: 3, doesn't it work?

I think we should use abspath and add a warning in the README.md and config.yml files, warning the user that engine options must be relative to working_dir.