Remove memory mapping of DTZ files.

[mbabigian]

If you assume one engine is running, memory mapping DTZs creates no problems; however, every serious chess player uses multiple instances of an engine simultaneously. Having one instance memory mapping DTZs that prevents WDL files from getting RAM loaded in a second engine instance is a huge performance hit. DTZs should NEVER be memory mapped. Simple OS caching is more than enough for the DTZ files.

[vdbergh]

I think that if you memory map a file, only the parts that are accessed are effectively loaded into physical memory. So memory mapping already functions as a cache.

[vdbergh]

https://en.wikipedia.org/wiki/Memory-mapped_file

[vdbergh]

Also very likely if the same file is memory mapped by multiple processes, the different mappings share their physical memory.

[mbabigian]

You are completely missing the point. Having the files mapped creates enormous problems, especially using 7 piece TBs. Arguing that mapping acts like a cache is a water is wet statement. The point is that mapping buys nothing for typical use cases, but causes extreme memory issues for zero benefit. There is zero reason to try to speed access to DTZs at the expense of RAM that could be used for more productive purposes. I'm running multiple engines using this code and it turns RAM into such a train wreck, I need constant reboots as the system (Windows) can't even recover after the engines are halted. (Converting Leela training data to Stockfish format). The same would be true running say 32 Stockfish's in a match using let say 6 piece TBs. Say I have enough RAM to memory map the WDLs but not the DTZs being probed. Say 8 Stockfish's are in a late stage game mapping DTZs to RAM when the others need to map WDL for use in search. The WDL access times will get clobbered. WDLs need speed not DTZs.

DTZs should NEVER be mapped.

[vdbergh]

You seem to believe that if a file is memory mapped then the whole file is loaded into memory. This is simply not true. Only the pages that are accessed are loaded into (physical) memory. Moreover such pages can be ejected again if they are no longer used. Recall that DTZ files are only accessed at root. In other words very rarely.

[mbabigian]

You seem to believe in the Easter Bunny. Where did I say the whole file is mapped please quote me. You've clearly never used this code as I described above and seem to have a lot of "theory." I have a lot of practice. I have described the problem adequately well, if you don't believe in it - having a religious debate about your "beliefs/theories" is not a good use of my time. I suppose in your beliefs, I "imagine" the problems this causes and imagine that windows will leave my used RAM stuck at nearly 100% for hours after these processes end, and I imagine the immense disk thrashing when other programs need RAM during engine use or after all the engines are closed. I must also imagine my page file being hammered like I have no RAM free just as the OS reports.

Please stop. I know I will, good bye.

[vdbergh]

I think there is something wrong with your setup, or else you are misdiagnosing your issues (note that a modern OS will always try to use 100% of available physical memory).

An exited process will free its allocated memory, so if that doesn't happen then something is seriously wrong. Moreover memory used by a memory mapped file is by definition "file backed memory". So when reclaimed, it will not cause trashing (trashing is caused by data which is swapped out to disc when memory is tight).

[syzygy1]

The DTZ files will only be mapped when they are accessed, and this will only happen at the end of the game when a TB position has been reached. Only a fraction of the DTZ file relevant to the position will end up in memory. The engine could unmap the mapped DTZ files (and even the WDL files) at the beginning of a new game, but I doubt it would make a noticeable difference (as far as DTZ is concerned). I think you can manually force unmapping by Stockfish without restarting the engine by setting the SyzygyPath to something non-existent and then back.

Anyway, this is an engine issue. Not a generator issue.

[syzygy1]

So this can be closed.

[mbabigian]

The main issue I had with this behavior was with the Leela training data converter. It created catastrophic memory problems when rescoring using 7 man TBs. I agree it is an engine issue and should be fixed. After realizing how borked the rescorer was in general, I eventually wrote my own code to perform these functions properly. So although this is still a problem with engine testing/analysis with TBs and high concurrency, I don't see any engine authors taking it seriously. Likely because they don't play chess or use engines for serious analysis. Anyway I agree it is not a generator issue, but a probing issue. I appreciate the suggested workarounds but I don't think they are practical for something like Aquarium unattended IDEA analysis. This type of environment also developed memory issues if run long enough.