C++ | newCache v2

[JATothrim]

Hello!

I have some upgrades to the newCache + project code:

• libmapped_file: memory mapped file I/O library (see first commit) This is my own library in making, partly unrelated to opencubes project. OpenCubes just happens to be first client for the library outside my projects. The library source files have explicit MIT license prepended to them.
• Upgraded CacheReader with libmapped_file
• New CacheWriter that writes out the hashy in parallel
• Dropped old cache code.
• Reduced memory use for Cube struct (was 16-bytes, now 8-bytes)
• Refactored cubes.cpp thread scheduling.

Overall I can do now N=13 in less than 310 seconds give or take. :smile: There still is no any kind of compression of the PolyCube data...

To generate starting cache files run cubes -n 10 -t <threads> -w -s followed by cubes -n 11 -t <threads> -w -s -u to continue with split cache files.

EDIT: I have re-arranged my repository and enabled GPG verified commits. The default branch https://github.com/JATothrim/opencubes is now fork of mikepound/main and includes the changes in this PR. @bertie2 can you take a look on this?

[nsch0e]

A few thoughts:

• Reduced memory use for Cube struct (was 16-bytes, now 8-bytes)

Perhaps a check if the upper bits in pointers are used would prevent segfaults and could warn the user about the problem.

Also the copyout function should check if num<=size().

[JATothrim]

A few thoughts:

• Reduced memory use for Cube struct (was 16-bytes, now 8-bytes)

Perhaps a check if the upper bits in pointers are used would prevent segfaults and could warn the user about the problem.

Did you get segfaults? :eyes: I included the hack so that I could get feedback on it: The address hack is CPU architecture specific. The hack assumes the code is compiled for x86-64 with 48-bit VA and it "worked on my machine" at the time I wrote it. The possibility of packing Cube struct into 8-bytes was and still is too tempting for me.

I read the Intel 5-level paging wiki now and I realized the hack is likely broken in current form on many systems. Virtual addresses must be in canonical form for CPU not to segfault on them. The current hack just zeros the high 8-bits, so I might have been very lucky of being able to run this at all.

I convert this PR into draft because the code is outdated and likely breaks on some systems. I would like to have some discussion here of above hack and how to make it work in general.

Also the copyout function should check if num<=size().

Would an assert() suffice?

[nsch0e]

I did not have any segfaults but thought it would be easy to detect if we are truncating important bits from the pointers. Perhaps an extra function to do the check once is also a possibility.

Assert seems fine. Should be systemic error if num and size are not matching.

[JATothrim]

I did not have any segfaults but thought it would be easy to detect if we are truncating important bits from the pointers. Perhaps an extra function to do the check once is also a possibility.

For Intel x86-64 processors the high 8 or 16-bits of memory address must be replication of (depending if 5-level paging is enabled) 56-bit or 47-bit. Apparently I haven't found any memory address that has 47-bit set. I think this is an possibility on larger systems, so get() requires checking if the 56-bit or 47-bit is set on Intel x86-64.

The check is CPU Arch/system specific and not viable to do on runtime. I'll rebase an commit that I forgot about to my next branch that allows fixing this proper. It enables an configuration header with CMake. If I remember that commit also provides ./cubes -v that would print an git HEAD commit hash, compiler id and debug/release mode that was used to build the binary...