add an API to the standard library to learn the system page size

gpanders commented 2 years ago

Zig Version

0.10.0-dev.1434+676652865

Steps to Reproduce

I am running Linux on a MacBook Pro which uses 16 KiB pages; however, Zig's std lib always assumes 4 KiB pages on Linux running on aarch64.

Expected Behavior

Zig should infer/detect the correct page size.

Actual Behavior

Zig assumes 4 KiB pages, which results in errors in calls to msync or mmap (e.g. in std.debug.captureStackTrace).

https://github.com/ziglang/zig/issues/4082 is mentioned in the source code, so I'm guessing that is the long term solution. That issue also mentions a "hacky" solution of using madvise to check for a valid page size. A short term solution could be to check madvise at compile time when running Linux on aarch64 to check between all possible page sizes (ARM64 can also support 64 KiB pages). ~~If this is acceptable I can open a PR.~~ EDIT: It looks like syscalls can't run at comptime so I don't think this would work.

vielmetti commented 2 years ago

I've added this to the @AsahiLinux list of code that is known to have problems with a 16K page size, at

https://github.com/AsahiLinux/docs/wiki/Software-known-to-have-issues-with-16k-page-size

As a reference point, this particular issue was addressed in Go (golang) at https://github.com/golang/go/issues/10180 by determining system page size at runtime. The side effect is to improve compatibility on a variety of platforms with non-4K page sizes.

dkwo commented 1 year ago

@Vexu Why does this issue have an os-macos label, while it is specific of linux? also, any chance it could be assigned to an earlier milestone?

Vexu commented 1 year ago

I can put it on 0.11.0 but it doesn't mean much since it could be resolved earlier or later anyways.

dkwo commented 1 year ago

Would it be feasible to adopt for linux whatever solution zig already adopts for macos?

MichaByte commented 1 year ago

@dkwo According to The Asahi Linux docs, macOS has its own way of letting programs "choose" their own page size, so I'm guessing Zig just uses 4K pages and lets macOS deal with it. Linux doesn't have its own way of working with varied page sizes, and "likely never will" (quote from the Asahi docs)

So to answer your question, Zig doesn't need to worry about page sizes on macOS, so that's not really a solution that can be transferred to Linux.

P.S. If I got something horribly wrong, feel free to correct me ;)! I'm not an expert, this is just my understanding.

dkwo commented 1 year ago

Makes sense, thanks.

jart commented 1 year ago

Linux doesn't have its own way of working with varied page sizes

The kernel passes AT_PAGESZ in the auxiliary values at program startup. The C library should make this value available via sysconf(_SC_PAGESIZE).

andrewrk commented 1 year ago

Duplicate of #4082. std.mem.page_size is the minimum page size and is comptime-known. The actual page size can only be determined at runtime.

I am running Linux on a MacBook Pro which uses 16 KiB pages; however, Zig's std lib always assumes 4 KiB pages on Linux running on aarch64.

I am unable to reproduce your steps:

andy@ark ~> Zigs std lib always assumes 4 KiB pages on Linux running on aarch64.
Zigs: command not found
andy@ark ~ [127]>

Jokes aside, I'll change this issue into something that can actually address what I am guessing is your use case.

par5er commented 1 year ago

What is the status of this issue? I can take a shot at it if someone provides instructions

dkwo commented 1 year ago

still a bug, as far as i know

par5er commented 1 year ago

In case it helps anyone, I have made a temporary fix with binary download here: https://github.com/par5er/zig/releases

matu3ba commented 11 months ago

Windows has fixed ones, although supports smaller and larger one https://devblogs.microsoft.com/oldnewthing/20210510-00/?p=105200
- SYSTEM_INFO.dwPageSize from GetSystemInfo or the underlying ntdll call
- I dont think the page size info can be derived from the alignment in https://learn.microsoft.com/en-us/windows/win32/debug/pe-format, because it does not specify "The default is the page size for the architecture." in the object format, see "Optional Header Windows-Specific Fields (Image Only)"

Linux offers (unstable) /proc introspection ~~to save the syscall~~ https://serverfault.com/a/328090

/proc/pid/statm: Provides information about memory usage, measured in pages.
/proc/pid/status: Provides much of the information from /proc/pid/statm, but is easier to read (MMUPageSize).

getpagesize is the supposed libc command to be used instead of sysconf(_SC_PAGESIZE), if linked with libc. Additional context (nice read): https://lwn.net/Articles/631631/ and https://lwn.net/Articles/519085/
~~If Zig controls the runtime linker, then Zig could do the same as the C lib, but also provide it within the program (sounds potentially cursed and complex though).~~
From quick glimpse, Mach-O neither provides the system page size inside the object format (also only alignments).

https://en.wikipedia.org/wiki/Page_(computer_memory)#Getting_page_size_programmatically sounds like the most pragmatic approach, so YAGNI:

#include <stdio.h>
#include <unistd.h> /* sysconf(3) */
int main(void)
{
    printf("The page size for this system is %ld bytes.\n",
        sysconf(_SC_PAGESIZE)); /* _SC_PAGE_SIZE is OK too. */

    return 0;
}

#include <stdio.h>
#include <windows.h>
int main(void)
{
    SYSTEM_INFO si;
    GetSystemInfo(&si);
    printf("The page size for this system is %u bytes.\n", si.dwPageSize);
    return 0;
}

dkwo commented 11 months ago

@andrewrk Why was the bug label removed? if I cannot use zig to build e.g. river on aarch64 linux on my 16k page size kernel, this looks like a bug to me, not enhancement.

andrewrk commented 11 months ago

@andrewrk Why was the bug label removed? if I cannot use zig to build e.g. river on aarch64 linux on my 16k page size kernel, this looks like a bug to me, not enhancement.

If you want the bug label you need to fill out the bug template. This issue does not provide steps to reproduce, observed behavior, or actual behavior.

Perhaps you are interested in #16331 which is labeled as a bug.

dkwo commented 11 months ago

I see. Would the following be acceptable?

I'm using Void Linux on an Apple m1 machine (aarch64-glibc), which has a 16k page-size kernel, but any Linux with these requirements https://github.com/void-linux/void-packages#requirements and a non-standard (but perfectly fine) page-size kernel should be able to reproduce.

Clone the repo

git clone https://github.com/void-linux/void-packages.git
cd void-packages
./xbps-src binary-bootstrap

Then build zig https://github.com/void-linux/void-packages/blob/master/srcpkgs/zig/template (version 0.10.1)

./xbps-src pkg zig

Then build (e.g.) river https://github.com/void-linux/void-packages/blob/master/srcpkgs/river/template (version 0.2.4)

./xbps-src pkg river

The build process fails with

thread 2820 panic: reached unreachable code 
Panicked during a panic. Aborting.

More details:

=> xbps-src: updating repositories for host (aarch64)... 
[*] Updating repository `https://repo-default.voidlinux.org/current/aarch64/bootstrap/aarch64-repodata' ...
[*] Updating repository `https://repo-default.voidlinux.org/current/aarch64/aarch64-repodata' ...
[*] Updating repository `https://repo-default.voidlinux.org/current/aarch64/nonfree/aarch64-repodata' ...
[*] Updating repository `https://repo-default.voidlinux.org/current/aarch64/debug/aarch64-repodata' ...
=> xbps-src: updating software in / masterdir... 
=> xbps-src: cleaning up / masterdir... 
=> river-0.2.4_2: removing autodeps, please wait... 
=> river-0.2.4_2: the following build options are set: 
   xwayland: Enable Xwayland support (ON) 
=> river-0.2.4_2: building with [zig-build] for aarch64... 
   [host] pkg-config-0.29.2_3: found (https://repo-default.voidlinux.org/current/aarch64)
   [host] wayland-devel-1.22.0_1: found (https://repo-default.voidlinux.org/current/aarch64)
   [host] scdoc-1.11.2_1: found (https://repo-default.voidlinux.org/current/aarch64)
   [host] zig-0.10.1_2: found (/host/binpkgs) 
   [target] wlroots-devel-0.16.2_1: found (https://repo-default.voidlinux.org/current/aarch64)
   [target] libevdev-devel-1.13.0_1: found (https://repo-default.voidlinux.org/current/aarch64)
   [target] pixman-devel-0.42.2_1: found (https://repo-default.voidlinux.org/current/aarch64)
   [target] wayland-protocols-1.32_1: found (https://repo-default.voidlinux.org/current/aarch64)
   [target] libxkbcommon-devel-1.5.0_1: found (https://repo-default.voidlinux.org/current/aarch64)
   [target] wayland-devel-1.22.0_1: found (https://repo-default.voidlinux.org/current/aarch64)
   [runtime] xorg-server-xwayland-23.1.2_1: found (https://repo-default.voidlinux.org/current/aarch64)
=> river-0.2.4_2: installing host dependencies: pkg-config-0.29.2_3 wayland-devel-1.22.0_1 scdoc-1.11.2_1 zig-0.10.1_2 ...
=> river-0.2.4_2: installing target dependencies: wlroots-devel-0.16.2_1 libevdev-devel-1.13.0_1 pixman-devel-0.42.2_1 wayland-protocols-1.32_1 libxkbcommon-devel-1.5.0_1 wayland-devel-1.22.0_1 ...
=> river-0.2.4_2: running do-fetch hook: 00-distfiles ... 
=> river-0.2.4_2: running do-extract hook: 00-distfiles ... 
=> river-0.2.4_2: extracting distfile(s), please wait... 
=> river-0.2.4_2: running do-patch hook: 00-patches ... 
=> river-0.2.4_2: running pre-configure hook: 00-gnu-configure-asneeded ...
=> river-0.2.4_2: running pre-configure hook: 01-override-config ...
=> river-0.2.4_2: running pre-configure hook: 02-script-wrapper ...
=> river-0.2.4_2: running pre-build hook: 02-script-wrapper ... 
=> river-0.2.4_2: running do_build ... 
thread 2820 panic: reached unreachable code 
Panicked during a panic. Aborting. 
error: the following build command crashed: 
/builddir/river-0.2.4/zig-cache/o/fd98d92d81de57486a870485e6d8aa9e/build /usr/bin/zig /builddir/river-0.2.4 /builddir/river-0.2.4/zig-cache /tmp/.cache/zig --sysroot  --libc xbps_zig_libc.txt -Dtarget=aarch64-linux-gnu -Dcpu=baseline -Drelease-safe --prefix /usr install -Dxwayland -Dpie
=> ERROR: river-0.2.4_2: do_build: 'DESTDIR="zig-out" zig build --sysroot "${XBPS_CROSS_BASE}" --libc xbps_zig_libc.txt -Dtarget="${zig_target}" -Dcpu="${zig_cpu}" -Drelease-safe --prefix /usr install ${configure_args}' exited with 1
=> ERROR:   in do_build() at common/build-style/zig-build.sh:31

andrewrk commented 11 months ago

Yes that looks like a proper bug report. It also looks like a duplicate of #16331. I suggest subscribing to #16331.

dkwo commented 11 months ago

Rather, that bug looks like a duplicate of this one :) since that was opened 3 weeks ago (vs 1+ year), and this bug is also tracked by asahi-linux, which is kind of useful.

andrewrk commented 11 months ago

If you have time to argue with me on the issue tracker, you have time to submit a patch :P

monomycelium commented 9 months ago

In case it helps anyone, I have made a temporary fix with binary download here: https://github.com/par5er/zig/releases

I have applied the same patch to Zig 0.11.0 with a binary download at https://marsh.digitya.com/cache/zig-linux-aarch64-0.11.0.tar.xz if it helps anyone using Asahi Linux on an Apple M1 chip (use only if you are sure that this is not malicious). I compiled it using ReleaseFast, so the binary is quite large (around 130 MB). Again, it's a temporary fix; the software compiled using this would use 16k page sizes on Linux (unless you edit mem.zig?). Good luck!

Update

I have compiled Zig 0.12.0 with the temporary patch (with -mcpu=baseline) with a download at https://fs.ilmul.com/zig-linux-aarch64-0.12.0.tar.xz.

RossComputerGuy commented 9 months ago

I've managed to add a function for checking the page size at runtime in #17382. It seems like this issue somewhat is related to my PR. It supports setting the page size for a target so cross compiling would work. The only issue atm is it works best when your compiling on the device your using it for. This means prebuilt Zig versions won't work on Apple Silicon without a specific build. But I hope to fix that somewhat soon by replacing some of the std.mem.page_size usage with the runtime function.

vielmetti commented 7 months ago

It is worthwhile to note that the new Pi 5 also has 16K pages by default, so it's likely that code running on it will be affected by this issue.

https://www.raspberrypi.com/documentation/computers/config_txt.html#kernel is the Pi 5 kernel config doc that's relevant.

RossComputerGuy commented 7 months ago

@vielmetti Yeah, I won't be able to get a Pi 5 to test out #17382 on it but since I have an Apple Silicon device which the PR works for, this should work. Though Windows on ARM may not support the Pi 5, the code should work as soon as I add the Windows code today. But Linux does work so that should mean it will work.

I'm working on adding CPU-specific cross-compile to change the page size so if you specify the Pi CPU and Zig detects it then it should be able to use 16k pages. Though, the functionality might break programs if a non-16k page sized kernel is used. Work to detect this better could be done.

RoundDuckKira commented 5 months ago

I see. Would the following be acceptable?

I'm using Void Linux on an Apple m1 machine (aarch64-glibc), which has a 16k page-size kernel, but any Linux with these requirements https://github.com/void-linux/void-packages#requirements and a non-standard (but perfectly fine) page-size kernel should be able to reproduce.

Clone the repo

git clone https://github.com/void-linux/void-packages.git
cd void-packages
./xbps-src binary-bootstrap

Then build zig https://github.com/void-linux/void-packages/blob/master/srcpkgs/zig/template (version 0.10.1)

./xbps-src pkg zig

Then build (e.g.) river https://github.com/void-linux/void-packages/blob/master/srcpkgs/river/template (version 0.2.4)

./xbps-src pkg river

The build process fails with

thread 2820 panic: reached unreachable code 
Panicked during a panic. Aborting.

More details:

=> xbps-src: updating repositories for host (aarch64)... 
[*] Updating repository `https://repo-default.voidlinux.org/current/aarch64/bootstrap/aarch64-repodata' ...
[*] Updating repository `https://repo-default.voidlinux.org/current/aarch64/aarch64-repodata' ...
[*] Updating repository `https://repo-default.voidlinux.org/current/aarch64/nonfree/aarch64-repodata' ...
[*] Updating repository `https://repo-default.voidlinux.org/current/aarch64/debug/aarch64-repodata' ...
=> xbps-src: updating software in / masterdir... 
=> xbps-src: cleaning up / masterdir... 
=> river-0.2.4_2: removing autodeps, please wait... 
=> river-0.2.4_2: the following build options are set: 
   xwayland: Enable Xwayland support (ON) 
=> river-0.2.4_2: building with [zig-build] for aarch64... 
   [host] pkg-config-0.29.2_3: found (https://repo-default.voidlinux.org/current/aarch64)
   [host] wayland-devel-1.22.0_1: found (https://repo-default.voidlinux.org/current/aarch64)
   [host] scdoc-1.11.2_1: found (https://repo-default.voidlinux.org/current/aarch64)
   [host] zig-0.10.1_2: found (/host/binpkgs) 
   [target] wlroots-devel-0.16.2_1: found (https://repo-default.voidlinux.org/current/aarch64)
   [target] libevdev-devel-1.13.0_1: found (https://repo-default.voidlinux.org/current/aarch64)
   [target] pixman-devel-0.42.2_1: found (https://repo-default.voidlinux.org/current/aarch64)
   [target] wayland-protocols-1.32_1: found (https://repo-default.voidlinux.org/current/aarch64)
   [target] libxkbcommon-devel-1.5.0_1: found (https://repo-default.voidlinux.org/current/aarch64)
   [target] wayland-devel-1.22.0_1: found (https://repo-default.voidlinux.org/current/aarch64)
   [runtime] xorg-server-xwayland-23.1.2_1: found (https://repo-default.voidlinux.org/current/aarch64)
=> river-0.2.4_2: installing host dependencies: pkg-config-0.29.2_3 wayland-devel-1.22.0_1 scdoc-1.11.2_1 zig-0.10.1_2 ...
=> river-0.2.4_2: installing target dependencies: wlroots-devel-0.16.2_1 libevdev-devel-1.13.0_1 pixman-devel-0.42.2_1 wayland-protocols-1.32_1 libxkbcommon-devel-1.5.0_1 wayland-devel-1.22.0_1 ...
=> river-0.2.4_2: running do-fetch hook: 00-distfiles ... 
=> river-0.2.4_2: running do-extract hook: 00-distfiles ... 
=> river-0.2.4_2: extracting distfile(s), please wait... 
=> river-0.2.4_2: running do-patch hook: 00-patches ... 
=> river-0.2.4_2: running pre-configure hook: 00-gnu-configure-asneeded ...
=> river-0.2.4_2: running pre-configure hook: 01-override-config ...
=> river-0.2.4_2: running pre-configure hook: 02-script-wrapper ...
=> river-0.2.4_2: running pre-build hook: 02-script-wrapper ... 
=> river-0.2.4_2: running do_build ... 
thread 2820 panic: reached unreachable code 
Panicked during a panic. Aborting. 
error: the following build command crashed: 
/builddir/river-0.2.4/zig-cache/o/fd98d92d81de57486a870485e6d8aa9e/build /usr/bin/zig /builddir/river-0.2.4 /builddir/river-0.2.4/zig-cache /tmp/.cache/zig --sysroot  --libc xbps_zig_libc.txt -Dtarget=aarch64-linux-gnu -Dcpu=baseline -Drelease-safe --prefix /usr install -Dxwayland -Dpie
=> ERROR: river-0.2.4_2: do_build: 'DESTDIR="zig-out" zig build --sysroot "${XBPS_CROSS_BASE}" --libc xbps_zig_libc.txt -Dtarget="${zig_target}" -Dcpu="${zig_cpu}" -Drelease-safe --prefix /usr install ${configure_args}' exited with 1
=> ERROR:   in do_build() at common/build-style/zig-build.sh:31

wait, sorry that it isn't the most related to this discussion, but how are you running void, natively or in a VM?

dkwo commented 5 months ago

@RoundDuckKira native, as you figured out :)

ziglang / zig