Closed Veraellyunjie closed 2 years ago
patrickbkr
commented
2 years ago @Veraellyunjie How much RAM does the computer have you are building on? The Rakudo CORE.c.setting compilation (the part where it failed) does require quite some RAM (>1 GB).
Veraellyunjie
commented
2 years ago @patrickbkr Have 16 GB RAM
patrickbkr
commented
2 years ago Hm. The error points quite clearly at an out of memory situation. Can you determine at what memory use the build roughly errors? (e.g. by looking at top/htop/...) Do you know whether OpenBSD has any mechanisms to limit the resources a process can use and whether any such limit is in effect?
Veraellyunjie
commented
2 years ago Success. Your @patrickbkr cue made me look in the right direction.
Can you determine at what memory use the build roughly errors? (e.g. by looking at top/htop/...)
I recall it was 1 GB back then.
Now, for measuring memory consumtion of a successful build I tried:
/usr/bin/time -l rakubrew build # -l - The contents of the rusage structure are printed.
time: rakubrew: No such file or directory/usr/bin/time -l /opt/raku/rakubrew/bin/rakubrew build
The shell hook required to run rakubrew in either 'env' mode or with the 'shell' command seems not to be installed.
Run 'rakubrew init' for installation instructions if you want to use those features,
or run 'rakubrew mode shim' to use 'shim' mode which doesn't require a shell hook.Is there a way to run rakubrew via a wrapper programm like /usr/bin/time or torsocks?
while :; do ps -x -o rss,maxrss,vsz,command >> /tmp/rakubrew_build.ps.log; sleep 1; done
grep moar /tmp/rakubrew_build.ps.log | sort -h -k 1
grep moar /tmp/rakubrew_build.ps.log | sort -h -k 2
grep moar /tmp/rakubrew_build.ps.log | sort -h -k 3
Maximum in KB:
1803816 for resident set size; 1803816 for max resident set size (dunno the difference); 1934780 for virtual size.
Do you know whether OpenBSD has any mechanisms to limit the resources a process can use and whether any such limit is in effect?
datasize in /etc/login.conf and /etc/login.conf.d/${class}
MoarVM seems to honor datasize-cur soft limit - even when datasize-max hard limit is also set and is higher, MoarVM still stops at datasize-cur.
Here are defaults for OpenBSD 7.1 amd64: Without having added yourself to a class:
default:\
...
:datasize-cur=1024M:\
...root is like daemon:
daemon:\
...
:datasize=4096M:\
...It is common to add a user to staff:
staff:\
:datasize-cur=1536M:\
...Note raised limit for compiling ports:
pbuild:\
...
:datasize-cur=8192M:\
...So, I raised my user's class' datasize-cur to 8192m first, 2022.06 release did compile and measurements showed almost 2G.
I lowered datasize-cur to 2048m and 2022.04 release did compile. Similar memory usage.
If you are on OpenBSD and experience similar problem,
As regular user: ulimit -S -d, ulimit -H -d. If -Hard limit is higher and seems sufficient, raise your -Soft limit: ulimit -S -d $(ulimit -H -d).
As root:
usermod -L <login class> <username> - make the user belong to a class
edit /etc/login.conf.d/${class} or /etc/login.conf appropriately
Print current limit: su -c <login class>, ulimit -d
Memory exhaustion check:
while :; do ps auxw | { sleep 0.01; grep loop ;} ; sleep 0.1; donesomehow, ps doesn't show rakudo command name at the beginning of the line - it starts with arguments and at the end there is (rakudo) in parentheses, which may not be shown due to line length, so grep loop
raku -e 'my @a.[0] = 0; loop { @a.push: @a[^@a.elems/5].Str }'Better one-liners to test memory exhaustion are welcome.
The internet shows similar issues to be rather frequent, due to OpenBSD policy of tighter limits.
So if rakubrew team find it appropriate, I'd suggest adding to rakubrew webpage and built-in help:
OpenBSD note:
For building Raku from source you may need to bump up `datasize` system limit to `2048M` or higher.
As regular user, try: `ulimit -S -d $(ulimit -H -d)` - consult `man 1 ksh` http://man.openbsd.org/ksh#ulimit
As root: `:datasize-cur=2048M:` in `login.conf`, then relogin - consult `man 5 login.conf` http://man.openbsd.org/login.conf#CAPABILITIES
and maybe a check in code:
if ($^O eq 'openbsd') {
require BSD::Resource;
BSD::Resource->import;
my ($soft, $hard) = getrlimit("RLIMIT_DATA");
if ($soft < 2097152) {
my $warning = <<~"END";
system limit of $soft on datasize is likely to be insufficient for building Raku.
2048M was reported to be required for `moar-2022.06` on OpenBSD 7.1 amd64.
If build fails due to lack of memory, raise system limit on the size of the data area:
as regular user, try: `ulimit -S -d $(ulimit -H -d)` - consult `man 1 ksh`
as root: `:datasize-cur=2048M:` in `login.conf`, then relogin - consult `man 5 login.conf`
END
warn $warning;
}
}Thanks
OpenBSD 7.1 amd64 Tried 3 times with
rakubrew build moar master,rakubrew build moar-bleadandrakubrew build:MoarVM panic: Memory allocation failed; could not allocate 36864 bytesMoarVM panic: Memory allocation failed; could not allocate 660952 bytesMoarVM panic: Memory allocation failed; could not allocate 660152 bytesI don't know how to troubleshoot it. I understand it is barely rakubrew issue, but rather OpenBSD compatibility or something, but I don't know where to start.
Raku is packaged (in 3 parts) for OpenBSD, and it does work, so MoarVM is buildable: https://openports.se/lang/rakudo https://openports.se/lang/nqp https://openports.se/lang/moarvm
rakubrew rakubrew-version?:rakubrew v29 Build type: fatpack OS: openbsdenvor inshimmode? (runrakubrew modeto find out):envrakubrew hometo find out):/opt/raku/rakubrew