geaaru
commented
5 years ago As reported in bugzilla if I force -j1 this doesn't happen.
Closed geaaru closed 5 years ago
geaaru
commented
5 years ago As reported in bugzilla if I force -j1 this doesn't happen.
jpereira
commented
5 years ago @geaaru I am pretty sure that code isn't the same as can be found in our repo.
e.g:
Latest revision in https://github.com/FreeRADIUS/freeradius-server/tree/release_3_0_19
[root@docker-freeradius-ubuntu16 freeradius-server-v3.0.x.git]# git log -1
commit ab4c767099f263a7cd4109bcdca80ee74210a769
Author: Matthew Newton <matthew-git@newtoncomputing.co.uk>
Date: Wed Apr 10 10:11:23 2019 +0100
fix incorrectly named variable
[root@docker-freeradius-ubuntu16 freeradius-server-v3.0.x.git]# Using the same command as you showed.
[root@docker-freeradius-ubuntu16 freeradius-server-v3.0.x.git]# (./configure && make -j9 && make -j9 Q= ECHO=true LOCAL_CERT_PRODUCTS= R=/tmp/test/image install) 1> /dev/null 2>&1
[root@docker-freeradius-ubuntu16 freeradius-server-v3.0.x.git]# echo $?
0
[root@docker-freeradius-ubuntu16 freeradius-server-v3.0.x.git]#Conclusion: Probably is NOT the same codebase.
ps: My make/gcc versions.
[root@docker-freeradius-ubuntu16 freeradius-server-v3.0.x.git]# make --version | head -1
GNU Make 4.1
[root@docker-freeradius-ubuntu16 freeradius-server-v3.0.x.git]# gcc --version | head -1
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609
[root@docker-freeradius-ubuntu16 freeradius-server-v3.0.x.git]#@jpereira the sources are the same (3.0.19 is not yet in gentoo official portage but i will push it soon):
This is directly from Gentoo portage: https://github.com/gentoo/gentoo/blob/master/net-dialup/freeradius/freeradius-3.0.18.ebuild#L12
About gcc and make ... it's not so easy, different users could have a different version of GCC and make. I only can say that currently in my machine I have this:
# make --version
GNU Make 4.2.1
Built for x86_64-pc-linux-gnu
# gcc --version
gcc (Gentoo Hardened 7.3.0-r3 p1.4) 7.3.0
Copyright (C) 2017 Free Software Foundation, Inc.But I'm sure that this is a problem that happened in the past also with previous versions. The main problem is that is not always reproducible. I will try to investigate this, but for now it fine that there is a trace of this Gentoo users issue also here.
jpereira
commented
5 years ago @geaaru Okay, remember that in fact Gentoo always calls the wrapper emake instead of make. so, it could be a clue to figure out what is going on there. either way, good luck.
geaaru
commented
5 years ago Normally, emake call make as you can see at the begin of the attached log:
make -j9 Q= ECHO=true LOCAL_CERT_PRODUCTS= R=/var/tmp/portage/net-dialup/freeradius-3.0.19-r1/image install However, i will investigate this.
arr2036
commented
5 years ago Yes, please, @geaaru if you can figure out what's going on then it would definitely be a benefit across all platforms. It'd also help you get familiar with boilermake. Maybe when @alandekok comes back from vacation in a couple of weeks he could look at it too. But I don't think anyone really has much time to spare for this currently.
alandekok
commented
5 years ago The issue is likely that each “install” rule tried to do “mkdir”. Instead, each rule that requires a directory should depend on a rule which makes that directory
But the issue could also be in “make”
On Jun 6, 2019, at 2:57 AM, Arran Cudbard-Bell notifications@github.com wrote:
Yes, please, @geaaru if you can figure out what's going on then it would definitely be a benefit across all platforms. It'd also help you get familiar with boilermake. Maybe when @alandekok comes back from vacation in a couple of weeks he could look at it too. But I don't think anyone really has much time to spare for this currently.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.
geaaru
commented
5 years ago mmm, I'm not sure that it's strictly related to make but how Makefiles are been done in FreeRadius. I dunno if with a "pure" autotools tree (with Makefile.am, etc.), Makefiles created by automake tool maybe tries to create all directories before executing the final install commands.
Maybe a solution could be that the creation of all directories could be done before execute install commands.
I will do some tests. However, very low priority.
geaaru
commented
5 years ago I think that depends on the number of cores available.
Just, I can reproduce it that follow jpereira example:
$# for ((i=0;i<50;i++)); do make clean ; rm -rf /tmp/radius || true ; (./configure --prefix=/tmp/radius && make -j9 && make -j9 Q= ECHO=true R=/tmp/radius LOCAL_CERT_PRODUCTS= install) 1> /dev/null 2>&1 ; echo "RESULT $?" ; done
RESULT 2
RESULT 2
RESULT 0
RESULT 2
RESULT 0
RESULT 2
RESULT 2
RESULT 0
RESULT 2
RESULT 2
RESULT 2
RESULT 2
RESULT 2
RESULT 2
RESULT 2
RESULT 2
RESULT 0
RESULT 2
RESULT 2
RESULT 2
RESULT 2
RESULT 2
RESULT 2
RESULT 2
RESULT 2
RESULT 2
RESULT 2
RESULT 2
RESULT 0
RESULT 2
RESULT 2
RESULT 0
RESULT 2
RESULT 2
RESULT 2
RESULT 0
RESULT 2
RESULT 2
RESULT 2
RESULT 0
RESULT 2
RESULT 2
RESULT 0
RESULT 0
RESULT 2
RESULT 2
RESULT 2
RESULT 2
RESULT 2
RESULT 2What is normally done for a directory is something like this:
.PHONY: dir
dir:
@mkdir -p $@
dir/foo: foo | dirThis lets GNU make know that it should create dir before running the dir/foo rule. This works well in most places.
However, our Makefile rules create directories from "wildcard" rules. Which means that there are no rules specific to one directory.
I've added some rules which may help.
alandekok
commented
5 years ago Did the recent changes help?
geaaru
commented
5 years ago I will test soon and I will report feedback. Thanks
geaaru
commented
5 years ago Not yet at least in my env.
$# for ((i=0;i<50;i++)); do make clean ; rm -rf /tmp/radius || true ; (./configure --prefix=/tmp/radius && make -j9 && make -j9 Q= ECHO=true R=/tmp/radius LOCAL_CERT_PRODUCTS= install) 1> /dev/null 2>&1 ; echo "RESULT $?" ; done
RESULT 2
RESULT 0
RESULT 0
RESULT 2
RESULT 2
RESULT 2
RESULT 2
RESULT 2
RESULT 2
RESULT 2
RESULT 2
RESULT 2
RESULT 2
$# cat doc/all.mk | grep 'dir $@'
$(R)$(docdir)/%: doc/% | $(dir $@)
$# bc334d1b3c (HEAD -> v3.0.x, upstream/v3.0.x, origin/v3.0.x) Avoid daemonize radiusd with systemd and drop PIDFile
Just an extract of an error after your patch:
mkdir: cannot create directory ‘/tmp/radius/tmp/radius/share/doc/freeradius/schemas’: File exists
mkdir: cannot create directory ‘/tmp/radius/tmp/radius/share/doc/freeradius/schemas/ldap’: File exists
mkdir: cannot create directory ‘/tmp/radius/tmp/radius/share/doc/freeradius/schemas/ldap’: File exists
make: *** [/home/geaaru/Projects/freeradius-server/doc/all.mk:13: /tmp/radius/tmp/radius/share/doc/freeradius/schemas] Error 1
make: *** Waiting for unfinished jobs....
make: *** [/home/geaaru/Projects/freeradius-server/doc/all.mk:13: /tmp/radius/tmp/radius/share/doc/freeradius/schemas/ldap] Error 1Hmm... this looks like it's going to be difficult to fix, then.
geaaru
commented
5 years ago Yeah, not easy to fix. It's needed to see make documentation if there is a way to limit parallelization in some conditions.
I saw a lot of projects move away from autotools for problems of performance and parallelization.
Have you ever think to integrate meson, ninja and or other tools?
alandekok
commented
5 years ago @geaaru We've looked at other tools. They're almost all worse. "ninja" is a simple rule parser, and still needs something to generate the rules. Meson requires python, which means learning that. Cmake is just terrible by all measurements.
The system we're using is in scripts/boiler.make. It's pretty horrific to create the gnu makefile magic, but the resulting all.mk files are pretty trivial. Which is what we need.
I've pushed fixes to the master branch which mostly fix this, along with comments as to what to do next. I think for now we can leave v3 alone.
From v. 3.0.2 as reported in Gentoo Bug tracking system a lot of users receive errors on parallel installation.
It seems that this still is present also in the last v.3.0.19 and after migration to jlibtool.
In my case it happens with -j9: