gavinhoward / bc

An implementation of the POSIX bc calculator with GNU extensions and dc, moved away from GitHub. Finished, but well-maintained.
https://git.gavinhoward.com/gavin/bc
Other
145 stars 29 forks source link

test failures on Android #61

Closed enh-google closed 1 year ago

enh-google commented 1 year ago

i haven't been able to update for some time (5.0.1 was our last update!) because the tests fail in a weird way without much useful information. i've included the current failures for 6.2.2 below, for example.

my guess is that this is something to do with Android using mksh as its /bin/sh. the interleaved output makes it appear like there might be some disagreement about job handling between bash and mksh?

are the current bc tests known to work on shells other than bash (and in particular mksh)? any ideas where i should look (or some kind of parallelism i can just disable)?

~/aosp-master-with-phones/external/bc$ ./run-bc-tests-on-android.sh 
tests/: 413 files pushed, 0 skipped. 0.8 MB/s (1191155 bytes in 1.372s)
scripts/functions.sh: 1 file pushed, 0 skipped. 75.4 MB/s (12065 bytes in 0.000s)
***********************************************************************

Running bc tests...

Skipping bc lib2
Skipping bc shift
Skipping bc trunc
Skipping bc places
Skipping bc print test
SkRunning bc decimal...Running bc print2...ipping bc parse test
Running bc scale...Skipping bc scientific
Running bc length...Running bc sqrt...Running bc modulus...RuRunning bc add...nning bc divide...Running bc subtract...Skipping bc engineering
Running bc multiply...Running bc power...Running bc abs...Running bc functions...pass
Running bc boolean...Running bc comp...Running bc assignments...Running bc vars...pass
Skipping bc rand
pass
SRunning bc globals...kipping bc bitfuncs
pass
Running bc strings...pass
Running bc cosine...pass
Skipping bc strings2 test
Skipping bc bessel test
pass
Running bc sine...Running bc misc2...RRunning bc exponent...pRunning bc log...Running bc misc8...Running bc misc...Running bc misc4...Running bc pi...upass
Skipping bc leadingzero
nning bc misc1...ppasRunning bc arrays...asss

ass
Running bc arctangent...Running bc misc6...Running bc modexp...Running bc void...Running bc misc3...Running bc letters...pass
pass
pRunning bc misc7...ass
pass
pass
Running bc divmod...Running bc recursive_arrays...Running bc misc5...pass
pass
pass
pass
pass
pass
pass
ppass
ass
Running bc is_number...pass
pass
pass
pass
Running bc asciify_array...Running bc line_loop_quit1...pass
ppass
pass
pass
pass
pass
ass
Running bc is_string...Running bc line_by_line1...Running bc line_by_line2...Running bc line_loop_quit2...Running bc command-line error tests...Running bc stdin tests...pass
Running bc error file 01.txt with clamping...pass
pass
pass
Running bc read...Running bc error file 02.txt with clamping...Running bc error file 03.txt with clamping...pass
pass
pass
pass
pass
pass
pass
pass
Running bc error file 01.txt without clamping...Running bc error file 04.txt with clamping...ppass
Skipping bc script multiply.bc
pass
ass
Running bc error file 02.txt without clamping...pass
Running bc error file 05.txt with clamping...Skipping bc script divide.bc
Running bc error file 03.txt without clamping...Skipping bc script subtract.bc
pass
Running bc errors...Running bc read errors...pass
Skipping bc script add.bc
Running bc error file 06.txt with clamping...Running bc error file 04.txt without clamping...pRunning bc error file 07.txt with clamping...ass
Running bc error file 01.txt through cat with clamping...pass
Running bc error file 08.txt with clamping...pass
Skipping bc script print.bc
Running bc error file 02.txt through cat with clamping...Running bc error file 03.txt through cat with clamping...pass
pass
pass
Running bc error file 05.txt without clamping...pass
pass
Running bc error file 09.txt with clamping...pass
Running bc error file 07.txt without clamping...pass
Skipping bc script parse.bc
RunRunning bc error file 04.txt through cat wnith clamping...ing bc error file 06.txt without clamping...pass
Running bc error file 10.txt with clamping...RRunning bc empty read...Running bc error file 01.txt through cat without clamping...unning bc script array.bc...RRunning bc error file 03.txt through cat without clamping...pass
paRunning bc error file 08.txt without clamping...Running bc error file 11.txt with clamping...pss
unning bc script array2.bc...ass
pass
pass
Running bc error file 02.txt through cat without clamping...pass
pass
Running bc script atan.bc...Running bc error file 09.txt without clamping...Running bc error file 05.txt through cat with clamping...pass
pass
Running bc error file 10.txt without clamping...Running bc error file 06.txt through cat with clamping...Running bc error file 12.txt with clamping...RunRunning bc error file 04.txt through cat without clamping...pass
Skipping bc script bessel.bc
pass
ning bc error file 07.txt through cat with clamping...Running bc script functions.bc...pass
Running bc error file 11.txt without clamping...Running bc error file 13.txt with clamping...pass
pass
Running bc error file 08.txt through cat with clamping...pass
pass
Running bc error file 14.txt with clamping...pass
Running bc read EOF...pass
Running bc error file 09.txt through cat with clamping...pass
pass
pass
Running bc error file 05.txt through cat without clamping...pass
pRunning bc script globals.bc...pass
ass
Running bc error file 15.txt with clamping...Skipping bc script: rand.bc
RRunning Rbc error ufninlien g10.txt  through cbca ts cwriitpht  len.bc...clamping...pass
unning bc error file 07.txt through cat without clamping...pass
RRunning bc error file 06.txt through cat without clamping...pass
Running bc error file 11.txt through cat with clamping...unning bc error file 12.txt without clamping...pass
Running bc error file 08.txt through cat without clamping...pass
Running bc error file 16.txt with clamping...pass
Running bc error file 14.txt without clamping...pass
Running bc error file 13.txt without clamping...pass
Running bc script references.bc...Running bc error file 17.txt with clamping...Running bc error file 09.txt through cat without clamping...pass
pass
pass
pass
pass
pass
RuRunning bc error file 15.txt without clamping...nning bc error file 18.txt with clamping...Running bc script screen.bc...pass
Rpass
papass
unning bc error file 12.txt through cat with clamping...ss
Running bc error file 11.txt through cat without clamping...Rpass
unning bc error file 10.txt through cat without clamping...Running bc error file 19.txt with clamping...Skipping bc script strings2.bc
pass
Running bc error file 13.txt through cat with clamping...pass
Running bc error file 16.txt without clamping...pass
Running bc error file 14.txt through cat with clamping...Running bc error file 20.txt with clamping...pass
Running bc error file 17.txt without clamping...pass
pass
Running bc error file 21.txt with clamping...paspass
Running bc script ifs.bc...s
Running bc error file 18.txt without clamping...pass
Running bc error file 15.txt through cat with clamping...pass
Running bc script ifs2.bc...Rpass
pass
Running bc error file 12.txt through cat without clamping...pass
unning bc error file 19.txt without clamping...Running bc erpass
ror file 22.txt with clamping...Running bc error file 13.txt through cat without clamping...Rpass
pRunning bc error file 20.txt without clamping...pass
ass
RuRRuunning bc error file 14.txt through cat without clamping...nnning bc script afl1.bc...unning bc error file 23.txt with clning bcamp einrrg..or. file 16.txt through cat with clamping...pass
pass
Running bc error file 17.txt through cat with clamping...Running bc error filepass
pass
pass
RunniRunning bc error file 15.txt through cat without clamping... 21.txtng bc error file 24.txt with clamping... without clamping...Running bc error file 18.txt through cat with clamping...pass
pass
pass
pass
pass
Running bc error file 25.txt with clamping...pass
Running bc error file 22.txt without clamping...Running bc error file 19.txt through cat with clamping...pass
Running bc error file 16.txt through cat without clamping...Running bc error file 23.txt without clamping...pass
pass
pass
Running bc error file 20.txt through cat with clamping...pass
Running bc error file 26.txt with clamping...Running bc error file 17.txt through cat without clamping...Running bc error file 24.txt without clamping...Running bc error file 27.txt with clamping...pass
Running bc error file 18.txt through cat without clamping...pass
pass
pass
Running bc error file 21.txt through cat with clamping...pass
Running bc error file 19.txt through cat without clamping...pass
Running bc error file 22.txt through cat with clamping...pass
Running bc error file 25.txt without clamping...Running bc error file 28.txt with clamping...pass
pass
pass
pass
pass
Running bc error file 23.txt through cat with clamping...Running bc error file 20.txt through cat without clamping...RuRunning bc error file 29.txt with clamping...Running bc error file 24.txt through cat with clamping...pass
nning bc error file 26.txt without clamping...pass
Running bc error file 27.txt without clamping...pass
pass
pass
pass
Running bc error file 22.txt through cat without clamping...Running bc error file 30.txt with clamping...Running bc error file 21.txt through cat without clamping...Running bc error file 31.txt with clamping...pass
pass
Running bc error file 25.txt through cat with clamping...Running bc error file 28.txt without clamping...pass
Running bc error file 23.txt through cat without clamping...pass
Running bc error file 27.txt through cat with clamping...pass
Running bc error file 32.txt with clamping...Running bc error file 24.txt through cat without clamping...Skipping problematic bc error file 33.txt...
Running bc error file 26.txt through cat with clamping...pass
pass
pass
pass
paRunning bc error file 29.txt withss
pout clamping...ass
pass
RuRupass
A test failed!
Running bc error file 25.txt through cat without clamping...Running bc error file 30.txt without clamping...passnning bc error file 28.txt through nning bc error file 31.txt without clamping...
cat with clamping...pass
Running bc error file 27.txt through cat without clamping...pass
pass
A test failed!
Running bc error file 34.txt with clamping...Running bc error file 32.txt without clamping...pass
pass
A test failed!
Running bc error file 26.txt through cat without clampiRunning bc error file 29.txt through cat with clamping...pass
ng...pRunning bc error file 35.txt with clamping...pass
Running bc error file 31.txt through cat with clamping...A Running bc error file 36.txt with clamping...pass
Running bc error file 30.txt through cat with clamping...
Running bc quit test...test failed!
ass
pass
A test failed!
pass
Running bc error file 32.txt through cat with clamping...Running bc error file 28.txt through cat without clamping...Running bc error file 34.txt without clamping...pass
pass
pass
pass
RA pass
test failed!
unning bc error file 29.txt through cat without clamping...Running bcpass
 error file 35.txt without clamping...Running bc error file 30.txt through cat without clamping...pass
Running bc error file 31.txt through cat without clamping...Running bc error file 36.txt without clamping...A pass
test failed!
pass
Running bc error file 34.txt through cat with clamping...pA Running bc error file 32.txt through cat without clamping...ass
test failed!
pass
pass
pass
pass
A test failed!
RuRunning bc error file 35.txt through cat with clamping...nning bc error file 36.txt through cat with clamping...Rupass
nning bc environment var tests...pass
A test failed!
Running bc error file 34.txt through cat without clamping...A pass
test failed!
pass
Running bc error file 36.txt through cat without clamping...A test failed!pass
Running bc error file 35.txt through cat witpass

hout clamping...Running keyword redefinition test...paA test failed!
ss
pass
A test failed!
A test failed!
A test failed!
A test failed!
A test failed!
pass
A test failed!
Running multiline comment expression file test...A test failed!
A test failed!
A test failed!
A test failed!
pass
A test failed!
A test failed!
A Rtest ufnaniilnegd !m
ultiline comment expression file error test...A test failed!
A test failed!
A test failed!
pass
A test failed!
A test failed!
Running multiline string expression file test...A test failed!
A test failed!
A test failed!
A test failed!
pass
Running multiline string expression file error test...A test failed!
A test failed!
pass
A test failed!
Running bc line length tests...A test failed!
A test failed!
A test failed!
A test failed!
A test failed!
A test failed!
A test failed!
A test failed!
A test failed!
A test failed!
A test failed!
A test failed!
pass
A test failed!
A test failed!
Running bc arg tests...pass
Running bc builtin variable arg tests...pass
Running bc directory test...pass
Running bc binary file test...pass
Running bc binary stdin test...pass
Running bc limits tests...pass
pass
Running bc posix_errors...pass
~/aosp-master-with-phones/external/bc$ 
gavinhoward commented 1 year ago

i haven't been able to update for some time (5.0.1 was our last update!)

Oof...I'm sorry. Have I been missing communication with you?

because the tests fail in a weird way without much useful information. i've included the current failures for 6.2.2 below, for example.

I agree; that is useless.

my guess is that this is something to do with Android using mksh as its /bin/sh. the interleaved output makes it appear like there might be some disagreement about job handling between bash and mksh?

I think that's a great guess, and it's my guess too because the tests/all.sh script does use job control. I tried to write it according to the POSIX standard, but I suspect that job control is finicky in general.

are the current bc tests known to work on shells other than bash (and in particular mksh)? any ideas where i should look (or some kind of parallelism i can just disable)?

They are known to work on tcsh (FreeBSD), but other than that, bash, and dash, I don't know.

any ideas where i should look (or some kind of parallelism i can just disable)?

To disable parallelism, change the last line of run-bc-tests-on-android.sh to:

exec adb shell $dash_t /data/local/tmp/bc-tests/tests/all.sh -n bc 0 1 0 0 0 bc

I made two changes:

My hope (and it is only a hope) is that that missing integer argument is what is causing the weird failures because the script would otherwise treat the last bc argument as the last integer argument and cause some sort of failure. bash fails because it has a non-integer argument, but I don't know how mksh would fail in that case.

The other reason there might be failures is because I added a test that is "problematic." It's my test for malloc() failure, and unfortunately, it only works on Linux systems with swap disabled and some other changes. I don't think it would work on Android. I did add an option to disable that test in tests/all.sh; that is the missing integer option, in fact. But I don't think that's the reason because I added it in the fourth position (the last is whether to time the tests), so it is a 0 in your current script.

Regardless, with the parallelism disabled, you should see exactly what test is failing, and you should be able to give me that output, even if either of those are not the reason for the failure.

In the meantime, I'll work on my end. Android is a first-class user of my bc, and I need to start treating it that way, including setting up a building and testing environment. I'll do my best to do so myself, but if possible, can you give me a helpful link?

enh-google commented 1 year ago

i haven't been able to update for some time (5.0.1 was our last update!)

Oof...I'm sorry. Have I been missing communication with you?

no, this is just the first time i've had time to actually look at the problem and file a bug!

are the current bc tests known to work on shells other than bash (and in particular mksh)? any ideas where i should look (or some kind of parallelism i can just disable)?

They are known to work on tcsh (FreeBSD), but other than that, bash, and dash, I don't know.

yeah, there's been a lot about job control on the POSIX mailing list recently too.

any ideas where i should look (or some kind of parallelism i can just disable)?

To disable parallelism, change the last line of run-bc-tests-on-android.sh to:

exec adb shell $dash_t /data/local/tmp/bc-tests/tests/all.sh -n bc 0 1 0 0 0 bc

yes, that works. thanks!

In the meantime, I'll work on my end. Android is a first-class user of my bc, and I need to start treating it that way, including setting up a building and testing environment. I'll do my best to do so myself, but if possible, can you give me a helpful link?

there's https://source.android.com/docs/setup/build/building (and other pages about running), but it's a pretty large undertaking.

tbh, i've only had two problems with bc updates:

  1. build changes such as the move from a shell script to a .c file for strgen. those are almost certainly easier for me to do anyway.
  2. test changes like this one. these have been harder for me to adapt to (although this is the first one i've failed with). there i'd say "yes, please make sure you add options like -n when you make scary changes" but one thing you haven't done but could that i think would help would be to fail hard --- making extra options optional (ho ho) makes my life harder. an error message would be more helpful, especially because changes to a row of 0s and 1s aren't necessarily immediately obvious!

anyway, https://android-review.googlesource.com/c/platform/external/bc/+/2385321 works for me locally, so it should make it through CI. (and if it doesn't, that's probably a question for me anyway :-) )

thanks!

gavinhoward commented 1 year ago

there's https://source.android.com/docs/setup/build/building (and other pages about running), but it's a pretty large undertaking.

I'll do my best anyway. Thank you!

tbh, i've only had two problems with bc updates

  1. build changes such as the move from a shell script to a .c file for strgen. those are almost certainly easier for me to do anyway.
  2. test changes like this one. these have been harder for me to adapt to (although this is the first one i've failed with). there i'd say "yes, please make sure you add options like -n when you make scary changes" but one thing you haven't done but could that i think would help would be to fail hard --- making extra options optional (ho ho) makes my life harder. an error message would be more helpful, especially because changes to a row of 0s and 1s aren't necessarily immediately obvious!

I think both of these are oversights on my part. I should add sections to my NEWS.md entry for each version about building, packaging, and testing changes. I've never thought they were necessary to list, but here is hard evidence that yes, they are.

I apologize. I will do that going forward. And I also apologize for accidentally using you as my test subject while I learn how to run a project.

Now, about making things fail hard: I agree that I messed up by making arguments optional.

However, because I have downstream users that could depend on the options staying optional, I'm not sure I can make them required at this point.

Nevertheless, I think I can do the next best thing: do error checking on each argument. This would (hopefully) allow me to add the "fail hard" you want (and it's a good idea!) while not causing problems for other downstream users. (Unless they have bugs in their scripts, in which case, they should be fixed anyway.)

I have implemented the ideas in d5e6dbdf328407bddf53efb655abe4b9c2fcb90f, so they'll be in the next release. I hope that helps.

enh-google commented 1 year ago

I apologize. I will do that going forward. And I also apologize for accidentally using you as my test subject while I learn how to run a project.

no worries --- we're a bit of an outlier anyway. it's only when you get included in a "ridiculously large" meta-project (most commonly an operating system [or moral equivalent] such as android or chromium or whatever) that you tend to find people not using your build system because the pain of having to duplicate your build system is still slightly less than the pain of trying to incorporate your build system into theirs!

Now, about making things fail hard: I agree that I messed up by making arguments optional.

However, because I have downstream users that could depend on the options staying optional, I'm not sure I can make them required at this point.

Nevertheless, I think I can do the next best thing: do error checking on each argument. This would (hopefully) allow me to add the "fail hard" you want (and it's a good idea!) while not causing problems for other downstream users. (Unless they have bugs in their scripts, in which case, they should be fixed anyway.)

yeah, maybe if you ever need more arguments, make them named? (like --foo=bar or foo=bar?)

I have implemented the ideas in d5e6dbd, so they'll be in the next release. I hope that helps.

thanks. since you're obviously interested in the feedback, i'll try to let you know sooner next time i have an update where manual intervention is required, if i have any trouble :-)