./jtrts.pl claims john returned with return code 1

I see this for dynamic_12 and dynamic_1503

$ ./jtrts.pl -type dynamic_12 -q
-------------------------------------------------------------------------------
- JtR-TestSuite (jtrts). Version 1.13, Dec 21, 2014.  By, Jim Fougeron & others
- Testing:  John the Ripper 1.8.0.6-jumbo-1-199-geacddf1 OMP [linux-gnu 64-bit AVX2-autoconf]
--------------------------------------------------------------------------------
Warning: SAP-B format should never be UTF-8.
Use --target-encoding=iso-8859-1 or whatever is applicable.
.pot CHK:dynamic_12               guesses:    0 -show=   0 unk unk : Expected count(s) (1500)  [!!!FAILED6!!! exited, return code 1]
.pot CHK:dynamic_12               guesses:    0 -show=   0 unk unk : Expected count(s) (1500)  [!!!FAILED6!!! exited, return code 1]
Some tests had Errors. Performed 2 tests.  2 errors reprocessing the .POT files  2 runs had non-clean exit
Time used was 2 seconds
$ ./jtrts.pl -type dynamic_12 -v -stoponerror
-------------------------------------------------------------------------------
- JtR-TestSuite (jtrts). Version 1.13, Dec 21, 2014.  By, Jim Fougeron & others
- Testing:  John the Ripper 1.8.0.6-jumbo-1-199-geacddf1 OMP [linux-gnu 64-bit AVX2-autoconf]
--------------------------------------------------------------------------------

John Jumbo build detected.
Warning: SAP-B format should never be UTF-8.
Use --target-encoding=iso-8859-1 or whatever is applicable.
--pot=NAME option is valid
--encoding=NAME option is valid
all.chr (../run/all.chr) not found
alnum.chr found, inc_alnum_ee8763c850dee8e4b88ef547a8ed39b8 added as a capability
Types to filter on:
dynamic_12
Capabilities in this build of john:
jumbo core inc local_pot_valid encode_valid utf8 cp1252 cp1251 koi8r cp437 cp737 cp850 cp858 cp866 iso8859-1 iso8859-15 inc_alnum_ee8763c850dee8e4b88ef547a8ed39b8

form=dynamic_12                   guesses: 1500 0:00:00:00 DONE  [PASSED]
.pot CHK:dynamic_12               guesses:    0 -show=   0 unk unk : Expected count(s) (1500)  [!!!FAILED6!!! exited, return code 1]
Exiting on error. The .pot file tst.pot contains the found data
The command used to run this test was:

../run/john -ses=tst  -pot=tst.pot dynamic_12_tst.in --wordlist=pw.dic

But executing

../run/john -ses=tst  -pot=tst.pot dynamic_12_tst.in --wordlist=pw.dic

cracks the expected 1500 passwords, and exit code is 0:

$ ../run/john -ses=tst  -pot=tst.pot dynamic_12_tst.in --wordlist=pw.dic
Warning: invalid UTF-8 seen reading dynamic_12_tst.in
Using default input encoding: UTF-8
Loaded 1500 password hashes with 1500 different salts (dynamic_12 [md5(md5($s).md5($p)) (IPB) 256/256 AVX2 8x3])
Warning: no OpenMP support for this hash type, consider --fork=4
Press 'q' or Ctrl-C to abort, almost any other key for status
aksarban__5      (u471-dynamic_12)
[...]

Skipping and& Dipping__4 (u362-dynamic_12)
1500g 0:00:00:00 DONE (2015-07-03 19:03) 3571g/s 4000p/s 6000Kc/s 6000KC/s Skippin� an�*..IwasJu$there__19
Use the "--show" option to display all of the cracked passwords reliably
Session completed
$ echo $?
0

$ ../run/john -show -pot=tst.pot dynamic_12_tst.in Warning: invalid UTF-8 seen reading dynamic_12_tst.in u0-dynamic12:Skippin� an�:0:0:Skippin� an�_:: [...] u1499-dynamic_12:118:1499:0:118::

1500 password hashes cracked, 0 left $ echo $? 0

So, what command did jtrts.pl really run which didn't crack any passwords and ended with exit code 1?
Obviously not the command listed with `-v`.

openwall / john-tests

./jtrts.pl claims john returned with return code 1 #53