sympy / sympy-bot-old

SymPy pull request helper
http://reviews.sympy.org/
Other
24 stars 16 forks source link

The traceback is not included if a test kills the test runner #35

Open asmeurer opened 13 years ago

asmeurer commented 13 years ago

At http://pastehtml.com/view/b91akw9zi.html, the test runner was killed. This was the output in my terminal:

sympy/functions/special/tests/test_gamma_functions.py[9] .........          [OK]
sympy/functions/special/tests/test_hyper.py[5] .....                        [OK]
sympy/functions/special/tests/test_spec_polynomials.py[5] .....             [OK]
sympy/functions/special/tests/test_spherical_harmonics.py[5] .....          [OK]
sympy/functions/special/tests/test_tensor_functions.py[2] ..                [OK]
sympy/functions/special/tests/test_zeta_functions.py[2] ..                  [OK]
Traceback (most recent call last):
  File "setup.py", line 249, in <module>
    'audit' : audit,
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", line 974, in run_commands
    self.run_command(cmd)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", line 994, in run_command
    cmd_obj.run()
  File "setup.py", line 164, in run
    if sympy.test():
  File "/private/var/folders/wc/dppcpmxs1tlb36nqcw853wkm0000gn/T/sympy-bot-tmpJf2kIk/sympy/sympy/utilities/runtests.py", line 193, in test
    return t.test(sort=sort)
  File "/private/var/folders/wc/dppcpmxs1tlb36nqcw853wkm0000gn/T/sympy-bot-tmpJf2kIk/sympy/sympy/utilities/runtests.py", line 518, in test
    self.test_file(f)
  File "/private/var/folders/wc/dppcpmxs1tlb36nqcw853wkm0000gn/T/sympy-bot-tmpJf2kIk/sympy/sympy/utilities/runtests.py", line 532, in test_file
    execfile(filename, gl)
  File "/private/var/folders/wc/dppcpmxs1tlb36nqcw853wkm0000gn/t/sympy-bot-tmpjf2kik/sympy/sympy/galgebra/tests/test_ga.py", line 15, in <module>
    if not disabled:
NameError: name 'disabled' is not defined
> switch back to master: git checkout master
> delete temporary branch: git branch -D next-test
Done.

View report at: /var/folders/wc/dppcpmxs1tlb36nqcw853wkm0000gn/T/sympy-bot-tmpJf2kIk/out/report.html
View log at: /var/folders/wc/dppcpmxs1tlb36nqcw853wkm0000gn/T/sympy-bot-tmpJf2kIk/out/log
> Uploading test results
> Uploaded report at: http://pastehtml.com/view/b91akw9zi.html
> Review:
Test results html report: http://pastehtml.com/view/b91akw9zi.html

**Note** A custom test command was used: python2.5 setup.py test

Summary: There do not appear to be any problems.

Please double check this against the test report given above before merging with master.

Automatic review by [sympy-bot](https://github.com/sympy/sympy-bot).
> Uploading the review to the GitHub pull request ...
>     Done.
> Check the results: https://github.com/sympy/sympy/pull/625
> Merged pull request available in: /var/folders/wc/dppcpmxs1tlb36nqcw853wkm0000gn/T/sympy-bot-tmpJf2kIk

You can see that the traceback was not included in the test report. My guess is that we need to pipe stderr to the report in addition to stdout, but I haven't checked.

asmeurer commented 13 years ago

I believe this is the problem, and it leads to another annoying things not being included in the test report as well, such as warnings. For example, if you create a file called test.py with the following:

import warnings
warnings.warn('test')
print 'print test'
raise RuntimeError

and run

$python test.py 2> test

Then print test will be printed to the terminal, and the file test will contain the output from standard error:

test.py:2: UserWarning: test
  warnings.warn('test')
Traceback (most recent call last):
  File "test.py", line 4, in <module>
    raise RuntimeError
RuntimeError

I'm not sure how to fix this though. The basic way that the test report is written (in sympy-next) is

out = subprocess.Popen(command, stdout=subprocess.PIPE).stdout

report = []
def my_join(file):
    while True:
        char = file.read(1)
        if not char:
            break
        log.write(char)
        log.flush()
        if verbose:
            sys.stdout.write(char)
        yield char
def my_split(iter):
    buf = ''
    for c in iter:
        buf += c
        splits = ['sympy/', 'doc/']
        for s in splits:
            if buf.endswith(s):
                r = buf[:-len(s)]
                buf = s
                yield r
    yield buf

for line in my_split(my_join(out)):
    import re
    good = None
    if line.find('[OK]') != -1:
        good = True
    elif line.find('[FAIL]') != -1:
        good = False
    elif re.search('     \[\d*\]', line):
        good = False
    if good is None:
        continue
    report.append((line.split('[')[0].split()[0], good))

Note how we get the output of stdout from the command to write to the file (stderr is written the the console automatically). How do we also write the output of stderr in the order (or roughly the same order) that it was written to the console, so that we can see where the problems were?