search

Chris-Johnston / Easier68k

Work In Progress - A Python M68k assembler and simulator.
https://chris-johnston.me/Easier68k/
MIT License
15 stars 6 forks source link

Add pytest-cov support #114

Closed bpas247 closed 5 years ago

bpas247 commented 6 years ago

Fixes #65.

Currently testing to see if Travis generates a code coverage report.

bpas247 commented 6 years ago

I'm not sure what's happening, but it appears as if some tests are now failing. It seems as though it is not able to open certain test files, although I'm not sure why. Will investigate it when I have the time.

Chris-Johnston commented 6 years ago

Weird, unfortunately I don't have much time to test this myself, but I wonder if others have found solutions to this issue.

bpas247 commented 6 years ago

Finally got this to work. I had to re-write how the file paths were determined for the affected failing tests. Is there anything else that needs to be done for this PR?

Chris-Johnston commented 6 years ago

In the CI output (https://travis-ci.org/Chris-Johnston/Easier68k/jobs/398509794#L3201), you can see that the pytests are being re-run. We should only run these once, with the test coverage. That, and the coverage percentage is reporting 0%, which appears to be wrong.

bpas247 commented 6 years ago

@Chris-Johnston

In the CI output (https://travis-ci.org/Chris-Johnston/Easier68k/jobs/398509794#L3201), you can see that the pytests are being re-run. We should only run these once, with the test coverage.

This occurred because Travis was running 2 commands that both called py.test. Now Travis only calls it once.

That, and the coverage percentage is reporting 0%, which appears to be wrong.

I'm not entirely sure why this was happening, but after I fixed the first error it seems to be reporting the coverage just fine now.

Chris-Johnston commented 6 years ago

The output looks a lot better, but it appears that the test coverage is only being calculated for the files under the test directory. This explains why we have nearly 100% code coverage reported.

Maybe there is a way we can 'include' another directory when calculating the code coverage. Code coverage for our tests doesn't do us much good.

bpas247 commented 6 years ago

Okay now I really don't understand why it's not working. Perhaps it's ignoring the .coverage file or not even generating it at all? I really don't understand how there are hits (and misses) for these tests yet no coverage percentage.

Edit: The weirdest part is, if you run the tests ("run_pytest") locally it will show the percentages.

Chris-Johnston commented 6 years ago

This is such a weird issue. It's writing to a .coverage file, but all of the lists of line numbers that should appear in that file are empty. When run locally, this file contains a lot of numbers that just aren't present here. I guess that would make sense, because it is reporting no data was collected.

Chris-Johnston commented 6 years ago

I don't know why that works. Oh wait, that's code coverage for the tests. So it still doesn't work.

I think it's because of specifying --cov first, or adding source in the .coverageac.

Chris-Johnston commented 6 years ago

Changing the install argument of python setup.py install to develop fixed the output. I'm not sure if that might have some other consequences. Might have to look into what the difference is between those arguments.

Chris-Johnston commented 6 years ago

Apparently, there's a --user flag that we could have been using for the ./.testingScript.sh.

This no longer requires running this script as root (which is much safer). If you have previously run this script, you may need to remove the directories .eggs/ and easier68k.egg-info/ first.

bpas247 commented 6 years ago

Spotted another error. Apparently in the Travis instructions (.travis.yml) there was an instruction between the doc_test.py and run_test.py that changed the directory (cd ..). This resulted in some weird behavior, particularly with an inconsistency between the Travis build and running run_test.py locally. So I removed it and it seems to be working.

Chris-Johnston commented 6 years ago

I might have left that in by accident when messing with things.

bpas247 commented 5 years ago

What else needs to be done here? seems to be working

bpas247 commented 5 years ago

Yikes, looks like os.path.join might not work with how we are accessing the current directory. Gonna have to look into this more

bpas247 commented 5 years ago

Easy enough fix, now awaiting for final approval