Yelp / mrjob

Run MapReduce jobs on Hadoop or Amazon Web Services
http://packages.python.org/mrjob/
Other
2.61k stars 587 forks source link

make tests work on Windows #293

Open coyotemarin opened 12 years ago

coyotemarin commented 12 years ago

I tried running the automated tests on a Windows 7 machine (well, a virtual machine), and about half of them failed. The most common problems had to do with tzset() (which isn't available on Windows) and trying to unlink (delete) files that were still in use.

It would make releases a lot smoother if we could just run the test suite on Windows.

coyotemarin commented 12 years ago

Some other issues:

coyotemarin commented 12 years ago

Yep, the counter parsing was a real bug. Using just \n in StringIOs seems to work fine.

I've changed rstrip('\n') to rstrip('\r\n') everywhere in the code. Usually, Python's Unix line endings compatibility stuff works fine, but in some cases it fails (such as $ in regexes).

coyotemarin commented 12 years ago

Also, Windows sort is apparently case-insensitive (see Issue #298), causing some of the tests in tests.runner_test.TestInvokeSort to fail. Probably should have a separate test for case-sensitivity that can fail or be skipped on Windows.

coyotemarin commented 12 years ago

Symlinks are a problem on Windows.

Also, we sometimes end up not having permission to delete files created by tests; not sure what that's about.

coyotemarin commented 12 years ago

Not entirely sure how to create fake binaries on Windows. Maybe .bat files?

coyotemarin commented 8 years ago

Appveyor CI is integrated with GitHub and tests on Windows.

coyotemarin commented 7 years ago

Got partway through this. This is too much hassle to do on both v0.5.x and v0.6.0; some of the test framework is very unix-oriented. Going to make sure v0.6.0 works on Windows, and may backport fixes for any obvious problems.

coyotemarin commented 7 years ago

The lack of sh on Windows means we'll need to skip a lot of local mode tests.

coyotemarin commented 7 years ago

This is harder than I thought. Let's try to finish this in v0.6.1.