Make test.script_helper more comprehensive, and use it in the test suite

[pitrou]

BPO	9517
Nosy	@ncoghlan, @pitrou, @ezio-melotti, @bitdancer, @voidspace, @berkerpeksag, @serhiy-storchaka
Dependencies	bpo-23981: Update test_unicodedata.py to use script_helpers bpo-24033: Update _test_multiprocessing.py to use script helpers bpo-24279: Update test_base64 to use test.support.script_helper bpo-24398: Update test_capi to use test.support.script_helper
Files	script_helper_del_refcount.patch iss9517_move_script_helpers_py.patch iss9517_move_script_helpers_py.patch iss9517_move_script_helpers.patch issue9517.diff iss9517_move_script_helpers_review1.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['type-feature', 'tests'] title = 'Make test.script_helper more comprehensive, and use it in the test suite' updated_at = user = 'https://github.com/pitrou' ``` bugs.python.org fields: ```python activity = actor = 'BreamoreBoy' assignee = 'none' closed = False closed_date = None closer = None components = ['Tests'] creation = creator = 'pitrou' dependencies = ['23981', '24033', '24279', '24398'] files = ['19975', '39003', '39038', '39176', '39177', '39189'] hgrepos = [] issue_num = 9517 keywords = ['patch'] message_count = 55.0 messages = ['112917', '112969', '113145', '119517', '119741', '123503', '123504', '123516', '123601', '123602', '123633', '123637', '173718', '221971', '222027', '240305', '240524', '240555', '240664', '240801', '240818', '240982', '240990', '241001', '241010', '241069', '241101', '241103', '241112', '241120', '241122', '241165', '241186', '241221', '241238', '241240', '241244', '241295', '241836', '241839', '241842', '241848', '241906', '241913', '242090', '242641', '242690', '242934', '242945', '243000', '243030', '244001', '244924', '245920', '247985'] nosy_count = 11.0 nosy_names = ['ncoghlan', 'pitrou', 'ezio.melotti', 'r.david.murray', 'michael.foord', 'Rodrigue.Alcazar', 'python-dev', 'berker.peksag', 'serhiy.storchaka', 'bobcatfish', 'flipmcf'] pr_nums = [] priority = 'normal' resolution = None stage = 'patch review' status = 'open' superseder = None type = 'enhancement' url = 'https://bugs.python.org/issue9517' versions = ['Python 3.5'] ```

[pitrou]

test.script_helper has a couple of dedicated functions to launch a Python interpreter instance in a subprocess. Unfortunately, it is little used and most test modules use their own ad hoc calls to subprocess instead.

Remedying the situation would require:

• adding functions to script_helper (currently, the available functions merge stdout and stderr together, which is clearly undesireable)
• perhaps improve the existing functions (kill_python() does a strange dance instead of calling communicate() on the subprocess.Popen object, is there a reason for that?)
• convert most uses of subprocess.\<some_func>([sys.executable, ...]) in the test suite to use script_helper instead

This was suggested by Nick in bpo-477863.

[ncoghlan]

(The email daemon was not in a happy place, so posting directly)

On Thu, Aug 5, 2010 at 8:50 AM, Antoine Pitrou \report@bugs.python.org\ wrote:

• perhaps improve the existing functions (kill_python() does a strange dance instead of calling communicate() on the subprocess.Popen object, is there a reason for that?)

script_helper just factored out the old test_cmd_line approach which was in turn based on a minimalistic change from a popen2 based implementation (see http://svn.python.org/view/python/trunk/Lib/test/test_cmd_line.py?r1=54386&r2=55245). Doing something smarter probably isn't a bad idea.

[ncoghlan]

One other feature for the new-and-improved helpers: add a flag to allow "-E" to be omitted (as per the comment in test_cmd_line)

[ncoghlan]

I still think this is a good idea, I'm just not actively working on it. It might make a good project for someone wanting to get to know the process of working on CPython without having to deal with anything that is particularly tricky to understand.

[ef24ed3e-2f9a-4ea4-a112-cca227b0870a]

someone wanting to get to know the process of working on CPython without having to deal with anything that is particularly tricky to understand.

That sounds exactly like me :)

I can have a look at this ticket.

[bitdancer]

I just tried using script_helper in a new test, so I have a couple of comments.

I don't see stdout and stderr being conflated, it looks to me like they are returned separately, at least by the assert methods.

The assert methods return results, which is unlike other assert methods. This is very useful, even essential, and I wouldn't want to give it up. That conflicts with the current unittest conventions, though.

It would be a big help if 'err' were returned with the refcount line removed if it is there, which would make tests using the methods return the same 'err' regardless of whether they are run under a debug build or not.

I think the names of the two assert functions should follow the current unit test conventions (assertPythonRunOK and asssertPythonRunNotOK, perhaps?)

[pitrou]

I just tried using script_helper in a new test, so I have a couple of comments.

I don't see stdout and stderr being conflated, it looks to me like they are returned separately, at least by the assert methods.

That's because I wrote the assert methods since this issue was opened :)

It would be a big help if 'err' were returned with the refcount line removed if it is there, which would make tests using the methods return the same 'err' regardless of whether they are run under a debug build or not.

Indeed.

I think the names of the two assert functions should follow the current unit test conventions (assertPythonRunOK and asssertPythonRunNotOK, perhaps?)

Well, they are functions, not methods, so I don't think they have to follow the other convention.

[bitdancer]

OK, fine on the convention, but I'd still like a more memorable name for assert_python_failure. I've been working on this issue off and on today, and I've had to look up that name at least four times. I can remember assert_python_ok, but I can't remember whether its inverse is assert_python_fails, assert_python_bad, or what. For some reason I haven't guessed 'failure' even once so far :) (I know it's not assert_python_not_ok because I remember it isn't parallel...)

[bitdancer]

Here is a patch that causes _assert_python to remove the refcount lines from stderr.

[bitdancer]

Hmm. Having posted that it occurs to me that it could be useful to have the _remove_refcount function in test.support as remove_refcount instead.

[pitrou]

Having posted that it occurs to me that it could be useful to have the _remove_refcount function in test.support

There's already strip_python_stderr() :)

[bitdancer]

Oh, good, I'll use that then. I could have sworn I looked for that functionality a couple weeks ago and couldn't find it.

[ezio-melotti]

There's already strip_python_stderr() :)

This is already used in Lib/test/script_helper.py:42, and a way to call _assert_python() without using -E has also been introduced0.

The next step would be to check if these functions could be used elsewhere in the other tests. Also note that these functions are not currently documented anywhere.

[83d2e70e-e599-4a04-b820-3814bbdb9bef]

@Rodrigue did you ever make any progress with this?

[ef24ed3e-2f9a-4ea4-a112-cca227b0870a]

@Mark, I had a look at the time I commented on the ticket but didn't get a chance to progress much in the end.

[bitdancer]

Script helpers has been "made more discoverable" by moving it into the test.support namespace, but it has not been documented in the test.support docs. I can no longer remember if this is intentional or not. Regardless, this issue is still valid insofar as there are probably a number of places in the test suite where script_helpers can be used that they aren't being used. We probably don't want a multi-module changeset, though, so this could become a meta issue for new issues for converting particular test files to use script_helpers.

[8f1ab454-ee12-4032-b9d0-031d0f052a3b]

Hello @r.david.murray!

We probably don't want a multi-module changeset, though, so this could become a meta issue for new issues for converting particular test files to use script_helpers.

Does this mean you'd like to leave this issue open until all of the test files are updated in separate issues?

Or is it that you'd like separate changesets for each module, but all associated with this issue?

Thanks!

[bitdancer]

It could be done either way, but I suspect that indovidual issues for test module changes, entered as dependencies on this issue, is probably the more effective in this case.

[8f1ab454-ee12-4032-b9d0-031d0f052a3b]

Okay thanks @r.david.murray, I'll take a look at doing this. I'll open an issue when I figure out which module to start on.

[bitdancer]

Apparently I was imaginig that script_helpers had been moved into test.support. test.support was changed into a module in prep for doing so, but as far as I can see it was never moved. So that does remain to be done as well, and should be done first.

[8f1ab454-ee12-4032-b9d0-031d0f052a3b]

Okee dokee, I'll tackle that first!

[cf1faa91-0864-4e33-8fce-569bfae69060]

I have a patch for moving script_helper as R. David suggested.

Here it is.

[cf1faa91-0864-4e33-8fce-569bfae69060]

bobcatfish: Sorry, I didn't refresh and see your comment before submitting my patch. It only moves script_helpers, but doesn't address the original OP.

Hopefully I made your life easier, not harder.

[cf1faa91-0864-4e33-8fce-569bfae69060]

re: test.test_tools.py - should this also move into test.support ?

[bitdancer]

test_tools is a test suite that tests the tools in the scripts directory, it is not tools for testing. So no, it doesn't belong in test.support.

[8f1ab454-ee12-4032-b9d0-031d0f052a3b]

Hey there @flipmcf, is the change which adds script_helpers to test.support is missing from your patch?

>>> import test.support.script_helper
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named 'test.support.script_helper'

I'm thinking also, maybe instead of putting script_helper into the test.support namespace, we could move the individual methods and helpers from script_helper into test.support, e.g.:

test.support.run_python_until_end vs. test.support.script_helper.run_python_until_end

[cf1faa91-0864-4e33-8fce-569bfae69060]

Yes, I did miss that.

(Root cause: git diff instead of git diff --staging)

Sorry. I'm learning, and decided to take the git path - might not have been the best choice.

I'll submit a new patch shortly.

[cf1faa91-0864-4e33-8fce-569bfae69060]

I'm thinking also, maybe instead of putting script_helper into the test.support namespace, we could move the individual methods and helpers from script_helper into test.support, e.g.:

test.support.run_python_until_end vs. test.support.script_helper.run_python_until_end1

Yes, I also agree that this looks better.

But as this was first humble patch submission to CPython, I followed David Murry's instructions during my review - which was to move the entire module.

I would be opposed to adding the methods directly into __init__.py,

I would recommend still moving script_helper into test.support , and adding a list of symbols from scripthelper.py into \_init__

eg:

test/support/init.py: from script_helper import run_python_until_end

test/some_test.py: from test.support import run_python_until_end

I am currently in the PyCon CPython sprint room if you are also here.

[cf1faa91-0864-4e33-8fce-569bfae69060]

Uploading new patch that includes the creation of Lib/test/support/script_helper.py

[8f1ab454-ee12-4032-b9d0-031d0f052a3b]

I would recommend still moving script_helper into test.support , and adding a list of symbols from scripthelper.py into \_init__

Sounds like a great plan to me @flipmcf!

[bitdancer]

I actually originally wanted to do just that (move the script helpers routine into the old support.py) but I was in the minority. test.support was turned into a package exactly so that script_helpers could become a sub-namespace. As I remember, it was Nick Coghlan who championed the sub-namespace position :)

[cf1faa91-0864-4e33-8fce-569bfae69060]

namespaces are a honkin' great idea

I don't have enough information to say that 'script_helper' is the right name for the namespace, tho.

[8f1ab454-ee12-4032-b9d0-031d0f052a3b]

@r.david.murray and @flipmcf:

I've audited all test modules using sys.executable for a starting point, and these tests look like they could benefit from helper methods to invoke python (probably the ones in script_helper, but we may want to create others if the use cases don’t quite fit):

• _test_multiprocessing.py
• test_base64.py
• test_capi.py - some of this looks legitimate, but some of it could maybe benefit from the helpers
• test_cmd_line.py
• test_faulthandler.py
• test_file_eintr.py
• test_gc.py
• test_keyword.py
• test_pdb.py
• test_popen.py
• test_quopri.py
• test_site.py
• test_source_encoding.py
• test_sys.py
• test_sysconfig.py
• test_tcl.py - something complex is going on in here, not sure if it’s necessary or not
• test_threading.py
• test_pindent.py
• test_traceback.py
• test_unicodedata.py
• test_json/test_tool.py

@r.david.murray I'm going to start with test_unicodedata.py for a nice easy starting point so I'll create a separate issue for it.

@flipmcf should I start using your patch set as-is or is there more to come?

[cf1faa91-0864-4e33-8fce-569bfae69060]

@Christie - Nope. This patch can stand on it's own, simply moving the package to where it belongs.

In fact, I'd recommend applying the patch first.

[8f1ab454-ee12-4032-b9d0-031d0f052a3b]

So @flipmcf we're sticking with the test.support.script_helper namespace?

[berkerpeksag]

Alternatively, you can import public functions from support/script_helper.py in support/init.py.

Then you can do

    from test.support import assert_python_ok

[berkerpeksag]

Reviewing iss9517_move_script_helpers_py.patch:

• Can you move script_helper.py by using "hg mv"?

[8f1ab454-ee12-4032-b9d0-031d0f052a3b]

@r.david.murray I've created bpo-23981, I don't think I have permissions to add it as a dependency to this issue.

[8f1ab454-ee12-4032-b9d0-031d0f052a3b]

Hey @berker.peksag, @r.david.murray!

Here's another change set where script_helpers.py is moved with an hg mv (still looks in the patch like the file was deleted and re-added, not sure if that's expected).

I've ALSO removed some unused imports from script_helpers.py - temp_dir was being imported from script_helpers in a few places tho, so I had to fix that. If you guys would rather not include that change here, please let me know!

[8f1ab454-ee12-4032-b9d0-031d0f052a3b]

Created bpo-24033. If it and bpo-23981 could be added as dependencies of this issue (or I could get permission to add them?) that would be swell!

[berkerpeksag]

Thanks for the patch, Christie. Review comments: http://bugs.python.org/review/9517/

(still looks in the patch like the file was deleted and re-added, not sure if that's expected).

I don't know if it's important. I'm using a very old version of Mercurial (2.0.2 :)), so it can be related to different Mercurial versions. Attaching a patch for reference.

[serhiy-storchaka]

You should create the patch with "hg diff --git" to preserve moving. And then apply it with "hg import", not the patch command. Unfortunately Rietveld don't like patches in git format.

[8f1ab454-ee12-4032-b9d0-031d0f052a3b]

Hey @berker.peksag, I've added a new patch in response to your review!

You should create the patch with "hg diff --git" to preserve moving. And then apply it with "hg import", not the patch command. Unfortunately Rietveld don't like patches in git format.

Ick, I'm guessing it's okay to just use plain old "hg diff" then, if Rietveld doesn't like the patches in git format.

[serhiy-storchaka]

The committer should be careful, and manually make "hg mv" and apply the patch with the patch command, not "hg import". We shouldn't lost the history.

[8f1ab454-ee12-4032-b9d0-031d0f052a3b]

@serhiy.storchaka - just double checking, do you guys need me to make any more changes to the patch?

And is there any more review needed, or is it possible for this to be merged?

Thanks very much!

[1762cc99-3127-4a62-9baf-30c3d0f51ef7]

New changeset f65174aef9ea by Berker Peksag in branch 'default': Issue bpo-9517: Move script_helper to the support package. https://hg.python.org/cpython/rev/f65174aef9ea

[8f1ab454-ee12-4032-b9d0-031d0f052a3b]

Thanks very much @berker.peksag!

[8f1ab454-ee12-4032-b9d0-031d0f052a3b]

Hey @berker.peksag, @r.david.murray, @serhiy.storchaka,

If you get a chance I've got some changes up for review at:

• http://bugs.python.org/issue24033
• http://bugs.python.org/issue23981

Thanks!

[ncoghlan]

Just noting that bpo-18576 has a draft patch for test.support.script_helper documentation (the "move to the support module" part of the current patch there is obsolete)

[8f1ab454-ee12-4032-b9d0-031d0f052a3b]

Cool, thanks @ncoghlan! Would you like someone to take on the work of updating the latest patch on bpo-18576?