search

python / cpython

The Python programming language
https://www.python.org
Other
63.38k stars 30.35k forks source link

Batch file to mimic 'make' on Windows #61099

Closed zware closed 10 years ago

zware commented 11 years ago
BPO 16895
Nosy @terryjreedy, @tjguk, @jkloth, @zware, @zooba
Files
  • Make.bat: Mimic 'make' on Windows
  • win_make.diff: Patch to add make.bat
  • win_configure-make.diff: Updated patch
  • win_configure-make.v2.diff: Version 2
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields: ```python assignee = 'https://github.com/zware' closed_at = created_at = labels = ['type-feature', 'OS-windows', 'build'] title = "Batch file to mimic 'make' on Windows" updated_at = user = 'https://github.com/zware' ``` bugs.python.org fields: ```python activity = actor = 'zach.ware' assignee = 'zach.ware' closed = True closed_date = closer = 'zach.ware' components = ['Build', 'Windows'] creation = creator = 'zach.ware' dependencies = [] files = ['28632', '29363', '29667', '30351'] hgrepos = [] issue_num = 16895 keywords = ['patch'] message_count = 21.0 messages = ['179369', '183843', '183868', '183879', '183886', '183890', '183903', '183904', '183907', '185934', '186201', '186318', '189875', '190241', '190242', '190244', '190247', '190248', '190249', '222889', '223060'] nosy_count = 7.0 nosy_names = ['terry.reedy', 'tim.golden', 'jkloth', 'BreamoreBoy', 'sbt', 'zach.ware', 'steve.dower'] pr_nums = [] priority = 'normal' resolution = 'postponed' stage = None status = 'closed' superseder = None type = 'enhancement' url = 'https://bugs.python.org/issue16895' versions = ['Python 3.5'] ```

    zware commented 11 years ago

    I find myself frustrated with some of the hoops I have to jump through and extra things I have to remember when I want to play around with Python on Windows. To make things a little easier, I've created a 'make.bat' file that can sit in the root of the repository and do a few of the more common tasks:

    make [build] [-64] -- Runs Tools\buildbot\build[-amd64].bat make clean [-64] -- Runs Tools\buildbot\clean[-amd64].bat make doc [args] -- Runs Doc\make.bat [args] with %PYTHON% set to 'py.exe -2' make patchcheck -- Runs Tools\scripts\patchcheck.py make test [test args] -- Runs the test suite with passed arguments

    It seems to work pretty well for me and I think it could be very useful to others, particularly new contributors. I'm sure there's plenty of room for improvement, though :)

    I don't know if this script might interfere or conflict with Cygwin or other tools; if so and it were decided to include this file, it could easily be renamed to 'wmake.bat' or some such.

    Hope this is useful enough to include, or at least useful to someone else :)

    terryjreedy commented 11 years ago

    +1

    Add make external[-64] to runs Tools/buildbot/external[-amd64].bat and explain this in the devguide (if not already) rather than buried in the opaque PCBuild/readme.

    Even after reading build.bat, I do not understand what is the different net effect of that versus just double clicking pcbuild.sln and building. So some explanation in the devguide would be needed.

    zware commented 11 years ago

    Here's an updated version in the form of a patch.

    Features include:

    Everything seems to work ok for me, but my testing platforms are limited to a single Win7 laptop, currently. This issue is also somewhat constrained by bpo-17202; it is supposed to work without any change to .hgeol, but that can't be guaranteed.

    e26428b1-70cf-4e9f-ae3c-9ef0478633fb commented 11 years ago

    +1

    To use Tools/builbot/*.bat doesn't the current directory have to be the main directory of the repository? Then I see no point in the "-C" argument: just set the correct directory automatically.

    I think make.bat should also support creation of non-debug builds. (Maybe have targets "release" and "debug"?)

    Tools/buildbot/build.bat already calls external.bat and clean.bat. This currently makes the "ready" target unnecessary. However, I don't think build should be calling clean.bat (or external.bat). Perhaps you should just inline the necessary parts of Tools/buildbot/build.bat.

    zware commented 11 years ago

    Richard Oudkerk added the comment:

    +1 Thank you :)

    To use Tools/builbot/*.bat doesn't the current directory have to be the main directory of the repository? Then I see no point in the "-C" argument: just set the correct directory automatically.

    That is true, and it does already automatically set the correct directory. The point of the -C option is for Doc/make.bat--from the root dir, call "make -C Doc html", and make cd's into Doc, and there calls "make.bat html", and the html docs are made. I didn't just hardwire the option to "-C Doc" even though that's the only place that can use -C, just in case there is someday another make.bat floating around the repository. The original version I posted just had a "doc" target that called Doc/make.bat with supplied arguments, but forcing the use of the UNIXy command makes documenting how to do things easier: just one command on all platforms.

    I think make.bat should also support creation of non-debug builds. (Maybe have targets "release" and "debug"?)

    I agree. I've had a thought for this; what about adding a "configure.bat" that takes a few of the applicable options from the UNIX "configure" script, and sets some environment variables (and/or writes out a "win-config.dat") for make.bat's use?

    Tools/buildbot/build*.bat already calls external.bat and clean.bat. This currently makes the "ready" target unnecessary.

    I thought the same initially, but I realized that it is useful for building using the VS GUI--clone the repo, call "make ready", then open pcbuild.sln in Visual Studio and you're ready to build everything there.

    However, I don't think build should be calling clean.bat (or external.bat). Perhaps you should just inline the necessary parts of Tools/buildbot/build*.bat.

    I've thought about this too, and am thinking that it would make a certain amount of sense to inline the entirety of Tools/buildbot. Make everything possible from just make.bat, and eventually migrate the buildbots to use it, and then the Tools/buildbot directory can go away alltogether. This would also make doing release vs debug easier.

    terryjreedy commented 11 years ago

    One must run external.bat if one is to subsequently build from the vs gui with *most* of the external dependencies. I think the command to run it should be 'external', not 'ready'. 'External' means get the external dependencies and anyone who has run external.bat will know what it means. 'Ready' does not mean anything in particular.

    Just curious: Why do the buildbots run make clean before re-compiling? I seems like lots of extra work to re-compile things that are up to date?

    What does running 'kill-python before re-building python do? I have not seen it mentioned in the in the devguide or pcbuild/readme.

    zware commented 11 years ago

    Terry J. Reedy added the comment:

    One must run external.bat if one is to subsequently build from the vs gui with *most* of the external dependencies. I think the command to run it should be 'external', not 'ready'. 'External' means get the external dependencies and anyone who has run external.bat will know what it means. 'Ready' does not mean anything in particular.

    Fair point. Thinking about it again after some sleep, I agree. Next version will revert to 'external' rather than 'ready'.

    Just curious: Why do the buildbots run make clean before re-compiling? I seems like lots of extra work to re-compile things that are up to date?

    I think the idea is to make sure everything is fresh, and avoid phantom problems from things that shouldn't cause problems, but do.

    What does running 'kill-python before re-building python do? I have not seen it mentioned in the in the devguide or pcbuild/readme.

    I believe it's what actually performs the cleaning process. Tools/buildbot/clean.bat builds kill_python.exe and then runs it.

    e26428b1-70cf-4e9f-ae3c-9ef0478633fb commented 11 years ago

    What does running 'kill-python before re-building python do? I have not seen it mentioned in the in the devguide or pcbuild/readme.

    It kills any currently running python(_d).exe processes. This is because Windows does not allow program or library files to be removed or overwritten while they are being used, potentially causing compilation failures.

    terryjreedy commented 11 years ago

    Thanks, clean and kill make sense for unattended build-bots. For interactive use, when 2 minutes rebuilding is a big deal, clean is a last resort. Let's just make sure we do not somehow suggest that it needs to be done routinely. Or to put it another way, do document that 'build' is intended for fail-safer unattended buildbots and not necessarily for interactive human use. And also that kill does what a human would normally do by closing windows on the task bar, or, as a last resort, with task manager.

    zware commented 11 years ago

    I was rather off about kill_python, wasn't I?

    Anyway, I've finally gotten another version put together that I'd like to get comments on. Major differences between this patch and the previous one, in no particular order:

    Several things have gone through an iteration or two since the last patch; I'll try to go through and add some comments on Rietveld myself.

    Since make.bat is now stand-alone, if it is accepted (and it is deemed desirable to do so), the Windows buildbots could be converted to use make.bat instead of Tools/buildbot/*, and that directory could actually be removed. This would also make it easy to set up a buildbot that tests a Release configuration of Python.

    All comments welcome :)

    e26428b1-70cf-4e9f-ae3c-9ef0478633fb commented 11 years ago

    You seem to end your subroutines (or whatever they are called) using "goto end" rather than "exit /b". Since popd follows the "end" label, does this mean that you get a popd after calling each subroutine? Is this intended and can it cause unmatched pushd/popd-s?

    (I am not familiar with writing batch files.)

    Also, I think 32 bit builds should be the default. Many people with 64 bit Windows are using Visual Studio Express which only has 32 bit support.

    zware commented 11 years ago

    You seem to end your subroutines (or whatever they are called) using "goto end" rather than "exit /b". Since popd follows the "end" label, does this mean that you get a popd after calling each subroutine?

    Yes.

    Is this intended and can it cause unmatched pushd/popd-s?

    It is intended and it can, but so far I haven't run into any trouble with it. However, I have realized that I haven't done any testing with some pushd's already on the stack... I'll do some more looking into that, and may have to implement either some form of callback labels or counting pushd's to properly popd.

    (I am not familiar with writing batch files.)

    Also, I think 32 bit builds should be the default. Many people with 64 bit Windows are using Visual Studio Express which only has 32 bit support.

    Fair point. I suppose I had been assuming that the error message from not finding the proper vcvars*.bat file would be an indication to pass '--without-64-bit' to configure.bat, but it is much nicer to just fall back to 32 bit. Also, it would be nice to just implode from the start if there is no compiler available. Next version of the patch will do both of these things in configure.bat; quit with an error message if %VS100COMNTOOLS% is not defined, and check for the existence of %VS100COMNTOOLS%\..\..\VC\bin\x86_amd64\vcvarsx86_amd64.bat before setting %PY_PLATFORM% to x64.

    Thanks for the comments.

    zware commented 11 years ago

    Here's a new version of the patch, which is a major rewrite. Among the changes:

    Everything works as expected for me, but I do only have a 32bit machine available to me currently. I hope to be able to test on a 64bit machine soon, but have no guarantees. For anyone testing this, I would suggest to run the command "prompt $+%PROMPT%" before testing; this will add a "+" to the beginning of your prompt for every dir on the pushd/popd stack. Neither configure.bat nor make.bat should ever remove dirs from that stack, and should not leave any extras when they're done (unless forcibly killed in the middle of running).

    e26428b1-70cf-4e9f-ae3c-9ef0478633fb commented 11 years ago

    I can't say I know enough about batch files to understand much of the code, but a few notes:

    Windows XP does not have the command "where" which you use -- Python 3.4 will still support XP.

    Except perhaps for looping I would prefer to get rid of the use of goto. The fact that some goto targets end in "exit /b ..." make it very confusing as to where "exit /b" will return control.

    The initial pushd is matched by various popd's which are scattered over hundreds of lines (including one in :usage). I think it would be better to keep matching pushd/popd reasonably close together. For instance, I think you could do something like

    ...
pushd "%~dp0"
call :main ...
popd
exit /b

:main
...
exit /b

    It would also be helpful if the end of the subroutines were marked with a comment like

    rem end :foo
    briancurtin commented 11 years ago

    Can't this just be a Python script?

    e26428b1-70cf-4e9f-ae3c-9ef0478633fb commented 11 years ago

    Can't this just be a Python script?

    That would cause bootstrap issues for people who do not already have python installed.

    zware commented 11 years ago

    Richard Oudkerk added the comment:

    I can't say I know enough about batch files to understand much of the code, but a few notes:

    Windows XP does not have the command "where" which you use -- Python 3.4 will still support XP.

    Oh, that is an issue. I don't have an XP machine to test on anymore, thank you for that catch. I found a workaround on StackOverflow that looks short enough to be usable instead.

    Except perhaps for looping I would prefer to get rid of the use of goto. The fact that some goto targets end in "exit /b ..." make it very confusing as to where "exit /b" will return control.

    The only goto's that are not part of loops are now one near the beginning for the -C option, in the target validation routine to show the usage message and die, and in a couple of routines which use "goto no-configure" to show a common message and die. I'd rather not have to copy that message every place it is used, but that is an option. Would just adding comments explaining where execution is going and whether it is coming back be sufficient?

    The initial pushd is matched by various popd's which are scattered over hundreds of lines (including one in :usage). I think it would be better to keep matching pushd/popd reasonably close together. For instance, I think you could do something like

    ...
pushd "%~dp0"
call :main ...
popd
exit /b

:main
...
exit /b

    Fair enough, I can change that. I tried to keep the matches to the initial pushd to a minimum, but perhaps there are a couple more I can eliminate.

    It would also be helpful if the end of the subroutines were marked with a comment like

    rem end :foo

    Easy enough, consider it done :)

    briancurtin commented 11 years ago

    Don't we already require an existing Python to build some of the third-party stuff, e.g., OpenSSL?

    I don't think the bootstrapping issue holds that much weight. Adding some huge batch script that maybe one or two people even know how to modify is a much higher cost than just having someone install Python.

    zware commented 11 years ago

    Brian Curtin added the comment:

    Don't we already require an existing Python to build some of the third-party stuff, e.g., OpenSSL?

    Only for building a 64-bit Python on 32-bit Windows. Otherwise, OpenSSL is built using the just-built interpreter.

    I don't think the bootstrapping issue holds that much weight. Adding some huge batch script that maybe one or two people even know how to modify is a much higher cost than just having someone install Python.

    Fair enough, but even when Python is installed, there's still the issue of whether it will be findable on PATH, whether .py is in PATHEXT, what version of Python is installed (and which one is on PATH), etc. etc...

    However, you've made me think; perhaps configure.bat could build a minimal Python that can then be used for a make.py. I'll do some experimenting and see what I can come up with.

    83d2e70e-e599-4a04-b820-3814bbdb9bef commented 10 years ago

    Is this still relevant or has it been overtaken by other work from Zach or Steve?

    zware commented 10 years ago

    Still relevant, but the current patch is overkill (Brian was right, it's just way more batch than is healthy to check in). Things are changing rapidly, though; I'll close it as "postponed" for now, with the expectation of reopening someday.