Failures when python is python3

[psmit]

At this moment kaldi assumes that python is actually python2. Most distro's have this indeed configured this way, but I have at least 2 cases that this doesn't hold

1) Arch linux 2) When using a module system and both python2 and python3 versions are loaded, sometimes this causes python to be linked to python2.

I think the best solution would be to write explicitly #!/usr/bin/env python2 instead of #!/usr/bin/env python3. Afaik this would work in all situations correctly. Would you be open to a PR that implements this change?

[jtrmal]

I think that could work. https://www.python.org/dev/peps/pep-0394/

I'm not sure, however, if all distributions and other OS (Mac?) do support python2 link. Also, according to the PEP, it's actually Arch that is causing the incompatibility, so I don't think we should spend too much of our time trying to deal with that.

[dresen]

I'm in the middle of installing Kaldi on a 2015 macbook pro and the solution with bash alias'ing python to python2.7 doesn't work - extras/check_dependencies.sh still reports version 3.*

[jtrmal]

@dresen -- I think that's because bash by default does not expand aliases when in non-interactive mode. I think shopt -s expand_aliases might help

[jtrmal]

btw, when I'm thinking about it -- expanding the aliases could have a lot of negative results. Perhaps it would be better to define a shell function

function python() {
  python2 "$@"
}

or something like that -- I don't have time to test it out now.

[danpovey]

python2 is not defined on my mac. I believe the Python community has accepted that python3 is essentially a separate language for compatibility purposes, and that python2 will always be invoked as "python", and python3 as "python3".

In my opinion Arch linux has painted itself into a corner by this choice, because it's not compatible with the rest of the world.

Dan

On Fri, Apr 28, 2017 at 10:58 AM, jtrmal notifications@github.com wrote:

btw, when I'm thinking about it -- expanding the aliases could have a lot of negative results. Perhaps it would be better to define a shell function

function python() { python2 "$@" }

or something like that -- I don't have time to test it out now.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/kaldi-asr/kaldi/issues/1592#issuecomment-298021391, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJVuz5a3S3lDr2kK_qDuTjC5y1v_a2iks5r0f6zgaJpZM4NLXjo .

[psmit]

I disagree Dan. In the linked PEP is written that this choice will be revisited every couple years. Also, this PEP came only after Arch changed it's default python. There was no python community opinion before they did that; the community saw the effects and created this PEP as a reaction.

Anyway, the advise of this PEP is to use python2 in the shebang. Could you maybe look why your mac does not have python2 defined; the PEP indicates that the mac installers have been updated to provide that.

[danpovey]

Hm.

The PEP does seem to say that python2 is the preferred way to invoke python. However, since my mac doesn't have python2 installed, and I think probably a bunch of people have the same issue (regardless of the exact reason), so I'm inclined to wait a bit before changing all the python's to python2.

Dan

On Mon, May 1, 2017 at 12:34 AM, Peter Smit notifications@github.com wrote:

I disagree Dan. In the linked PEP is written that this choice will be revisited every couple years. Also, this PEP came only after Arch changed it's default python. There was no python community opinion before they did that; the community saw the effects and created this PEP as a reaction.

Anyway, the advise of this PEP is to use python2 in the shebang. Could you maybe look why your mac does not have python2 defined; the PEP indicates that the mac installers have been updated to provide that.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kaldi-asr/kaldi/issues/1592#issuecomment-298284680, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJVuwrsAyUdeOVeogjdYCFrTfhFWWZ1ks5r1WDjgaJpZM4NLXjo .

[psmit]

Fair point about the mac. On the other hand, in our group we still have a problem if some loads python modules in the 'wrong' order or e.g. enables a python3 virtual environment (for their own scripts).

Would it be a suggestion to reverse the warning; giving people the command how to create a python2 symlink if it doesn't exist? Or add a symlink automatically in the tools path somewhere if 'python2' doesn't exist?

[danpovey]

Hm, that is a reasonable idea. But it's @jtrmal who understands best how the env.sh works, so it's a question of when he has time.

(However, I'd say that it probably a bad idea for your group to purposely set things up so you need to invoke python3 as just 'python', as the pep3 standard definitely doesn't recommend that).

On Tue, May 2, 2017 at 6:16 AM, Peter Smit notifications@github.com wrote:

Fair point about the mac. On the other hand, in our group we still have a problem if some loads python modules in the 'wrong' order or e.g. enables a python3 virtual environment (for their own scripts).

Would it be a suggestion to reverse the warning; giving people the command how to create a python2 symlink if it doesn't exist? Or add a symlink automatically in the tools path somewhere if 'python2' doesn't exist?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kaldi-asr/kaldi/issues/1592#issuecomment-298593439, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJVu41k6hzyG-KVt4-QiwhBwfvXwlF1ks5r1wKPgaJpZM4NLXjo .

[jtrmal]

What could work is create a correct symlink somewhere in, say, tools and setting PATH so that this symlink will be preferred. That might be more reliable than aliases or functions. I just worry if we want to add another layer of indirection. Perhaps you could write a PR in that direction and test it on your setup?

On the users -- I agree with Dan, mostly -- if the user or users purposely maps/aliases python to python3, I don't actually think I want to care. y.

On Tue, May 2, 2017 at 1:25 PM, Daniel Povey notifications@github.com wrote:

Hm, that is a reasonable idea. But it's @jtrmal who understands best how the env.sh works, so it's a question of when he has time.

(However, I'd say that it probably a bad idea for your group to purposely set things up so you need to invoke python3 as just 'python', as the pep3 standard definitely doesn't recommend that).

On Tue, May 2, 2017 at 6:16 AM, Peter Smit notifications@github.com wrote:

Fair point about the mac. On the other hand, in our group we still have a problem if some loads python modules in the 'wrong' order or e.g. enables a python3 virtual environment (for their own scripts).

Would it be a suggestion to reverse the warning; giving people the command how to create a python2 symlink if it doesn't exist? Or add a symlink automatically in the tools path somewhere if 'python2' doesn't exist?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kaldi-asr/kaldi/issues/1592#issuecomment-298593439, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJVu41k6hzyG-KVt4- QiwhBwfvXwlF1ks5r1wKPgaJpZM4NLXjo .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kaldi-asr/kaldi/issues/1592#issuecomment-298703141, or mute the thread https://github.com/notifications/unsubscribe-auth/AKisX0G0AI1OAiAMu6LjUK4tyhhpJ0axks5r12cCgaJpZM4NLXjo .

[jtrmal]

BTW, I'm ok with changing the shebang to python2 if that helps in making things more robust but so far it looks like opening another can of issues without actually solving anything -- instead of archlinux, the problems will (probably) start popping out somewhere else. y.

On Tue, May 2, 2017 at 2:39 PM, Jan Trmal jtrmal@gmail.com wrote:

What could work is create a correct symlink somewhere in, say, tools and setting PATH so that this symlink will be preferred. That might be more reliable than aliases or functions. I just worry if we want to add another layer of indirection. Perhaps you could write a PR in that direction and test it on your setup?

On the users -- I agree with Dan, mostly -- if the user or users purposely maps/aliases python to python3, I don't actually think I want to care. y.

On Tue, May 2, 2017 at 1:25 PM, Daniel Povey notifications@github.com wrote:

Hm, that is a reasonable idea. But it's @jtrmal who understands best how the env.sh works, so it's a question of when he has time.

(However, I'd say that it probably a bad idea for your group to purposely set things up so you need to invoke python3 as just 'python', as the pep3 standard definitely doesn't recommend that).

On Tue, May 2, 2017 at 6:16 AM, Peter Smit notifications@github.com wrote:

Fair point about the mac. On the other hand, in our group we still have a problem if some loads python modules in the 'wrong' order or e.g. enables a python3 virtual environment (for their own scripts).

Would it be a suggestion to reverse the warning; giving people the command how to create a python2 symlink if it doesn't exist? Or add a symlink automatically in the tools path somewhere if 'python2' doesn't exist?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://github.com/kaldi-asr/kaldi/issues/1592#issuecomment-298593439 , or mute the thread https://github.com/notifications/unsubscribe-auth/ ADJVu41k6hzyG-KVt4-QiwhBwfvXwlF1ks5r1wKPgaJpZM4NLXjo .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kaldi-asr/kaldi/issues/1592#issuecomment-298703141, or mute the thread https://github.com/notifications/unsubscribe-auth/AKisX0G0AI1OAiAMu6LjUK4tyhhpJ0axks5r12cCgaJpZM4NLXjo .

[danpovey]

I'd be OK to wait until we hear more reports of problems, maybe few people use arch linux anyway. Peter Smit seemed to be saying that people in his group were purposely mapping python to python3, it wasn't even an arch linux issue. The fact that they were doing this intentionally makes me rather less sympathetic to their situation than I was before.

On Tue, May 2, 2017 at 2:49 PM, jtrmal notifications@github.com wrote:

BTW, I'm ok with changing the shebang to python2 if that helps in making things more robust but so far it looks like opening another can of issues without actually solving anything -- instead of archlinux, the problems will (probably) start popping out somewhere else. y.

On Tue, May 2, 2017 at 2:39 PM, Jan Trmal jtrmal@gmail.com wrote:

What could work is create a correct symlink somewhere in, say, tools and setting PATH so that this symlink will be preferred. That might be more reliable than aliases or functions. I just worry if we want to add another layer of indirection. Perhaps you could write a PR in that direction and test it on your setup?

On the users -- I agree with Dan, mostly -- if the user or users purposely maps/aliases python to python3, I don't actually think I want to care. y.

On Tue, May 2, 2017 at 1:25 PM, Daniel Povey notifications@github.com wrote:

Hm, that is a reasonable idea. But it's @jtrmal who understands best how the env.sh works, so it's a question of when he has time.

(However, I'd say that it probably a bad idea for your group to purposely set things up so you need to invoke python3 as just 'python', as the pep3 standard definitely doesn't recommend that).

On Tue, May 2, 2017 at 6:16 AM, Peter Smit notifications@github.com wrote:

Fair point about the mac. On the other hand, in our group we still have a problem if some loads python modules in the 'wrong' order or e.g. enables a python3 virtual environment (for their own scripts).

Would it be a suggestion to reverse the warning; giving people the command how to create a python2 symlink if it doesn't exist? Or add a symlink automatically in the tools path somewhere if 'python2' doesn't exist?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://github.com/kaldi-asr/kaldi/issues/1592# issuecomment-298593439 , or mute the thread https://github.com/notifications/unsubscribe-auth/ ADJVu41k6hzyG-KVt4-QiwhBwfvXwlF1ks5r1wKPgaJpZM4NLXjo .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/kaldi-asr/kaldi/issues/1592#issuecomment-298703141 , or mute the thread https://github.com/notifications/unsubscribe-auth/ AKisX0G0AI1OAiAMu6LjUK4tyhhpJ0axks5r12cCgaJpZM4NLXjo .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kaldi-asr/kaldi/issues/1592#issuecomment-298726269, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJVu2rX8qLF_IIwr_sAXFP51wysBkCLks5r13rBgaJpZM4NLXjo .

[psmit]

We do not purposely map python to python3, but that symlink is created by different installations. For example when anaconda3 is installed, or when a python3 virtual environment is activated.

In our group it is like this: The cluster has a version of python2 installed through rpm's, but the admins advise to use a different python, one that they have installed and which can be put on the path using the module system https://www.tacc.utexas.edu/research-development/tacc-projects/lmod . The reason for this is that many people desire up-to-date versions of all kind of important python packages and the Redhad rpm's aren't updated so often. If you want to use python3 at all you have to load their python3 installation and that will include a symlink of python to python3.

The reason this is the case, is that when anaconda3 is installed (the python installation they provide), it will also install a python symlink at its install location. This is normally not a problem as those users will most likely want to run only python3 specific code.

No-one actually counted on as complicated installations as speech recognition systems. We do have some python3 code of our own and we want to run it interleaved with kaldi-scripts. Meaning we do have to load the python3 module (and also the python2 module). If we load them in the right order, with the python2-python somewhere earlier on the path everything is peachy; if not we have python linking to python3.

To summarize:

• I think there a valid cases that python points to python3. The linked PEP is anyway mostly geared towards distributions and the symlinks I mainly talk about now are regarding python installers or virtual environments
• The PEP recommendation is to make scripts have a pythonx shebang if it only runs on one version of python
• There seems to be an expectation that the python2 symlink should be present on all distributions, but at least on Dan's mac it seems there is not.

I think it would be beneficial to make the change I proposed; things should be working on all platforms afterwards. I can make a PR beginning next week (I'm on holiday the rest of the week) and test it at least on our systems. Maybe Dan can then test it on his mac to see if things work. Is that a good plan?

[danpovey]

The plan sounds OK to me.

On Tue, May 2, 2017 at 11:27 PM, Peter Smit notifications@github.com wrote:

We do not purposely map python to python3, but that symlink is created by different installations. For example when anaconda3 is installed, or when a python3 virtual environment is activated.

In our group it is like this: The cluster has a version of python2 installed through rpm's, but the admins advise to use a different python, one that they have installed and which can be put on the path using the module system https://www.tacc.utexas.edu/research-development/tacc-projects/lmod . The reason for this is that many people desire up-to-date versions of all kind of important python packages and the Redhad rpm's aren't updated so often. If you want to use python3 at all you have to load their python3 installation and that will include a symlink of python to python3.

The reason this is the case, is that when anaconda3 is installed (the python installation they provide), it will also install a python symlink at its install location. This is normally not a problem as those users will most likely want to run only python3 specific code.

No-one actually counted on as complicated installations as speech recognition systems. We do have some python3 code of our own and we want to run it interleaved with kaldi-scripts. Meaning we do have to load the python3 module (and also the python2 module). If we load them in the right order, with the python2-python somewhere earlier on the path everything is peachy; if not we have python linking to python3.

To summarize:

• I think there a valid cases that python points to python3. The linked PEP is anyway mostly geared towards distributions and the symlinks I mainly talk about now are regarding python installers or virtual environments
• The PEP recommendation is to make scripts have a pythonx shebang if it only runs on one version of python
• There seems to be an expectation that the python2 symlink should be present on all distributions, but at least on Dan's mac it seems there is not.

I think it would be beneficial to make the change I proposed; things should be working on all platforms afterwards. I can make a PR beginning next week (I'm on holiday the rest of the week) and test it at least on our systems. Maybe Dan can then test it on his mac to see if things work. Is that a good plan?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kaldi-asr/kaldi/issues/1592#issuecomment-298815684, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJVu-49amkfb1vJUsj-vThCzGVaoeopks5r1_Q8gaJpZM4NLXjo .

[psmit]

Just a small note, I have been sick in the past 2 weeks and now I'm first swamped with some other work stuff. I'll hope to get to making the PR in one to two weeks from now.

[jprobichaud]

In case this could help, I would like to point to this AUR package (ArchLinux Repository): https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=kaldi

The author of that build has created some build prepare steps that address semi/automatically setting python shebang lines to "python2" along with few other manips. Maybe this can serve as a source of inspiration?

For a toolkit of the scope of Kaldi, it's probably a good thing to wrap a bit more tightly the tools used anyways (while I admit it could be less convenient in general to have to do $PYTHON tools/script_name.py instead of simply tools/script_name.py

[danpovey]

It's easy to change the shebangs. I'm waiting for a time when I feel the systems when "python" points to "python3" outnumber the systems where "python2" is not defined. Right now on my mac, which is running 10.12 (not the latest 10.13 but I'll install it when it pushes me to install it), python2 is still not defined.

I'm very annoyed at how the Python team handled this transition. What they are doing makes about as much sense as redefining "awk" to invoke perl.

Dan

On Tue, Oct 31, 2017 at 8:39 AM, Jp notifications@github.com wrote:

In case this could help, I would like to point to this AUR package (ArchLinux Repository): https://aur.archlinux.org/ cgit/aur.git/tree/PKGBUILD?h=kaldi

The author of that build has created some build prepare steps that address semi/automatically setting python shebang lines to "python2" along with few other manips. Maybe this can serve as a source of inspiration?

For a toolkit of the scope of Kaldi, it's probably a good thing to wrap a bit more tightly the tools used anyways (while I admit it could be less convenient in general to have to do $PYTHON tools/script_name.py instead of simply tools/script_name.py

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kaldi-asr/kaldi/issues/1592#issuecomment-340749697, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJVu1Dp-3bii0Tv0l38_AM5vgIM_L1Fks5sxxT1gaJpZM4NLXjo .

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed by a bot strictly because of inactivity. This does not mean that we think that this issue is not important! If you believe it has been closed hastily, add a comment to the issue and mention @kkm000, and I'll gladly reopen it.

[bonrybon]

currently working in MacOS 12.5 and Im having the same issue for running the makecommand in kaldi/tools

extras/check_dependencies.sh extras/check_dependencies.sh: python2.7 is not installed extras/check_dependencies.sh: The following prerequisites are missing; install them first: python2.7 make: *** [check_required_programs] Error 1

how should I proceed on this?

[jtrmal]

you could comment it out in the script. I will think about what could be done, but even brew does not carry python2.7 anymore. y.

On Sun, Oct 2, 2022 at 1:15 AM Bonry Dorado, ECE, CLF, SAA < @.***> wrote:

currently working in MacOS 12.5 and Im having the same issue for running the make command in kaldi/tools

extras/check_dependencies.sh extras/check_dependencies.sh: python2.7 is not installed extras/check_dependencies.sh: The following prerequisites are missing; install them first: python2.7 make: *** [check_required_programs] Error 1

how should I proceed on this?

— Reply to this email directly, view it on GitHub https://github.com/kaldi-asr/kaldi/issues/1592#issuecomment-1264554300, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACUKYX3FOS2PCQDRXTVXXLTWBEK7BANCNFSM4DJNPDUA . You are receiving this because you were mentioned.Message ID: @.***>