pyreadline bug

[hutcho]

Firstly, thank you for making these tools. They are very useful!

Environment: Python 3.12.7 Windows 10 latest updates.

Steps: Downloaded appendfilename using docs instructions here: https://karl-voit.at/demo-filetags-setup/ pip install pypiwin32 integratethis filetags date2name appendfilename

Install appendfilename using integratethis appendfilename --confirm

Right click a file and select appendfilename causes error:

C:\Users\USERNAME\AppData\Local\Programs\Python\Python312\Lib\site-packages\appendfilename\__init__.py:29: SyntaxWarning: invalid escape sequence '\d'
  WITHTIME_AND_SECONDS_PATTERN  = re.compile('^(\d{4,4}-[01]\d-[0123]\d(([T :_-])([012]\d)([:.-])([012345]\d)(([:.-])([012345]\d))?)?)[- _.](.+)')
C:\Users\USERNAME\AppData\Local\Programs\Python\Python312\Lib\site-packages\appendfilename\__init__.py:63: SyntaxWarning: invalid escape sequence '\.'
  FILE_WITH_EXTENSION_REGEX = re.compile("(.*?)(( -- .*)?(\.\w+?)?)$")
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python312\Scripts\appendfilename.exe\__main__.py", line 4, in <module>
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python312\Lib\site-packages\appendfilename\__init__.py", line 21, in <module>
    import readline  # for raw_input() reading from stdin
    ^^^^^^^^^^^^^^^
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python312\Lib\site-packages\readline.py", line 34, in <module>
    rl = Readline()
         ^^^^^^^^^^
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python312\Lib\site-packages\pyreadline\rlmain.py", line 422, in __init__
    BaseReadline.__init__(self)
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python312\Lib\site-packages\pyreadline\rlmain.py", line 62, in __init__
    mode.init_editing_mode(None)
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python312\Lib\site-packages\pyreadline\modes\emacs.py", line 633, in init_editing_mode
    self._bind_key('space',       self.self_insert)
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python312\Lib\site-packages\pyreadline\modes\basemode.py", line 162, in _bind_key
    if not callable(func):
           ^^^^^^^^^^^^^^
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python312\Lib\site-packages\pyreadline\py3k_compat.py", line 8, in callable
    return isinstance(x, collections.Callable)
                         ^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'collections' has no attribute 'Callable'
Hit ENTER to continue...

I think the problem is that appendfilename causes pyreadline to be installed, and uses pyreadline. The problem is that pyreadline doesn't work in the newest version of python. If I go into the __init__.py of appendfilename, and remove all lines referencing pyreadline, the program runs and does what I expect, but it still prints SyntaxWarnings:

C:\Users\USERNAME\AppData\Local\Programs\Python\Python312\Lib\site-packages\appendfilename\__init__.py:29: SyntaxWarning: invalid escape sequence '\d'
  WITHTIME_AND_SECONDS_PATTERN  = re.compile('^(\d{4,4}-[01]\d-[0123]\d(([T :_-])([012]\d)([:.-])([012345]\d)(([:.-])([012345]\d))?)?)[- _.](.+)')
C:\Users\USERNAME\AppData\Local\Programs\Python\Python312\Lib\site-packages\appendfilename\__init__.py:63: SyntaxWarning: invalid escape sequence '\.'
  FILE_WITH_EXTENSION_REGEX = re.compile("(.*?)(( -- .*)?(\.\w+?)?)$")

[novoid]

Hi,

I'm sorry, I don't have access to a Windows machine.

To analysis the issue, I would like you to use the cmd.exe environment, come up with the whole command including the --verbose switch and post the command + its output here.

If it is some Windows-specific but related to pyreadline, I don't know what I can do without help of somebody who is familiar with the package implications and Windows. pyreadline is necessary for anything where the user types something in interactive mode.

[novoid]

https://stackoverflow.com/questions/74034049/python-on-windows-starts-with-attributeerror-module-collections-has-no-attri suggests that you would need to make sure that pyreadline3 is the only Python readline version installed. Did you try that one? If that works, I may need to update the install instructions.

[nbehrnd]

@hutcho Try a partial reinstallation. Check first the additional modules installed; depending on the local setup, this runs for instance by

pip list

or

python3 -m pip list

You should see pyreadline because in appendfilename 2023.7.15.1 (as distributed on PyPI/via pip) lists this in file requirements.txt as a dependency to resolve. Uninstall pyreadline by

python -m pip uninstall pyreadline

In case you equally found pyreadline3 (which is for instance a dependency of filetags), remove both, i.e. run

python -m pip uninstall pyreadline pyreadline3

instead. After clearance, install only pyreadline3, i.e.

python -m pip install pyreadline3

and you should be set (could be you need to restart Windows though). Which worked for me in an instance of Windows 10 with Python 3.12.4 (from python.org) and pip 24.2 and likely works similar on the (portable) WinPython, too.

An answer a little bit more detailed: pyreadline3 enjoys continued maintenance and is based on pyreadline. However, pyreadline didn't see an update to PyPI since 2015.

There is some overlap how to access the functionality by pyreadline3 and the one by (legacy?) pyreadline. This is comfortable for utilities adopting the newer one because it requires only small (if any) edits of their sources. On the other hand this can yield ambiguity in the namespace, can trouble the Python interpreter ("which one of the two tools of the same name should be used?"). Hence arguably the best approach is i) to uninstall both pyreadline and pyreadline3 altogether, and subsequently ii) to install pyreadline3 alone, from a clean slate.

@novoid Indeed, in lines of the above, a move appendfilename to use pyreadline3 would be an advantage because of then greater portability. The reason I didn't recognize this in 2022 include deploying a different setup than the OP (Linux Debian as operating system with a "frozen" virtual environment specific to appendfilename, etc.).

[novoid]

@nbehrnd currently, https://karl-voit.at/demo-filetags-setup/ lists:

pip install pypiwin32 integratethis filetags date2name appendfilename

What should be the new line? Just replace pypiwin32 with pyreadline3? Is there nothing in pypiwin32 that is necessary?

[hutcho]

@nbehrnd That works! Thankyou! The weird thing that I didn't expect was that I had to uninstall pyreadline3 just like your instructions @nbehrnd, then reinstall pyreadline3. I had installed pyreadline3 in advance of installing appendfilename. Then I uninstalled pyreadline and still got errors trying to run appendfilename. The trick was to also uninstall pyreadline3 (then reinstall pyreadline3). Unexpected! My findings below. I tried all the below steps also using python -m pip and results were the same. I use uv below, since uv pip tree output looks nice. Here's my steps:

This doesn't work and gives a ModuleNotFoundError: No module named 'readline' error:

uv venv
uv pip install pyreadline3
uv pip install appendfilename
uv pip uninstall pyreadline
appendfilename -> ModuleNotFoundError: No module named 'readline' 

This DOES work (uninstall pyreadline3 here just in case it was installed previously somehow in the past):

uv pip install appendfilename
uv pip uninstall pyreadline pyreadline3
uv pip install pyreadline3
appendfilename WORKS

The error I experienced due to pyreadline3 not being installed properly and pyreadline being installed instead is now fixed! This was the error that is now fixed for me:

AttributeError: module 'collections' has no attribute 'Callable'

However, now there is a SyntaxWarning when I launch appendfilename.

appendfilename\__init__.py:29: SyntaxWarning: invalid escape sequence '\d'
appendfilename\__init__.py:63: SyntaxWarning: invalid escape sequence '\.'

Just for more info, here's the package lists that I have after each step:

uv venv
uv pip install appendfilename
uv pip tree

appendfilename v2022.1.4.1
└── pyreadline v2.1

rmdir .venv /S
uv venv
uv pip install appendfilename
uv pip uninstall pyreadline pyreadline3
uv pip install pyreadline3
uv pip tree

appendfilename v2022.1.4.1
pyreadline3 v3.5.4

I think if appendfilename dependency was changed to pyreadline3 instead of pyreadline, this would fix the AttributeError.

[hutcho]

As for the SyntaxWarnings, you might be able to just change the lines to raw strings, but I don't know how to test this change. I ran pytest but every test failed :|

re.compile(r'...')

[hutcho]

Can I try help out with these updates? What is your goal for this project in terms of python version support? I wouldn't mind helping to modernize and help keep things up to date? I only have access to Win10 at the moment, but could also obtain other OSes depending on your vision for support/testing?

[nbehrnd]

@novoid The command

pip install pypiwin32 integratethis filetags date2name appendfilename

was correct, and still is correct, too. The problem is appendfilename as currently available via PyPI and pip has a requirements.txt file requesting the elder pyreadline. PR19 [https://github.com/novoid/appendfilename/pull/19] thus substitutes the old by the new form.

[nbehrnd]

@hutcho In my case of an instance of Windows, the similarity of pyreadline and pyreadline3 to the Python interpreter (and hence one roadblock) can be observed by i) installation of pyreadline and subsequent uninstallation of pyreadline with pip (or your Python module/package manager of preference), and later ii) installation of pyreadline3 and its uninstallation. In both cases, just prior to actually removing the installation, cmd.exe revealed the identical path to

c:\users\ACCOUNTNAME\appdata\local\programs\python\python312\lib\sitepackages\readline.py

As for the reports by pylint for instance on raw strings, I agree with you. I saw this in move2archive (https://github.com/nbehrnd/move2archive/commit/b26191c5eacbe010540575b5858385cff3473a10) but didn't continue (a.k.a. filed a PR [yet]) on this and other points because it didn't affect the functionality -- as far I was able to see, back then.

[nbehrnd]

@hutcho Running pytest in a virtual environment set up with manually installed pytest (version 8.3.3) and pyreadline3 installed via the revised requirements.txt file PR18 is about, there are 1080 tests launched. 144 failed, 936 passed, and 10 warnings are reported. I agree with you, passing about 86% isn't satisfying. It will take a little bit to identify the cause (because in 2022 it worked just fine) and provide an improvement.

In similar tune to the other tools (date2name, filetags, etc) usage of Python is both for the simplicity as well as portability of implementation for every platform supporting Python (Windows, Linux, MacOS, etc.).

[novoid]

@nbehrnd I just switched all of my projects to pyreadline3 for now and I'm closing this issue as the issue does seem to be resolved for the OP.

[novoid]

Can I try help out with these updates? What is your goal for this project in terms of python version support? I wouldn't mind helping to modernize and help keep things up to date? I only have access to Win10 at the moment, but could also obtain other OSes depending on your vision for support/testing?

Well, the tools "works for me"(TM) and I don't plan to invest much time as long as my requirements don't change.

If you've got ideas for improvement, I'm open for that as long as my personal use-cases don't change much.

I don't have the Python knowledge nor the time to modernize my source.

If you would like to be a co-maintainer, I'm also very open about that since you've shown to be a really great help around here!

I added you to this repo. Drop me a line if you also want to join other repositories of mine.