Python 3.10 alpha 7 introduces new changes to pathlib

alexdelorenzo commented 3 years ago

The following works on alpha 6 and below, but alpha 7 introduces this error:

In [1]: from aiopath import AsyncPath                                                                                                                                  

In [2]: home = await AsyncPath.home()                                                                                                                                  
---------------------------------------------------------------------------                                                                                            
TypeError                                 Traceback (most recent call last)                                                                                            
<ipython-input-2-ef16332cd56b> in <module>                                                                                                                             
----> 1 home = await AsyncPath.home()                                                                                                                                  

~/.pyenv/versions/3.10-dev/lib/python3.10/site-packages/aiopath/path.py in home(cls)                                                                                   
    391     returned by os.path.expanduser('~')).                                                                                                                      
    392     """                                                                                                                                                        
--> 393     coro = cls()._flavour.gethomedir(None)                                                                                                                     
    394     homedir: str = await coro                                                                                                                                  
    395                                                                                                                                                                

~/.pyenv/versions/3.10-dev/lib/python3.10/site-packages/aiopath/path.py in __new__(cls, *args, **kwargs)                                                               
    133       cls = AsyncWindowsPath if os.name == 'nt' else AsyncPosixPath                                                                                            
    134 
--> 135     self = cls._from_parts(args, init=False)
    136 
    137     if not self._flavour.is_supported:

TypeError: PurePath._from_parts() got an unexpected keyword argument 'init'

alexdelorenzo commented 3 years ago

The specific error in this ticket is fixed in this branch: https://github.com/alexdelorenzo/aiopath/tree/Python-3.10

Versions 0.6.x on PyPI work with Python 3.10a7: python3 -m install aiopath==0.6.1

alexdelorenzo commented 3 years ago

Using the latest branch of pytest (via https://github.com/pytest-dev/pytest) with 3.10 support, it appears that the coroutine method AsyncPath.resolve() is broken.

______________________________________________________________________________ test_home ______________________________________________________________________________                                                                           

    @pytest.mark.asyncio                                                                                                                                                                                                                          
    async def test_home():                                                                                                                                                                                                                        
      home = Path.home()                                                                                                                                                                                                                          
      ahome = await AsyncPath.home()                                                                                                                                                                                                              

>     await _test_is(home, ahome)                                                                                                                                                                                                                 

tests.py:91:                                                                                                                                                                                                                                      
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _                                                                           
tests.py:67: in _test_is                                                                                                                                                                                                                          
    await _test_is(path.parent, apath.parent, test_parent=False)                                                                                                                                                                                  
tests.py:74: in _test_is                                                                                                                                                                                                                          
    await apath.resolve(),                                                                                                                                                                                                                        
aiopath/path.py:491: in resolve                                                                                                                                                                                                                   
    s: Optional[str] = await self._flavour.resolve(self, strict=strict)                                                                                                                                                                           
aiopath/flavours.py:90: in resolve                                                                                                                                                                                                                
    result = await _resolve(base, str(path))                                                                                                                                                                                                      
aiopath/flavours.py:73: in _resolve                                                                                                                                                                                                               
    target = await accessor.readlink(newpath)                                                                                                                                                                                                     
../../../.pyenv/versions/3.10-dev/lib/python3.10/site-packages/aiofiles/os.py:13: in run                                                                                                                                                          
    return await loop.run_in_executor(executor, pfunc)                                                                                                                                                                                            
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _                                                                           

self = None                                                                                                                                                                                                                                       

    def run(self):                                                                                                                                                                                                                                
        if not self.future.set_running_or_notify_cancel():                                                                                                                                                                                        
            return                                                                                                                                                                                                                                

        try:                                                                                                                                                                                                                                      
>           result = self.fn(*self.args, **self.kwargs)                                                                                                                                                                                           
E           TypeError: readlink() takes exactly 1 positional argument (2 given)                                                                                                                                                                   

../../../.pyenv/versions/3.10-dev/lib/python3.10/concurrent/futures/thread.py:52: TypeError

alexdelorenzo commented 3 years ago

Here is a list of changes to the upstream pathlib package from the 3.10 alpha 7 stdlib:

Full list of changes here: https://github.com/python/cpython/commits/master/Lib/pathlib.py

These changes will need to be ported to aiopath's Python-3.10 branch.

alexdelorenzo commented 3 years ago

Just some notes for myself:

I started this project as a fork of pathlib for an opportunity to port synchronous code to asynchronous Python and to work with some standard library code. I wanted to keep the same class hierarchy and API to address use cases that might consume those interfaces, even if I personally don't plan on using them.

It might be worth profiling trio's approach of just wrapping synchronous I/O pathlib methods with decorators that submit the methods to a threadpool in the background: https://github.com/python-trio/trio/blob/6754c74eacfad9cc5c92d5c24727a2f3b620624e/trio/_path.py

That way I'm not beholden to upstream maintainers' changes in the stdlib, and as long as the API stays the same, aiopath won't have to change much if at all between changes to the stdlib.

The downside is punting some CPU-bound code to the threadpool, which, all in all, probably has an insignificant overhead.

It can also introduce problems like these:

However, Barney Gale's updates to pathlib that appeared in 3.10 alpha 7 are stepping stones on a roadmap towards an extensible pathlib: https://discuss.python.org/t/make-pathlib-extensible/3428/

In a world with an extensible pathlib, it makes sense for aiopath to have the current class hierarchy it has (with AsyncPath subclassing AsyncPurePath etc). I'd also assume in such a world, pathlib would also have a stable interface to support consumers like aiopath.

barneygale commented 3 years ago

I'm sorry my changes have broken things! As always with extending private APIs in the standard library, it can be a bit of a minefield. As you mentioned, I'm working towards a pathlib.AbstractPath class which supports subclassing. I'm hoping this can land in Python 3.11, all being well :)

alexdelorenzo / aiopath

Python 3.10 alpha 7 introduces new changes to pathlib #4