switch to python 3.6+

[thehesiod]

given 3.7 is already out, this would allow us to use async generators and make the code a lot easier to match to botocore

@aio-libs/aiobotocore-admins what do you guys think?

[jettify]

I all for deprecation python3.5, but not sure how many users still on py3.5

[jaddison]

Not sure what the rationality is for pushing Python version deprecation so rapidly?

Major Linux distributions (for example) have LTS releases that maintain 3.5 (without easy Python upgrading) until 2021 (eg Ubuntu 16.04: https://wiki.ubuntu.com/Releases).

If you want to encourage adoption, you need to cater to a fairly large demographic. Basically, pick a deprecation/upgrade policy that meets the needs of most users.

Edit: seeing this as an open issue automatically makes me reconsider using this lib in a project I'm working on.

[thehesiod]

it's to get rid of custom code, less code to maintain = better

[jaddison]

I agree it's technically better - as long as you are willing to accept that it will unfortunately be at the expense of your user base.

I suppose you could advise anyone using 3.5 to use version 'x' (the last version to support 3.5), and that they cannot upgrade - but it would still turn a lot of people off (after all, they won't get fixes or new features), unless you were maintaining two streams (3.5 and >3.5). Which is obviously more work.

[thehesiod]

ya part of this is to get feedback for how many people can't upgrade to 3.6 to see if we can't update yet, or need to support a backport. Is there a technical reason you guys can't switch to 3.6?

[jaddison]

Sorry, I thought I was pretty clear in my previous messaging:

Major Linux distributions (for example) have LTS releases that maintain 3.5 (without easy Python upgrading) until 2021 (eg Ubuntu 16.04: https://wiki.ubuntu.com/Releases).

My use case: I'm looking at using aiobotocore in a product I'm planning to open source. I'm choosing to target users who are Python3.5+, because of the LTS reason mentioned above. If I use aiobotocore and have to target Python3.6, that will exclude many people who rely on Ubuntu LTS support (just an example, I assume other distros are Python version-locked as well).

In a nutshell: I personally can switch to Python3.6, but not everyone can or is willing to until other factors like a distro's LTS release schedule plus their business' upgrade schedule allows them to.

To me, open-sourcing code means supporting it through the restrictions of its users - even when it's frustrating to do so. If many asyncio users aren't currently able to upgrade off of Python3.5, then aiobotocore won't have the wider adoption it needs if you drop support for it.

I will admit that I'm using the Ubuntu LTS distro as a basis for the argument; not everyone is using it, nor does everyone that uses it codes with Python. However, it is a very popular server distribution and so I think it is a fairly safe assumption here.

[jaddison]

I do think that it's important for open source projects like aiobotocore to have a deprecation policy that allows them to refactor and drop support for dependencies to make project maintenance easier. Absolutely.

But that deprecation policy needs to be driven by outside factors that are hard to gauge. I know you guys are trying to do this by having this issue open for discussion - kudos to you! My input is simply that you can extrapolate from factors like Python versioning in Linux distros and other outside factors.

BTW, awesome work on aiobotocore. It's a solid contribution to the asyncio world.

[thehesiod]

using LTS distro does lock one to the version it's available with. We use jessie w/ 3.6 by compiling python ourselves. Also there are things like pipenv which allow you to install in user land.

update: yet another: pyinstaller

[thehesiod]

btw, python devs seem to not care too much about backporting important fixes :( For example, 3.6 has a horrible bug where each warning will trigger a memory leak, only fixed in 3.7 :( We had to build 3.6 with the patch ourselves: https://github.com/python/cpython/pull/4587

[thehesiod]

lets keep the conversation going, really appreciate the feedback!

[jaddison]

python devs seem to not care too much about backporting important fixes

Perhaps subjective on my part, but I'd consider Python not back-porting bugfixes very different from a Python package removing support for older Python versions - an apples to oranges comparison.

We use jessie w/ 3.6 by compiling python ourselves. Also there are things like pipenv which allow you to install in user land. update: yet another: pyinstaller

Agreed, people can upgrade Python, but I don't think many would. It sort of defeats the purpose of going with an LTS distro. Pyinstaller is pretty niche; pipenv (I don't think?) allows you to install arbitrary Python versions.

However, you are right in that there are indeed options that allow for the dropping of Python3.5 in aiobotocore. I'm just not sure it's realistic to assume people will want to/be able to do so.

[thehesiod]

whoops, pyenv: https://github.com/pyenv/pyenv. We can definitely keep a backport, and backport important fixes.

Having a release without important fixes effectively makes it worse than dead, because now you have to avoid that version and are forced to use the new (or older) version...whereas if aiobotocore were to drop support for 3.5 you could presumably still use the old release, but wouldn't get new features until you upgraded to 3.6. Thinking this way, if we were to do anything, I now think we should open a 3.5 branch and always backport important bugfixes so we do better than core python. And if it's really important we do point botocore upgrades as individual users need them. We'd have to think how versioning would work across these two branches

I'm not the final word on any of this btw, it's @jettify . And there's no timeline for any of this.

[jaddison]

@thehesiod I agree completely with your last message. I think that is the best approach.

Edit: for versioning, I'd probably use the common approach of 1.x is locked python 3.5, and 2.x is 3.6+. Backporting fixes gives you 1.x.z or even 1.y.

[asvetlov]

@thehesiod could you please enumerate of Python 3.5 spikes you want to get rid? I mean new syntax and other backward incompatible changes.

[thehesiod]

Will start it in this post:

• greatly simplify this: https://github.com/aio-libs/aiobotocore/blob/master/aiobotocore/paginate.py#L31 to look more like this: https://github.com/boto/botocore/blob/1.10.58/botocore/paginate.py#L238
• similarly with https://github.com/aio-libs/aiobotocore/blob/master/aiobotocore/response.py#L109 and https://github.com/boto/botocore/blob/1.10.58/botocore/response.py#L116

[asvetlov]

If all you need is async generator support -- please consider https://pypi.org/project/async_generator/ usage.

[thehesiod]

cool, will look into it, thanks

[dimaqq]

I too would argue for 3.6+. 3.6 made great "usability" improvements over 3.5 in terms of asyncio.

While there surely are many users on 3.5, I hazard a guess those are not asyncio users. Then there's a tradeoff between existing users vs. new users -- given that aiobotocore is kinda new, I think new users should be preferred over existing.

A few other projects are following this line, some even going as far as requiring 3.7 (e.g. quart). I think it's acceptable if the older versions are still around.

[asvetlov]

3.6 made great "usability" improvements over 3.5 in terms of asyncio.

Would you enumerate these improvements?

[dimaqq]

3.6 made great "usability" improvements over 3.5 in terms of asyncio.

Would you enumerate these improvements?

Good flame bait, Andrew. I believe you know pros and cons quite well.

IMO, for this project, the main improvement is not having to pass the event loop around. For me, that's a dramatic usability improvement. In fact if I see an asyncio library or project that claims to support 3.5, my gut reaction is "how can I make sure they support 3.5 correctly".

Internal asyncio changes are said to have been backported to 3.5.x. I, for one, am not sure to which 3.5.x.

Asynchronous generators (and the inverse, asynchronous comprehensions) are ever so neat to express e.g. streaming transfers.

While we can argue about exact supported versions and how exactly something can be implemented, I'd rather see a clear, reasonable statement for users of the library. I think it could be "please use 3.6+, it's good for you".

Ref: Python Changelogmentry for 3.6.0:

The asyncio module has received new features, significant usability and performance improvements, and a fair amount of bug fixes. Starting with Python 3.6 the asyncio module is no longer provisional and its API is considered stable.

[thehesiod]

in above PR I used the async-generator module @asvetlov recommended and it worked well, going to close this issue as that was the major pain point. If anyone else feels strongly for something we can do better by dropping old versions please re-open new issue.

[asvetlov]

Python 3.5.3+ doesn't need to pass event loop instance everywhere. aiobotocore (and aiohttp) requires 3.5.3 starting from aiohttp 3.0 (about a year ago). async comprehension is a nice feature, but aiobotocore itself needs async generators only which is solved by async-generator module.