Allow pyarrow<7 as dependency and remove support for Python 3.6

[steffen-schroeder-by]

Description:

Allow pyarrow 4, 5, 6 as dependency Since Python 3.6 will be EOL by end of the month, we won't support it from Version 5.2 onwards

• [X] Changelog entry

[xhochy]

@steffen-schroeder-by 6.0 should also work. The issue you have seen is that conda-forge has dropped support for Python 3.6 builds and thus no new packages will pop there. Could we also drop Python 3.6 here, too?

[steffen-schroeder-by]

I also thought about dropping 3.6 but strictly this would be an incompatible change and require a new major version, wouldn't it? I'd like to "reserve" the major version bump to 6.0 for bringing in the 4.0 state (which we are still working on). My preferred way would be to drop 3.6 support with a 5.20 release. How have you done this in the past?

[xhochy]

I also thought about dropping 3.6 but strictly this would be an incompatible change and require a new major version, wouldn't it? I'd like to "reserve" the major version bump to 6.0 for bringing in the 4.0 state (which we are still working on). My preferred way would be to drop 3.6 support with a 5.20 release. How have you done this in the past?

We never dropped a Python version in Kartothek as it was started at roughly the same time where Python 3.6 was the default choice. Thus this is the first time.

I would do it as part of a minor release though. If we update https://github.com/JDASoftwareGroup/kartothek/blob/6bc7e868435e98cbda0b695900f29d1ff7d49110/setup.py#L32 correctly, new wheels should only be pulled by Python 3.7+ installations and Python 3.6 based environments wouldn't see a new release. Thus nothing should break. This is at least how pandas and co have done it in the last months.

[steffen-schroeder-by]

The tests themselves are not flaky. The point here is that the arrow-nightly tests fail. They are allowed to do so. They are an early-warn system if there are upstream version change/issue. If the tests for the stable version succeed, we are good. Having said that, it's still a pity that the arrow-nightlies lead to the other tests being aborted. Really thinking about disabling the fail-early in the github action to have a holistic picture for all supported versions despite of the arrow-nightlies.

[steffen-schroeder-by]

All tests (except the nightly) passed