search

UAL-RE / ReQUIAM

Identity and Access Management backend software for UA Research Data Repository (ReDATA)
https://requiam.readthedocs.io
MIT License
2 stars 2 forks source link

Enhancement: Update to Python 3.9 #170

Closed zoidy closed 2 years ago

zoidy commented 2 years ago

Summary Numpy 1.22 dropped support for Python 3.7. ReQUIAM uses Pandas which depends on Numpy. ReQUIAM uses Python 3.7 currently. Need to upgrade ReQUIAM to use a newer version of Python. Python 3.10 is still too new so a migration to Python 3.9 is preferred. It's also a version that is already included in the redata-commons.

Objectives Migrate to Python 3.9

Proposal

Testing notes Need to verify that the current version of Pandas works ok with the new version of Numpy. Testing is to be done on the main ReQUIAM functions

Additional notes

Implemented in: TBD

yhan818 commented 2 years ago

For requirements.txt:

ldap3==2.6.1 numpy==1.23.0 redata>=0.4.2

In addition, need to update file "setup.py"

yhan818 commented 2 years ago

/requiam/org_code_numbers.py uses np.zeros(), which has no change from the numpy 1.20.0 to numpy 1.23.0. OK.

yhan818 commented 2 years ago

"script_run" uses pandas to read "pd.read_csvI()" and "pd.DataFrame.from_dict()". Ok.

Current Pandas latest version is 1.4.3.
-- read_csv() has no changes.

-- DataFrame.from_dict() has slightly changes but has no impacts to the script.

Pandas 1.2.3 https://pandas.pydata.org/pandas-docs/version/1.2.3/reference/api/pandas.DataFrame.from_dict.html#pandas.DataFrame.from_dict

Panda 1.4.3 https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.from_dict.html#pandas.DataFrame.from_dict

From testing, upgrading pandas version from 1.2.3 to 1.4.3 requires to upgrade tabulate (0.8.3) to (>=0.8.7)

yhan818 commented 2 years ago

Need to update "redata-commons" as well for Python version and related stuff.

yhan818 commented 2 years ago

".travis.yml " needs to be updated.

zoidy commented 2 years ago

".travis.yml " needs to be updated.

We don't use Travis CI anymore. https://requiam.readthedocs.io/en/latest/continuous_integration.html This file could probably be deleted

yhan818 commented 2 years ago

"script_run" uses pandas to read "pd.read_csvI()" and "pd.DataFrame.from_dict()". Ok.

Current Pandas latest version is 1.4.3. -- read_csv() has no changes.

-- DataFrame.from_dict() has slightly changes but has no impacts to the script.

Pandas 1.2.3 https://pandas.pydata.org/pandas-docs/version/1.2.3/reference/api/pandas.DataFrame.from_dict.html#pandas.DataFrame.from_dict

Panda 1.4.3 https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.from_dict.html#pandas.DataFrame.from_dict

So, upgrading pandas version from 1.2.3 to 1.4.3 shall have no impacts.

I have tested Pandas yesterday, and found that Pandas relies on Tabulate newer version. So these dependencies (at least Pandas, Tabulate) shall be updated.

yhan818 commented 2 years ago

Currently Python 3.10 build will fail. see https://github.com/UAL-RE/ReQUIAM/runs/7081741135?check_suite_focus=true

because "numpy.distutils is deprecated since NumPy 1.23.0, as a result of the deprecation of distutils itself. It will be removed for Python >= 3.12. For older Python versions it will remain present. It is recommended to use setuptools < 60.0 for those Python versions. For more details, see: https://numpy.org/devdocs/reference/distutils_status_migration. "

Therefore, skip Python 3.10 for now.

zoidy commented 2 years ago

While I'm ok with skipping the Python 3.10 build for now if it's broken, I'm a bit confused. The message says that for Python < 3.12 numpy.disutils will remain. So on paper, its deprecation should have no effect. I feel like I'm missing something here.

yhan818 commented 2 years ago

While I'm ok with skipping the Python 3.10 build for now if it's broken, I'm a bit confused. The message says that for Python < 3.12 numpy.disutils will remain. So on paper, its deprecation should have no effect. I feel like I'm missing something here.

see https://github.com/UAL-RE/ReQUIAM/runs/7081741135?check_suite_focus=true

Dependencies installation failed.

zoidy commented 2 years ago

I see. I'm not fully convinced it's due to the disutils deprecation message though. In any case, I'm ok with removing the 3.10 test.

yhan818 commented 2 years ago

I see. I'm not fully convinced it's due to the disutils deprecation message though. In any case, I'm ok with removing the 3.10 test.

See Python 3.10 documentation https://docs.python.org/3/whatsnew/3.10.html# vs. https://docs.python.org/3/whatsnew/3.9.html#distutils

The discussion can be found at https://peps.python.org/pep-0632/ (This may result the issue when we have to migrate to Python >=3.10)

We need to keep an eye on this, when in the future Python 3.10 is required.