Temp fix dependency issues of kedro and pandas

Anna-Xiong commented 7 months ago

Description

add pyarrow dependency to fix unit test error

ERROR  - DeprecationWarning: 
Pyarrow will become a required dependency of pandas in the next major release of pandas (pandas 3.0),
(to allow more performant data types, such as the Arrow string type, and better interoperability with other libraries)
but was not found to be installed on your system.
If this would cause problems for you,
please provide us feedback at https://github.com/pandas-dev/pandas/issues/54466
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!

set temporary upper bound of pandas and kedro

Screenshot

Notice

[x] I acknowledge and agree that, by checking this box and clicking "Submit Pull Request":
- I submit this contribution under the Apache 2.0 license and represent that I am entitled to do so on behalf of myself, my employer, or relevant third parties, as applicable.
- I certify that (a) this contribution is my original creation and / or (b) to the extent it is not my original creation, I am authorized to submit this contribution on behalf of the original creator(s) or their licensees.
- I certify that the use of this contribution as authorized by the Apache 2.0 license does not violate the intellectual property rights of anyone else.
- I have not referenced individuals, products or companies in any commits, directly or indirectly.
- I have not added data or restricted code in any commits, directly or indirectly.

Anna-Xiong commented 7 months ago

@huong-li-nguyen @antonymilne Hi Li and Antony, here after we add the pyarrow dependency, there is still remaining issue of vizro-core, looks like there is something off from kedro.io.core.DatasetError.

huong-li-nguyen commented 7 months ago

Details

I'm not sure if it's related to Kedro directly. I think it has something to do with the latest pandas release. If you temporarily add "pandas<2.2" to the pyproject.toml of vizro-core - does it fix it? (It seems to do so locally on my machine)

Unless you manage to fix it for the latest pandas version quickly, I would suggest to add the upper bound for now and add a comment and ticket to investigate it. We should remove it as soon as possible, but if it's blocking urgent changes, then let's leave it for later

Anna-Xiong commented 7 months ago

Details

I'm not sure if it's related to Kedro directly. I think it has something to do with the latest pandas release. If you temporarily add "pandas<2.2" to the pyproject.toml of vizro-core - does it fix it? (It seems to do so locally on my machine)

Unless you manage to fix it for the latest pandas version quickly, I would suggest to add the upper bound for now and add a comment and ticket to investigate it. We should remove it as soon as possible, but if it's blocking urgent changes, then let's leave it for later

after setting the upper bound of pandas, python 3.11 and python 3.12 works. but the remaining versions still fail

huong-li-nguyen commented 7 months ago

Details

I'm not sure if it's related to Kedro directly. I think it has something to do with the latest pandas release. If you temporarily add "pandas<2.2" to the pyproject.toml of vizro-core - does it fix it? (It seems to do so locally on my machine) Unless you manage to fix it for the latest pandas version quickly, I would suggest to add the upper bound for now and add a comment and ticket to investigate it. We should remove it as soon as possible, but if it's blocking urgent changes, then let's leave it for later

after setting the upper bound of pandas, python 3.11 and python 3.12 works. but the remaining versions still fail

Ah yeah, that's already better 😄 I just saw that Kedro has just released a new version as well. I suspect we probably need to update something here as well? I initially thought it's the ConfigLoader, but not sure? If you add an upper bound on the Kedro version as well, does it help?

Anna-Xiong commented 7 months ago

Details

@huong-li-nguyen after trying a bit combination of versions, only setting upper bound of both pandas and kedro for vizro-core works to pass all tests with recent release from pandas and kedro. Agree that we can temporarily fix them and add ticket for further investigation

mckinsey / vizro