skymandr commented 6 months ago

In this PR I:

bump dependencies, most notably PyArrow in Lambda,
assure compatibility with pyarrow>=14.0.0 for users by tweaking the partitioning filter to use a bigger int size (using the minimal sizes cast the result to the lower size, causing overflow error) in read_parquet_functions.py,
fix a bug in read_parquet_functions.py that did not properly enforce timezone has been fixed. The corresponding logic can now be removed from MATS Utilities,
implement minor code style fixes.

Note that versions `12.x.x` and `13.x.x` of `PyArrow` have a bug in how timestamps are implemented and cannot be used with our dataset. These versions cannot and will not be supported!

Note also that staying on `10.x.x` or using `11.x.x` works fine – there is no immediate need for users to upgrade, but if they do upgrade, they must use `pyarrow>=14.0.0` (latest is `pyarrow==15.0.0`).

Resolves #149

lindamegner commented 6 months ago

Great that this is fixed. Is there a reason why it is not merged?

skymandr commented 6 months ago

Great that this is fixed. Is there a reason why it is not merged?

I want to have approval from someone in the project, not just from my boss, before merging. :)

lindamegner commented 6 months ago

Now you have the approval! :)