search

crate / cratedb-examples

A collection of clear and concise examples how to work with CrateDB.
Apache License 2.0
8 stars 7 forks source link

Time Series: Update to pandas 2.0 and SQLAlchemy 2.0 #430

Closed amotl closed 2 weeks ago

amotl commented 2 months ago

About

For the time series notebooks, update pandas and SQLAlchemy to use more recent versions.

Details

An alternative to GH-400, where anyone can collaborate on the same branch. This makes sense if it is a long running one, because, for example, when it needs contributions from more people while we go.

References

amotl commented 2 months ago

Observations

This patch is currently blocked on errors like this:

TypeError: Could not convert string 'BerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlinBerlin' to numeric

-- https://github.com/crate/cratedb-examples/actions/runs/8975962618/job/24651748395?pr=430#step:6:1839 -- https://github.com/crate/cratedb-examples/actions/runs/8975962618/job/24651748121?pr=430#step:6:825

Thoughts

It looks like it is a data shape error. Maybe the way the notebooks are working with pandas needs an update when using more recent pandas 2.x? The string repetition flaw reminds me of the famous »Wat« talk by Gary Bernhardt. ;]

amotl commented 1 month ago

Does anyone have any capacity to fix this?

/cc @surister, @wierdvanderhaar, @ckurze, @hammerhead

amotl commented 2 weeks ago

We've concluded the update to more recent versions of pandas and SQLAlchemy, according to validation capacities of relevant CI jobs.

In order to make it happen, we needed to skip two notebooks from testing on CI, because they are not compatible yet. There is a dedicated issue to track this. Any support is much appreciated.