eto-ai / rikai

Parquet-based ML data format optimized for working with unstructured data
https://rikai.readthedocs.io/en/latest/
Apache License 2.0
136 stars 19 forks source link

Support for Spark 3.3.1 #710

Closed da-tubi closed 1 year ago

da-tubi commented 1 year ago

Why Spark 3.3.0 should be tested with Python 3.9?

Databricks 11.3 LTS is using Spark 3.3.0 and Python 3.9

Renkai commented 1 year ago

Consider cache the LD_LIBRARY_PATH? The default behavior of cache:pip only caches very limited content.

https://github.com/eto-ai/rikai/actions/runs/3494328797/jobs/5850014468#step:10:8

da-liii commented 1 year ago

Well, this pr has passed the Python CI. Let us improve CI speed in another PR.

da-liii commented 1 year ago

Two CI failed, because it failed to download the pretrained model from torchvsion.