search

microsoft / hyperspace

An open source indexing subsystem that brings index-based query acceleration to Apache Spark™ and big data workloads.
https://aka.ms/hyperspace
Apache License 2.0
423 stars 115 forks source link

Bump Spark #413

Open clee704 opened 3 years ago

clee704 commented 3 years ago

Is there a reason that we can't bump the patch version?

I quickly checked and there was only one breaking test. Seems not serious.

imback82 commented 3 years ago

I quickly checked and there was only one breaking test.

Interesting, which test was failing?

clee704 commented 3 years ago

I quickly checked and there was only one breaking test.

Interesting, which test was failing?

I tried it again and this time there was no failing test.

clee704 commented 3 years ago

Current versions:

Latest versions:

clee704 commented 3 years ago

Can we use the latest versions? Is Azure Synapse Analytics blocking this?

imback82 commented 3 years ago

Any reason we want to use the latest patch version? Do we need to keep track/update this every time a new patch version is released?

Do you want to check if the latest patch releases introduce any breaking changes?

clee704 commented 3 years ago

Interesting question. I've never questioned that before. Maybe you are asking because we're not a direct user of Spark, and those bug/security fixes really don't matter for us?

In the long run, the latest patch version will most likely be the most used version among versions of the same major/minor version. So we should aim for maximum compatibility for the latest patch version. I don't think we should actively track and update for every patch version release, though. In practice, it should be okay to do this once in a while.

MironAtHome commented 2 years ago

3.2 is out, will test it out before tomorrow EOD.

vishalsurana commented 11 months ago

I'm evaluating the use of this package and I wanted to know the current status of this project. Latest Spark version is 3.5 and Scala 2.11 has been deprecated in favor of Scala 2.13 and Scala 3.

Secondly can you point me to some benchmarks that highlight the benefit of this package?