Spark version upgrade 3.4, 3.3

projectglow / glow

An open-source toolkit for large-scale genomic analysis

https://projectglow.io

Apache License 2.0

262 stars 106 forks source link

Spark version upgrade 3.4, 3.3 #531

Closed jisqaqov closed 5 months ago

jisqaqov commented 1 year ago

Hello. Could you please tell me if any new release upcoming to support Glow with Spark 3.3 and 3.4 versions?

zyd14 commented 1 year ago

Not a maintainer, but unfortunately it feels like this project has been mostly abandoned :(

Hoeze commented 11 months ago

See also https://github.com/projectglow/glow/issues/520. I really love this library for its ability to read and write VCF files with pyspark, but I can imagine that glow itself has many features which are hard to maintain, e.g. the whole-genome regression things or the interop with hail.

kermany commented 11 months ago

100% agreed on the importance of the VCF parser and the need for making the project more maintainable by focusing on critical functionalities - e.g efficient ingest of VCF, bgen etc. stay tuned ...

henrydavidge commented 5 months ago

Hi @Hoeze @zyd14 @jisqaqov, we've now merged support for Spark 3.4 and 3.5. I'll release new artifacts soon. Until then, it's possible to test by building Scala and Python artifacts from source.

doit-mattporter commented 4 months ago

Is there an ETA for this new 3.4 + 3.5 support to be pushed to your Maven repo?