Compatibility Issue: java.lang.NoSuchMethodError with glow.py on PySpark 3.5.1 - Githubissues

projectglow / glow

An open-source toolkit for large-scale genomic analysis

https://projectglow.io

Apache License 2.0

272 stars 111 forks source link

Compatibility Issue: java.lang.NoSuchMethodError with glow.py on PySpark 3.5.1 #724

Open nickzren opened 2 weeks ago

nickzren commented 2 weeks ago

When using glow.py in a Conda environment with the following setup:

Python version: 3.10
PySpark version: 3.5.1
Glow.py version: 2.0.0 (installed via pip install glow.py)

Attempting to load a VCF file results in the following error:

java.lang.NoSuchMethodError: 'java.lang.String org.apache.spark.sql.execution.datasources.PartitionedFile.filePath()'

Steps to Reproduce:

Create a Conda environment with Python 3.10.
Install PySpark 3.5.1 and glow.py via pip.
Attempt to load a VCF file using Glow.

Workaround: Downgrading to PySpark 3.3.2 resolves the issue, and the VCF file loads as expected.

kermany commented 2 weeks ago

can you try building the artifacts based on the latest release (2.0.3) and try again?