Upgrade pyspark version in evidence data source parsers

opentargets / issues

Issue tracker for Open Targets Platform and Open Targets Genetics Portal

Apache License 2.0

12 stars 2 forks source link

Open ireneisdoomed opened 3 months ago

ireneisdoomed commented 3 months ago

We want to upgrade the PySpark version in the evidence source parsers to 3.5 (currently on 3.3.1)

[ ] Run evidence generation for 24.09 with the latest PySpark release
[ ] We have written custom functions like unpivot that have been recently added, so it'd be good to move them to the Spark API.