danjweiner
commented
1 year ago Hi,
It looks like you are using Hail version 0.2.74, which was released back in 2021. I'd guess the issue is that this is an old version of Hail. Try installing the newest version of Hail (0.2.108) and then attempt to load that file again.
Best
Hello,
I'm trying to use your script "genebass_variant_filter_january_2023.py" to retrieve variant level summary stats for a couple of phenotypes from genebass. After dealing with a lot of issues installing and running Hail on my local Mac OS computer, I'm getting an error which I don't know how to deal with:
python3 genebass_variant_filter_january_2023.py Initializing Hail with default parameters... 2023-02-02 15:47:57 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 2023-02-02 15:47:58 WARN Hail:43 - This Hail JAR was compiled for Spark 3.1.1, running with Spark 3.1.3. Compatibility is not guaranteed. Running on Apache Spark version 3.1.3 SparkUI available at http://dhcp-10-102-43-57.wireless.northwestern.private:4040 Welcome to __ __ <>__ / /_/ /__ __/ / / __ / _/ / / // //_,//_/ version 0.2.74-0c3a74d12093 LOGGING: writing to hail-20230202-1547-0.2.74-0c3a74d12093.log Traceback (most recent call last): File "genebass_variant_filter_january_2023.py", line 47, inJava stack trace: is.hail.utils.HailException: incompatible file format when reading: gs://ukbb-exome-public/500k/results/variant_results.mt supported version: 1.5.0, found 1.6.0 The cause of this error is usually an attempt to use an older version of Hail to read files generated by a newer version. This is not supported (Hail native files are back-compatible, but not forward-compatible). To read this file, use a newer version of Hail. at is.hail.utils.ErrorHandling.fatal(ErrorHandling.scala:11) at is.hail.utils.ErrorHandling.fatal$(ErrorHandling.scala:11) at is.hail.utils.package$.fatal(package.scala:77) at is.hail.expr.ir.RelationalSpec$.readMetadata(AbstractMatrixTableSpec.scala:48) at is.hail.expr.ir.RelationalSpec$.readReferences(AbstractMatrixTableSpec.scala:68) at is.hail.variant.ReferenceGenome$.fromHailDataset(ReferenceGenome.scala:584) at is.hail.variant.ReferenceGenome.fromHailDataset(ReferenceGenome.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748)
Hail version: 0.2.74-0c3a74d12093 Error summary: HailException: incompatible file format when reading: gs://ukbb-exome-public/500k/results/variant_results.mt supported version: 1.5.0, found 1.6.0 The cause of this error is usually an attempt to use an older version of Hail to read files generated by a newer version. This is not supported (Hail native files are back-compatible, but not forward-compatible). To read this file, use a newer version of Hail.`
So basically what is says is that the file gs://ukbb-exome-public/500k/results/variant_results.mt was made with a newer version of Hail (1.6) but the supported version is 1.5, but at the same time Hail version is 0.2.74-0c3a74d12093. I tried to install the newest version of Hail available in the documentation, which is the one I have. So I don't know what is the problem here. I'll appreciate if you could help me with this.
Best,
B