Error running medaka_consensus: cannot import name 'dtensor' from 'tensorflow.compat.v2.experimental'

[alexanderwg-ornl]

I've been using medaka for a couple of years in a routine pipeline for genomic assembly. However, a number of months back it just stopped working. I've been skipping using it but now I find I desperately need to get it working. I don't recall changing any dependencies manually, but even when I forced a reinstall via pip I get the following error:

[M::main] Real time: 438.335 sec; CPU: 368.711 sec; Peak RSS: 1.152 GB [bam_sort_core] merging from 1 files and 1 in-memory blocks... Running medaka consensus [12:06:30 - Predict] Setting tensorflow inter/intra-op threads to 1/1. [12:06:30 - Predict] Processing region(s): cluster_001_consensus:0-2439215 cluster_007_consensus:0-19645 cluster_010_consensus:0-16342 cluster_013_consensus:0-3605 [12:06:30 - Predict] Using model: /usr/local/lib/python3.8/dist-packages/medaka/data/r941_min_hac_g507_model.tar.gz. [12:06:30 - Predict] Processing 6 long region(s) with batching. [12:06:30 - MdlStrTF] ModelStoreTF exception <class 'ImportError'> Traceback (most recent call last): File "/usr/local/bin/medaka", line 8, in <module> sys.exit(main()) File "/usr/local/lib/python3.8/dist-packages/medaka/medaka.py", line 724, in main args.func(args) File "/usr/local/lib/python3.8/dist-packages/medaka/prediction.py", line 160, in predict model = model_store.load_model(time_steps=args.chunk_len) File "/usr/local/lib/python3.8/dist-packages/medaka/datastore.py", line 180, in load_model self.model = model_partial_function(time_steps=time_steps) File "/usr/local/lib/python3.8/dist-packages/medaka/models.py", line 132, in build_model from tensorflow.keras.models import Sequential File "/home/5g6/.local/lib/python3.8/site-packages/keras/api/_v2/keras/__init__.py", line 12, in <module> from keras import __version__ File "/home/5g6/.local/lib/python3.8/site-packages/keras/__init__.py", line 21, in <module> from keras import models File "/home/5g6/.local/lib/python3.8/site-packages/keras/models/__init__.py", line 18, in <module> from keras.engine.functional import Functional File "/home/5g6/.local/lib/python3.8/site-packages/keras/engine/functional.py", line 26, in <module> from keras import backend File "/home/5g6/.local/lib/python3.8/site-packages/keras/backend.py", line 34, in <module> from keras.dtensor import dtensor_api as dtensor File "/home/5g6/.local/lib/python3.8/site-packages/keras/dtensor/__init__.py", line 22, in <module> from tensorflow.compat.v2.experimental import dtensor as dtensor_api ImportError: cannot import name 'dtensor' from 'tensorflow.compat.v2.experimental' (/usr/local/lib/python3.8/dist-packages/tensorflow/_api/v2/compat/v2/experimental/__init__.py) Failed to run medaka consensus. Any insight into how to fix this would be greatly appreciated.

[cjw85]

This seems like a dependency issue. I would suggest creating a fresh virtual environment and starting from scratch with the install if you have not already.

[cjw85]

Closing through lack of repsonse.