search

george-hall-ucl / dawnn

Dawnn: single-cell differential abundance with neural networks
GNU General Public License v3.0
1 stars 0 forks source link

Issue running vignette #4

Open flomato opened 6 months ago

flomato commented 6 months ago

Hello,

Trying to run the vignette but stuck here. Appreciate your advice. Best.

cells <- run_dawnn(cells, label_names = "label", label_1 = "Condition1",

  • label_2 = "Condition2", reduced_dim = "pca") WARNING:absl:Compiled the loaded model, but the compiled metrics have yet to be built. model.compile_metrics will be empty until you train or evaluate the model. WARNING:absl:Error in loading the saved optimizer state. As a result, your model is starting with a freshly initialized optimizer. Finding neighbors. Computing nearest neighbors Only one graph name supplied, storing nearest-neighbor graph only Generating neighbor labels. Creating adjacency matrix. Generating scores. Error in UseMethod("predict") : no applicable method for 'predict' applied to an object of class "c('keras.src.models.sequential.Sequential', 'keras.models.sequential.Sequential', 'keras.src.models.model.Model', 'keras.models.model.Model', 'keras.src.backend.tensorflow.trainer.TensorFlowTrainer', 'keras.backend.tensorflow.trainer.TensorFlowTrainer', 'keras.src.trainers.trainer.Trainer', 'keras.trainers.trainer.Trainer', 'keras.src.layers.layer.Layer', 'keras.layers.layer.Layer', 'keras.src.backend.tensorflow.layer.TFLayer', 'keras.backend.tensorflow.layer.TFLayer', 'keras.src.backend.tensorflow.trackable.KerasAutoTrackable', 'keras.backend.tensorflow.trackable.KerasAutoTrackable', 'tensorflow.python.trackable.autotrackable.AutoTrackable', 'tensorflow.python.trackable.base.Trackable', 'keras.src.ops.operation.Operation', 'keras.ops.operation.Operation', 'keras.src.saving.keras_saveable.KerasSaveable', 'keras.saving.keras_saveable.KerasSaveable', 'python.builtin.object')"
yingxuexiao commented 5 months ago

Hello, I also encountered the same problem, did you solve the problem？thanks a lot .

george-hall-ucl commented 5 months ago

Hi both,

Many thanks for using Dawnn and for reporting this problem! Sorry for being slow to get back to you.

I can reproduce this problem on my M2 Mac. What machines are you using?

I suspect that this might be due to needing to use a newer version of Tensorflow to work correctly on the Apple silicon, and then this newer version using a newer version of Keras, which then produces the error you see (as the model was trained on an older version of Keras). I have previously been using Dawnn (on an intel-based MacBook) with TensorFlow v2.12, but this version of TensorFlow produces stupid output when run on my M2 MacBook (but it still runs without the error message you have experienced). Hence why I think it is a combination of those two issues: Apple-silicon requires newer TensorFlow; newer TensorFlow cannot run the pre-trained model.

I can test my theory in a couple of days when I can run on my old (non-Apple-silicon) MacBook.

Thanks, George

george-hall-ucl commented 5 months ago

I've done some more digging and have come up with steps that eliminate this message for me, but the model still gives nonsense outputs. If you are on an Apple-silicon Mac, please can you try running:

conda create -y -n dawnn_test python=3.11
python -m pip install tensorflow-macos==2.12.0
pip install https://files.pythonhosted.org/packages/52/56/8373f5751011304a346f07e5423e69f809b626989d2541ae9e816ae7ced2/tensorflow_metal-1.1.0-cp311-cp311-macosx_12_0_arm64.whl

and let me know what happens when you try to run the vignette? This should install tensorflow/keras in such a way that it can load and run the model on Apple silicon (but still produces incorrect output...)

george-hall-ucl commented 5 months ago

Hi both,

In an effort to try to fix the issues with running on Apple-silicon Macs, I've released a Docker image that can run the underlying neural network. This should integrate seamlessly and make installation easier, but it does mean that you will need to first install Docker. Please let me know if it works!

I've currently released it as a branch, with installation instructions in the modified README. It is still a provisional, pre-release version as I haven't finished testing it. I'm pushing it to the repository in the hope that it will help you, but please use with a bit of caution and check that the results make sense!

Hope this solves the problems!

Thanks, George

yingxuexiao commented 5 months ago

Hi both,

In an effort to try to fix the issues with running on Apple-silicon Macs, I've released a Docker image that can run the underlying neural network. This should integrate seamlessly and make installation easier, but it does mean that you will need to first install Docker. Please let me know if it works!

I've currently released it as a branch, with installation instructions in the modified README. It is still a provisional, pre-release version as I haven't finished testing it. I'm pushing it to the repository in the hope that it will help you, but please use with a bit of caution and check that the results make sense!

Hope this solves the problems!

Thanks, George

Hi,George, Thank you very much for helping me solve the problem. I plan to organize the computer and then try "dawnn" in the next few days to avoid the problem of my computer. I will reply to you after I deal with it, looking forward to a good result! Thank you again for your many replies~

yingxuexiao commented 5 months ago

Hi George, Thank you for your previous suggestions. I updated TensorFlow to versions 2.12.0 and 2.16.1 in both Python3.9.6 and Conda environments to test, but encountered new errors. The issues with TensorFlow seem to be resolved this time, but now I'm facing different problems.

Specifically, I observed these errors on both Windows 10 and Mac M2 systems. I've verified that TensorFlow itself is not causing these new issues：

cells <- run_dawnn(cells, label_names = "label", label_1 = "Condition1",
                   label_2 = "Condition2", reduced_dim = "pca")

/Users/xiaoying/Library/Python/3.9/lib/python/site-packages/urllib3/init.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020 warnings.warn( Error in py_call_impl(callable, call_args$unnamed, call_args$named) : OSError: Unable to open file (file signature not found) Run reticulate::py_last_error() for details. reticulate::py_last_error()

── Python Exception Message ─────────────────────────────────────────────── Traceback (most recent call last): File "/Users/xiaoying/Library/Python/3.9/lib/python/site-packages/keras/saving/saving_api.py", line 212, in load_model return legacy_sm_saving_lib.load_model( File "/Users/xiaoying/Library/Python/3.9/lib/python/site-packages/keras/utils/traceback_utils.py", line 70, in error_handler raise e.with_traceback(filtered_tb) from None File "/Users/xiaoying/Library/Python/3.9/lib/python/site-packages/h5py/_hl/files.py", line 562, in init fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr) File "/Users/xiaoying/Library/Python/3.9/lib/python/site-packages/h5py/_hl/files.py", line 235, in make_fid fid = h5f.open(name, flags, fapl=fapl) File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "h5py/h5f.pyx", line 102, in h5py.h5f.open OSError: Unable to open file (file signature not found)

── R Traceback ────────────────────────────────────────────────────────────────────────────── ▆

  1. └─dawnn::run_dawnn(...)
  2. └─dawnn:::load_model_from_python(nn_model)
  3. └─keras::load_model_hdf5(model_path)
  4. └─keras:::load_model(filepath, custom_objects, compile)
  5. ├─base::do.call(keras$models$load_model, args)
  6. └─reticulate (local) <python.builtin.function>(...)
  7. └─reticulate:::py_call_impl(callable, call_args$unnamed, call_args$named) See reticulate::py_last_error()$r_trace$full_call for more details.

Have you encountered this problem before or do you have any insights on how to resolve it?

Thank you for your continued assistance.

Best regards, yingxue xiao

george-hall-ucl commented 4 months ago

Hi @yingxuexiao,

Thank you for continuing to test Dawnn on several machines! I suspect that this is still a reticulate problem (i.e. an issue with Tensorflow loading and/or running the model as I saved it).

I have two suggestions:

Many thanks again! George