search

MyungjaeSong / Paired-Library

52 stars 23 forks source link

Spearman Correlation Coefficient #10

Closed AhmadObeid closed 3 years ago

AhmadObeid commented 3 years ago

Dear authors, many thanks for sharing the codes of your paper.

Will you please share the Spearman Correlation calculation code? I used the spearmanr funciton from scipy.stats, and I am only getting around 0.68 coef. testing on the HT1-2 dataset, in contrast to the reported value in your paper of >0.75. The way I got this value is by loading your weights into the model, and testing on the HT1-2 dataset. I left the predicted Indel frequencies as well as the ones imported from the dataset as float values and used the abovementioned function.

Your feedback will be very appreciated.

Thank you, Ahmad

Sungtae-Lee commented 3 years ago

Dear Ahmad,

Thank you for showing interest to our project. Before providing details, I want to make sure one thing. Are you looking at our cpf-1 paper? "Deep learning improves prediction of CRISPR–Cpf1 guide RNA activity"

Sincerely, Sungtae Lee

AhmadObeid commented 3 years ago

Dear Sungtae-Lee, sorry for confusion. Indeed, that is the paper I was referring to. Specifically Figure1, c (HT1-2).

Ahmad

seonwoo-min commented 3 years ago

Hi, @AhmadObeid.

We have also used the Spearman function form scipy.stats package. The different outputs are most likely due to the different versions and backends. In particular, we have heard that using TensorFlow instead of Theano results in different results.

Please note that the supported versions of DeepCpf1 are python=2.7.12, theano=1.0.1, keras=2.1.5. I know that everyone, including myself, is now using the TensorFlow backend, but unfortunately, we did not have time to convert the DeepCpf1 to the TensorFlow version.

If you simply need DeepCpf1 results, webtool is available at http://deepcrispr.info/. I hope this helps with your issue.

Regards, Seonwoo

AhmadObeid commented 3 years ago

Dear all, I was not able to get the same score using the published weights. However, using your same architecture, and training the network from scratch on both MATLAB, and tensorflow 1.5, and python 3.6, with keras as backend, we are able to get close results.

Many thanks for your help. Ahmad