search

MicrobeLab / DeepMicrobes

DeepMicrobes: taxonomic classification for metagenomics with deep learning
https://doi.org/10.1093/nargab/lqaa009
Apache License 2.0
81 stars 21 forks source link

Different prediction results for the same fastq files #6

Open ramtinz opened 3 years ago

ramtinz commented 3 years ago

Hi Qiaoxing,

I used DeepMicrobes for species prediction in a system with GPU and a system without GPU (only CPUs). Surprisingly, predicted species are different for the same fastq samples. Indeed, the tfrec files of the same fastq files in the GPU-based system are a little larger in size than the tfrec files. I checked my scripts in both systems and they are the same. If the difference was related to the use of GPU, there must not be any difference in the tfrec files as the file conversion does not require GPU. In addition, the predictions in the GPU-based system had markedly higher confidence scores than the system without GPUs. Is there anything to explain this? Thank you in advance

ramtinz commented 3 years ago

I found some fundamental arguments that could explain the different prediction results in two different systems: There might be many reasons for this to happen according to raised issues by other developers using TensorFlow. As such is numerical instability: even simple multiplication might result in different numbers depending on word size (32 bit vs 64 bit) which itself is dependent on the system architecture. In addition, versions of TensorFlow, and their dependencies may contribute in different inference results for the same model. https://github.com/tensorflow/tensorflow/issues/19200 https://stackoverflow.com/questions/43221730/tensorflow-same-code-but-get-different-result-from-cpu-device-to-gpu-device

MicrobeLab commented 3 years ago

Hi Ramtin,

Sorry for the late reply. I did not notice these differences between CPU and GPU before. Thank you very much for this information.