search

Stanford-STAGES / stanford-stages

Automated sleep staging scoring and narcolepsy identification
76 stars 26 forks source link

Wrong calculations with Python 3.5 #5

Closed Gnork closed 3 years ago

Gnork commented 5 years ago

First of all, thank you to all the researchers and developers for publishing your code and models. We are researchers from CBMI - HTW Berlin and are currently investigating the use of the Narco App for hypnogram generation in our own work.

Problem Description

The narco_app.py tool gives wrong results when running it with Python 3.5 without crashing. This happens regardless of using a GPU or a CPU with tensorflow.

We first tried the setup with one of our own EDF files and found that the hypnogram almost exclusively shows the WAKE stage, which was not close to the corresponding human-made hypnogram. We then tried to validate the app with the CHP040.edf file provided on your Github README, with a similar result. The resulting hypnogram does not match your validation hypnogram, but instead mostly classifies the stages as WAKE.

Context

We are aware, that you tested the app with Python 3.6 as written in the Google Doc. We still think this is a problem, because it is very easy for users to use a different configuration.

For example, we are using Docker images for most of our experiments and tried to install the app on top of the official tensorflow:1.9 Docker base image provided by the tensorflow team. This image is based on Ubuntu 16.04 with Python 3.5, which shows the erroneous behaviour.

Suggestion

In the app, try to detect the Python version and crash/exit if an untested Python version is detected and give the user a helpful message. If you don't want to do this, we would suggest adding a warning on the Google Doc and Github README.

What we have tested so far

We could narrow the problem down to be related to Python 3.5, but we do not yet know, what exactly causes the problems. It could be the Python 3.5 interpreter itself, one of the dependencies misbehaving with this particular Python version or it might only be related to narco_app.py. We think it is worth to further investigate the issue, because this could affect other programs / machine learning models as well and we would appreciate your opinion on what the problem might be.

Docker Base Image Python version Correct
docker.io/tensorflow/tensorflow:1.13.1-py3 Python 3.5 no
docker.io/tensorflow/tensorflow:1.9.0-py3 Python 3.5 no
docker.io/ubuntu:16.04 Python 3.5 no
docker.io/ubuntu:18.04 Python 3.6 yes
docker.io/ubuntu:16.04 Python 3.5 via pyenv no
docker.io/ubuntu:16.04 Python 3.6 via pyenv yes
docker.io/ubuntu:18.04 Python 3.5 via pyenv no
docker.io/ubuntu:18.04 Python 3.6 via pyenv yes
docker.io/nvidia/cuda:10.0-cudnn7-runtime-ubuntu18.04 Python 3.6 yes
docker.io/tensorflow/tensorflow:1.13.1-gpu-py3 Python 3.5 no

For all images not including tensorflow by default, tensforflow was installed via pip. We found that it does not matter if tensorflow 1.9.0 or tensorflow 1.13.1 is used.

Best regards, Christoph Jansen and Prof. Dagmar Krefting

neergaard commented 5 years ago

Thanks for the heads up.

Of the top of my head, I can't really see why 3.5 would make such a huge difference, but we will definitely be checking up on that ASAP!

Meanwhile, until the issue is resolved, we will go with a warning in the gdoc and README.

informaton commented 3 years ago

Unable to resolve this issue. Python 3.6 or greater is required at this point.