Machine learning plots overwrite each other under some condition

[agitter]

As first reported in https://github.com/Reed-CompBio/spras/pull/80#pullrequestreview-1511807545, sometimes when running the Snakemake workflow with multiple cores the ML plots overwrite each other. I observed behaviors such as:

• the PCA plot for data 1 actually being the PCA plot for data 0
• a hierarchical clustering plot only being partially drawn
• plots being the wrong size

I suspect there could be some sort of race condition happening due to the parallelism and the randomness of the types of errors I see. Perhaps somehow the pyplot current figure is shared across multiple Snakemake rules executing in parallel?

What makes this confusing and hard to debug is that I can consistently reproduce this behavior and Neha cannot trigger it. She is on macOS, and I'm on Windows, so the platform could have to do with it. Our plan is to merge #80 and then ask more of the SPRAS team to help us try to reproduce the problem.

[agitter]

My conda info:

             user-agent : conda/4.7.11 requests/2.21.0 CPython/3.7.3 Windows/10 Windows/10.0.19041

The expected output images for config/config.yaml:

data0-hac-horizontal.png:

data0-hac-vertical.png

data0-pca.png

data1-hac-horizontal.png

data1-hac-vertical.png

data1-pca.png

[annaritz]

I went through the steps that @agitter posted on slack; however I'm getting an error with meo (I created a new GitHub issue #99 about that). When I set include: false for meo, I get the expected 6 *.png images (without MEO though, of course).

conda info

     active environment : spras
    active env location : /Users/aritz/opt/miniconda3/envs/spras
            shell level : 2
       user config file : /Users/aritz/.condarc
 populated config files : 
          conda version : 4.11.0
    conda-build version : not installed
         python version : 3.9.6.final.0
       virtual packages : __osx=10.16=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /Users/aritz/opt/miniconda3  (writable)
      conda av data dir : /Users/aritz/opt/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/osx-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /Users/aritz/opt/miniconda3/pkgs
                          /Users/aritz/.conda/pkgs
       envs directories : /Users/aritz/opt/miniconda3/envs
                          /Users/aritz/.conda/envs
               platform : osx-64
             user-agent : conda/4.11.0 requests/2.26.0 CPython/3.9.6 Darwin/22.4.0 OSX/10.16
                UID:GID : 502:20
             netrc file : None
           offline mode : False

data0 Files

data1 Files

[ellango2612]

my results: conda info:

active environment : spras
active env location : /Users/ellango/anaconda3/envs/spras
        shell level : 2
   user config file : /Users/ellango/.condarc
  populated config files : /Users/ellango/.condarc
      conda version : 23.3.1
      conda-build version : 3.24.0
     python version : 3.10.9.final.0
      virtual packages : __archspec=1=arm64
                      __osx=13.4.1=0
                      __unix=0=0
         base environment : /Users/ellango/anaconda3  (writable)
          conda av data dir : /Users/ellango/anaconda3/etc/conda
        conda av metadata url : None
             channel URLs : https://repo.anaconda.com/pkgs/main/osx-arm64
                      https://repo.anaconda.com/pkgs/main/noarch
                      https://repo.anaconda.com/pkgs/r/osx-arm64
                      https://repo.anaconda.com/pkgs/r/noarch
          package cache : /Users/ellango/anaconda3/pkgs
                      /Users/ellango/.conda/pkgs
          envs directories : /Users/ellango/anaconda3/envs
                      /Users/ellango/.conda/envs
           platform : osx-arm64
            user-agent : conda/23.3.1 requests/2.28.1 CPython/3.10.9 Darwin/22.5.0 OSX/13.4.1
            UID:GID : 501:20
           netrc file : None
          offline mode : False

Images:

These seem to match the expected images!

[Lyce24]

I'm using Linux Operation System (Ubuntu 22.04.2) and everything works fine for me.

conda info

    active env location : /home/liue24/miniconda3/envs/spras
            shell level : 2
       user config file : /home/liue24/.condarc
 populated config files : 
          conda version : 23.3.1
    conda-build version : not installed
         python version : 3.10.10.final.0
       virtual packages : __archspec=1=x86_64
                          __glibc=2.35=0
                          __linux=5.15.90.1=0
                          __unix=0=0
       base environment : /home/liue24/miniconda3  (writable)
      conda av data dir : /home/liue24/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/liue24/miniconda3/pkgs
                          /home/liue24/.conda/pkgs
       envs directories : /home/liue24/miniconda3/envs
                          /home/liue24/.conda/envs
               platform : linux-64
             user-agent : conda/23.3.1 requests/2.28.2 CPython/3.10.10 Linux/5.15.90.1-microsoft-standard-WSL2 ubuntu/22.04.2 glibc/2.35
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

Data0 Images

Data1 Images

The images match the expected images.

[Lyce24]

I tested with Windows 11 OS as well and things went fine.

conda info

     active environment : spras
    active env location : C:\Users\eriks\.conda\envs\spras
            shell level : 2
       user config file : C:\Users\eriks\.condarc
 populated config files :
          conda version : 23.5.2
    conda-build version : not installed
         python version : 3.11.4.final.0
       virtual packages : __archspec=1=x86_64
                          __win=0=0
       base environment : C:\ProgramData\miniconda3  (read only)
      conda av data dir : C:\ProgramData\miniconda3\etc\conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\ProgramData\miniconda3\pkgs
                          C:\Users\eriks\.conda\pkgs
                          C:\Users\eriks\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\eriks\.conda\envs
                          C:\ProgramData\miniconda3\envs
                          C:\Users\eriks\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/23.5.2 requests/2.29.0 CPython/3.11.4 Windows/10 Windows/10.0.22621
          administrator : False
             netrc file : None
           offline mode : False

Data0 Images

Data1 Images

The images match the expected images.

[agitter]

Thanks for the help testing this everyone. This makes me more confident that our code is okay. I would like to keep this issue open to alert other SPRAS users there may be a problem. However, because no one else can reproduce it, the problem could be isolated to my machine.

The next step to debug this could be to create a minimal example to reproduce the overwriting plots and ask the Snakemake team for help.

[livj4711]

I seem to have issues with my images after running.
Conda info:

    active environment : spras
    active env location : C:\Users\ojohnson\Documents\BDS\Anaconda3\envs\spras
            shell level : 2
       user config file : C:\Users\ojohnson\.condarc
 populated config files : C:\Users\ojohnson\.condarc
          conda version : 23.5.2
    conda-build version : 3.25.0
         python version : 3.11.3.final.0
       virtual packages : __archspec=1=x86_64
                          __win=0=0
       base environment : C:\Users\ojohnson\Documents\BDS\Anaconda3  (writable)
      conda av data dir : C:\Users\ojohnson\Documents\BDS\Anaconda3\etc\conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\ojohnson\Documents\BDS\Anaconda3\pkgs
                          C:\Users\ojohnson\.conda\pkgs
                          C:\Users\ojohnson\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\ojohnson\Documents\BDS\Anaconda3\envs
                          C:\Users\ojohnson\.conda\envs
                          C:\Users\ojohnson\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/23.5.2 requests/2.29.0 CPython/3.11.3 Windows/10
 Windows/10.0.19045
          administrator : False
             netrc file : None
           offline mode : False  

data0-hac-horizontal.png:

data0-hac-vertical.png:

data0-pca.png:

data1-hac-horizontal.png:

data1-hac-vertical.png:

data1-pca.png:

data0-hac-horizontal.png is expected
data0-hac-vertical.png is horizontal, not vertical
data0-pca.png labels are overlapping
data1-hac-horizontal.png bottom is cut off
data1-hac-vertical.png is expected
data1-pca.png does not match

[agitter]

Livvy and I both encountered the errors, and both of us are on Windows 10. Specifically looking at https://en.wikipedia.org/wiki/Windows_10_version_history:

• version 2004, 20H1, build 19041
• version 22H2, 22H2, build 19045

However, Windows 11 worked. Specifically, looking at https://en.wikipedia.org/wiki/Windows_11_version_history:

• version 22H2, Sun Valley 2, build 22621

There could be slight differences in our conda environments as well.