search

snakemake / snakeface

The snakemake interface, currently works like a notebook (under development)
https://snakeface.readthedocs.io/
Mozilla Public License 2.0
17 stars 6 forks source link

Guide to getting snakeface running #21

Open iamh2o opened 3 years ago

iamh2o commented 3 years ago

Hi-- I'm REALLY excited for this project. I decided to spend some time exploring the repo, and I'm thrilled with what I'm seeing. I also had to do a lot of hacking outside the docs to get things to run-- which I'd like to offer as a possible guide to help others get going. These are more or all of the steps from install through running snakeface (both running pipes itself and listening to externally executed pipes!). It turned out quite long-- if there is a better format to share this in, I'm happy to relocate- please let me know. I tried to keep my comments in comments, the code executed unmarked and the responses in blocks. There are several bugs/documentation points addressed throughout which felt more natural to leave in the broader step by step since they seemed to be OoO dependent.

conda deactivate                                                                                                                                    
mkdir snkfc                                                                                                                                         
cd snkfc                                                                                                                                            
mamba create -n SNKFC                                                                                                                               
conda activate SNKFC                                                                                                                                
mamba install --yes -n SNKFC python=3.8 pip  perl                                                                                                   

git clone git@github.com:snakemake/snakeface.git                                                                                                    
cd snakeface                                                                                                                                        

python -m venv env                                                                                                                                  
source env/bin/activate                                                                                                                             
perl -pi -e 's/^snakeface/\# snakeface/g;' requirements.txt  # Was this intended to be in the req.txt?                                              
pip install wheel # pip  install reqs.txt complained a lot for not having it                                                                        
pip install -r requirements.txt                                                                                                                     

pip install -e .                                                                                                                                    

pip install snakemake[reports] # not having this was causing me bugs latter on.        
snakeface

Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
June 06, 2021 - 08:27:19
Django version 3.0.8, using settings 'snakeface.settings'
Starting ASGI/Channels version 3.0.3 development server at http://127.0.0.1:5000/
Quit the server with CONTROL-C.
Listen failure: Couldn't listen on 127.0.0.1:5000: [Errno 98] Address already in use.

If i edit the DOMAIN_HOST or DOMAIN_PORT in the settings.yml, they do not get picked up. 

perl -pi -e 's/DOMAIN_NAME\: (http.*)/DOMAIN_NAME\: http:\/\/156.2.3.5/g;'  snakeface/settings.yml                                                  
perl -pi -e 's/DOMAIN_PORT\: (5000.*)/DOMAIN_PORT\: 4343/g;'  snakeface/settings.yml                                                                

# confirm changes                                                                                                                                   
grep 'DOMAIN' snakeface/settings.yml

DOMAIN_NAME: http://156.2.3.5
DOMAIN_PORT: 4343

run SF
snakeface

Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
June 06, 2021 - 08:34:12
Django version 3.0.8, using settings 'snakeface.settings'
Starting ASGI/Channels version 3.0.3 development server at http://127.0.0.1:5000/
Quit the server with CONTROL-C.
Listen failure: Couldn't listen on 127.0.0.1:5000: [Errno 98] Address already in use. 

# try env vars                                                                                                                                      
export SNAKEFACE_DOMAIN_HOST="http://156.2.3.5"                                                                                                     
export SNAKEFACE_PORT=4343                                                                                                                          

snakeface

...
Listen failure: Couldn't listen on 127.0.0.1:5000: [Errno 98] Address already in use. 

#  W notebook directive                                                                                                                           
snakeface notebook

...
Listen failure: Couldn't listen on 127.0.0.1:5000: [Errno 98] Address already in use. 

# specify on command line (this is a typo in the getting_started/notebook.html)                                                                     
snakeface notebook --port 4343

...
Listen failure: Couldn't listen on 127.0.0.1:5000: [Errno 98] Address already in use.

snakeface --port 7612 notebook # !SUCCESS (on the port at least)

Starting ASGI/Channels version 3.0.3 development server at http://127.0.0.1:4343/

I'm not really familiar with django, and could not figure out how to get the IP to not be 127.0.0.1, so i ended up setting up port forwarding with socats

sudo socat TCP4-LISTEN:3434,fork TCP4:127.0.0.1:4343 # i may have the ports on the wrong sides....
This worked, I'd get development server at http://127.0.0.1:7612/ and socates routed it such that browsers hitting the server now listening on 7621 brokered between the localhost port and the exposed port.

Awesome. I logged in

chrome-> IP:4343

snakeface server logs:

HTTP GET / 302 [0.05, 127.0.0.1:46852]
Enter token: 8d3509c3-a252-46b3-b5e1-f674c4e2776c

Login, asked to create workflow. Go follow tutorial

from package dir stil 

mkdir snakemake-tutorial                                                                                                                            
cd snakemake-tutorial                                                                                                                               
wget https://github.com/snakemake/snakemake-tutorial-data/archive/v5.24.1.tar.gz                                                                    
tar --wildcards -xf v5.24.1.tar.gz --strip 1 "*/data" "*/environment.yaml"                                                                          

emacs Snakefile #  Cut n paste the example in the docs: example_workflow.html                                                                       

# I want to operate not directly in the data dir                                                                                                    
cd ..                                                                                                                                               
mkdir toot                                                                                                                                          
mv snakemake-tutorial toot                                                                                                                          
cd toot                                                                                                                                             

# now from the snakemake-tutorial dir                                                                                                               
snakeface --port 4343 notebook

chrome-> IP:4343 # don't need to enter a new token, cool

cick create workflow.
! I like the ui a lot !
Choose the toot/snakemake-tutorial dir as my workdir (recall, i am 1 level above it)
set cores to '44'
set conda on
!!! WMS monitor IP:PORT is correct here, but i'm not sure if its from env vars or the yml
enter 'myreport.html' in Report

Hit Run Report

Logs show 

/workflows/command/ 200 [0.12, 127.0.0.1:47052]                                                                                                     
HTTP POST /workflows/command/ 200 [0.04, 127.0.0.1:47052]                                                                                           
Creating workflow                                                                                                                                   
Return value found, stopping.                                                                                                                       
Return value found, stopping.                                                                                                                       
Return value found, stopping.                                                                                                                       
Return value found, stopping.                                                                                                                       
Return value found, stopping.                                                                                                                       
Return value found, stopping.                                                                                                                       
Return value found, stopping.                                                                                                                       
HTTP POST /workflows/new/ 302 [5.64, 127.0.0.1:47052]                                  
Return value found, stopping.                                                                                                                                                                                                                                                                                                                 
Exception in thread Thread-9:                                                                                                                                                                                                                                                                                                                 
Traceback (most recent call last):                                                                                                                                                                                                                                                                                                            
  File "/locus/home/jmajor/conda/envs/SNKFC/lib/python3.8/threading.py", line 932, in _bootstrap_inner                                                                                                                                                                                                                                        
    self.run()                                                                                                                                                                                                                                                                                                                                
  File "/locus/home/jmajor/conda/envs/SNKFC/lib/python3.8/threading.py", line 870, in run                                                                                                                                                                                                                                                     
    self._target(*self._args, **self._kwargs)                                                                                                                                                                                                                                                                                                 
  File "/locus/home/jmajor/projects/pelagic/there_be_dragons/SNKFCE/snkfc/snakeface/snakeface/apps/main/tasks.py", line 147, in doRun                                                                                                                                                                                                         
    env={"WMS_MONITOR_TOKEN": user.token},                                                                                                                                                                                                                                                                                                    
  File "/locus/home/jmajor/projects/pelagic/there_be_dragons/SNKFCE/snkfc/snakeface/snakeface/apps/users/models.py", line 78, in token                                                                                                                                                                                                        
    return str(Token.objects.get(user=self))                                                                                                                                                                                                                                                                                                  
  File "/locus/home/jmajor/projects/pelagic/there_be_dragons/SNKFCE/snkfc/snakeface/env/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method                                                                                                                                                                  
    return getattr(self.get_queryset(), name)(*args, **kwargs)                                                                                                                                                                                                                                                                                
  File "/locus/home/jmajor/projects/pelagic/there_be_dragons/SNKFCE/snkfc/snakeface/env/lib/python3.8/site-packages/django/db/models/query.py", line 415, in get                                                                                                                                                                              
    raise self.model.DoesNotExist(                                                                                                                                                                                                                                                                                                            
rest_framework.authtoken.models.DoesNotExist: Token matching query does not exist.                                                                                                                                                                                                                                                            
HTTP GET /workflows/1/ 200 [0.08, 127.0.0.1:47052]                                                                                                                                                                                                                                                                                            
HTTP GET /static/css/font-awesome.min.css 304 [0.00, 127.0.0.1:47052]                                                                                                                                                                                                                                                                         
HTTP GET /static/css/bootstrap.min.css 304 [0.01, 127.0.0.1:47058]                                                                                                                                                                                                                                                                            
HTTP GET /static/css/demo.css 304 [0.01, 127.0.0.1:47060]

And a browser alert box pops up with this error: 

DataTables warning: table id=taskTable - Requested unknown parameter 'order' for row 0, column 1. For more information about this error, please see http://datatables.net/tn/4

Clicking OK does not clear the alert box. Things seem wedged. The stack trace above is not encouraging. Try restarting. 

/workflows/command/ 200 [0.12, 127.0.0.1:47052]                                                                                                                                                                                                                                                                                               
HTTP POST /workflows/command/ 200 [0.04, 127.0.0.1:47052]                                                                                                                                                                                                                                                                                     
Creating workflow                                                                                                                                                                                                                                                                                                                             
Return value found, stopping.                                                                                                                                                                                                                                                                                                                 
Return value found, stopping.                                                                                                                                                                                                                                                                                                                 
Return value found, stopping.                                                                                                                                                                                                                                                                                                                 
Return value found, stopping.                                                                                                                                                                                                                                                                                                                 
Return value found, stopping.                                                                                                                                                                                                                                                                                                                 
Return value found, stopping.                                                                                                                                                                                                                                                                                                                 
Return value found, stopping.                                                                                                                                                                                                                                                                                                                 
HTTP POST /workflows/new/ 302 [5.64, 127.0.0.1:47052]                                                                                                                                                                                                                                                                                         
Return value found, stopping.                                                                                                                                                                                                                                                                                                                 
Exception in thread Thread-9:                                                                                                                                                                                                                                                                                                                 
Traceback (most recent call last):                                                                                                                                                                                                                                                                                                            
  File "/locus/home/jmajor/conda/envs/SNKFC/lib/python3.8/threading.py", line 932, in _bootstrap_inner                                                                                                                                                                                                                                        
    self.run()                                                                                                                                                                                                                                                                                                                                
  File "/locus/home/jmajor/conda/envs/SNKFC/lib/python3.8/threading.py", line 870, in run                                                                                                                                                                                                                                                     
    self._target(*self._args, **self._kwargs)                                                                                                                                                                                                                                                                                                 
  File "/locus/home/jmajor/projects/pelagic/there_be_dragons/SNKFCE/snkfc/snakeface/snakeface/apps/main/tasks.py", line 147, in doRun                                                                                                                                                                                                         
    env={"WMS_MONITOR_TOKEN": user.token},                                                                                                                                                                                                                                                                                                    
  File "/locus/home/jmajor/projects/pelagic/there_be_dragons/SNKFCE/snkfc/snakeface/snakeface/apps/users/models.py", line 78, in token                                                                                                                                                                                                        
    return str(Token.objects.get(user=self))                                                                                                                                                                                                                                                                                                  
  File "/locus/home/jmajor/projects/pelagic/there_be_dragons/SNKFCE/snkfc/snakeface/env/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method                                                                                                                                                                  
    return getattr(self.get_queryset(), name)(*args, **kwargs)                                                                                                                                                                                                                                                                                
  File "/locus/home/jmajor/projects/pelagic/there_be_dragons/SNKFCE/snkfc/snakeface/env/lib/python3.8/site-packages/django/db/models/query.py", line 415, in get                                                                                                                                                                              
    raise self.model.DoesNotExist(                                                                                                                                                                                                                                                                                                            
rest_framework.authtoken.models.DoesNotExist: Token matching query does not exist.                                                                                                                                                                                                                                                            
HTTP GET /workflows/1/ 200 [0.08, 127.0.0.1:47052]                                                                                                                                                                                                                                                                                            
HTTP GET /static/css/font-awesome.min.css 304 [0.00, 127.0.0.1:47052]                                                                                                                                                                                                                                                                         
HTTP GET /static/css/bootstrap.min.css 304 [0.01, 127.0.0.1:47058]                                                                                                                                                                                                                                                                            
HTTP GET /static/css/demo.css 304 [0.01, 127.0.0.1:47060]

And a browser alert box pops up with: 

DataTables warning: table id=taskTable - Requested unknown parameter 'order' for row 0, column 1. For more information about this error, please see http://datatables.net/tn/4

Clicking OK does not clear the alert box. Things seem wedged. The stack trace above is not encoriaging. Try restarting.

ctrl+C ; snakeface --port 4343 notebk

I hack the url to get back to /
The workflow is in a pending state.
dialog box with same error as above re-appears. seems a js thing as it is causing no logging. but can't clear it
This time just hack the URL don't restart - back to dashboard
Return to the workflow, and click Cancel and confirm.
dialog box reappears, same error msg.
Back to dash, try new workflow. same settings except new report name

Run

Same stack trace logged, same alert box. Hack url back to dashboard.

You might need sit down :-)

I started hacking things out of the code until it worked. What did the trick was 

emacs /snakeface/env/lib/python3.8/site-packages/django/db/models/query.py/snakeface/env/lib/python3.8/site-packages/django/db/models/manager.py

And removed the following 

        if num == 1:                                                                                                                                                                                                                                                                                                                          
            return clone._result_cache[0]                                                                                                                                                                                                                                                                                                     
        # COMMENTED OUT THE FOLLOWING from line 414 on                                                                                                                                                                                                                                                                                        
        # if not num:                                                                                                                                                                                                                                                                                                                         
        #    raise self.model.DoesNotExist(                                                                                                                                                                                                                                                                                                   
        #        "%s matching query does not exist." %                                                                                                                                                                                                                                                                                        
        #        self.model._meta.object_name                                                                                                                                                                                                                                                                                                 
        #    )                                                                                                                                                                                                                                                                                                                                
        #raise self.model.MultipleObjectsReturned(                                                                                                                                                                                                                                                                                            
        #    'get() returned more than one %s -- it returned %s!' % (                                                                                                                                                                                                                                                                         
        #        self.model._meta.object_name,                                                                                                                                                                                                                                                                                                
        #        num if not limit or num < limit else 'more than %s' % (limit - 1),                                                                                                                                                                                                                                                           
        #    )                                                                                                                                                                                                                                                                                                                                
        #)

Run again

Load the first WF, cancel it 3x, then click edit, change the name, and re-run
Progress! The stacktrace is gone :-|
The dag appears.... But the alert box is back again with the same message.
Hack url back to dashboard. The workflow has now gone from 'pending' to 'error' state. That is new.

THE WORKFLOW has an error now:

Authorization is required with a WMS_MONITOR_TOKEN in the environment

I go edit the settings.yml to turn off auth. This works. 

perl -pi -e 's/REQUIRE_AUTH: (true.*)/REQUIRE_AUTH: false/g;' ../snakeface/settings.yml

REQUIRE_AUTH: false

restart, try re-running the first workflow. It runs, but new error: 

Building DAG of jobs...                                                                                                                                                                                                                                                                                                                       
MissingInputException in line 8 of /locus/home/jmajor/projects/pelagic/there_be_dragons/SNKFCE/snkfc/snakeface/toot/snakemake-tutorial/Snakefile:                                                                                                                                                                                             
Missing input files for rule bwa_map:                                                                                                                                                                                                                                                                                                         
data/genome.fa                                                                                                                                                                                                                                                                                                                                
data/samples/A.fastq

Suspect the server needs to be running from the snakemake execution dir.... stop snakeface, cd snakemake-tutorial, restart
Try again 

cd snakemake-tutorial                                                                                                                                                                                                                                                                                                                         
snakeface --port 4343 notebook

Go to first workflow, re-run.
Success! The exhaustive audit trail is there, the dag is there. It's looking really slick

Click on 'View Report'... it loads the interactive DAG plot! slick.

Somehow the alert box error has stopped showing up...

I'm still curious about the working dir thing, restart with workdir set. I'll add verbosity too 

cd ..                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
snakeface --workdir ~/projects/pelagic/there_be_dragons/SNKFCE/snkfc/snakeface/toot --verbosity 2 --port 4343 notebook                                                                                                                                                                                                                        
usage: snakeface [-h] [--version] [--noreload] [--verbosity {0,1,2,3}] [--workdir [WORKDIR]] [--auth {token}]                                                                                                                                                                                                                                 
                 [--port PORT] [--quiet] [--verbose] [--log-disable-color] [--log-use-threads]                                                                                                                                                                                                                                                
                 {notebook} ...                                                                                                                                                                                                                                                                                                               
snakeface: error: argument --verbosity: invalid choice: '2' (choose from 0, 1, 2, 3)                                                                                                                                                                                                                                                          
```                                                                                                                                                                                                                                                                                                                                           0,1,2,3 do not work.  --verbose worked                                                                                                                                                                                                                                                                                                      

Ok, just workdir                                                                                                                                                                                                                                                                                                                            

snakeface --workdir ~/projects/pelagic/there_be_dragons/SNKFCE/snkfc/snakeface/toot --port 4343 notebook                                                                                                                                                                                                                                      

back to dashboard. create new WF, set work dir to ./snakemake-tutorial                                                                                                                                                                                                                                                                      
same error, can't find the data....  move back, confirm

cd snakemake-tutorial
snakeface --workdir ~/projects/pelagic/there_be_dragons/SNKFCE/snkfc/snakeface/toot --port 4343 notebook 



Edited the WF to have the tutorial dir as work dir. The test ran, with report generated and all.                                                                                                                                                                                                                                            

I then tried running manual snakepipes from other locations on my server, pointing to the wms-url:port.... and things sort of worked!  
The remote WFs were created and logged.  I just had to guess their # and hack the URL to see the details.  I did a lot more crude chopping out auth and other stuff to get them visible from the dashboard and just get a sense for what did what.  My use case is definitely in a multi-user scenario, and the bones look really promising.  I hope this was helpful.

 This is a project I'm super excited by and look forward to helping where I can. Thank you-- jem
vsoch commented 3 years ago

Wow there is a lot here - thank you for testing so thoroughly! I am hopeful I'll be able to fix these bugs without needing to edit Django source code - I'm going out for a morning adventure and will walk through this carefully when I get back!

vsoch commented 3 years ago

okay I've thought about this a bit - I think we need a different design where the server settings and static are kept in the user HOME, possibly a ~/snakeface directory. The reason is because we should be able to just install with pip, and have the same software shared/used by multiple users (and also not rely on needing write where we install it). My one question for you (which led to most of your error and debugging) is what did you already have running on port 5000 that you couldn't use it?

vsoch commented 3 years ago

hey @iamh2o ! I've refactored the setup so that settings will now exist in your user home, along with all content that needs to be written (sqlite database, static files, migrations) and I have a PR for you to test here: https://github.com/snakemake/snakeface/pull/22. I think most of your issues above resulted from the fact that the settings weren't being read/honored, and possibly the notebook was disabled, and it led you down a long trail of issues (note that if you get to the point where you are editing Django source code please ask for help first!). I'm not sure why your port 5000 was busy, but you should be able to set a port in the settings or just use --port to specify a different one (at least I just tested and that worked for me!)

iamh2o commented 3 years ago

Hi @vsoch -- Yow, what a quick response. Thanks for taking a whack at this so soon, I'll give the PR a shot in the next few days. The port was not the main problem, but just part of the story which was 'some of the settings were not being picked up'. The port I was able to change on the command line, but being able to change the IP address to something other than 127.0.0.1 I could not for the life of me figure out how to do - and that was the bigger problem.
I'm running on a compute node in some server bunker where a lot of folks do computational work, so no one is local w/r/t the box. being able to set anything other than localhost or 127* is the only way to expose apps for us as no one is local. Usually when I play around with apps offering UIs there is an option to set 0.0.0.0, but ideally to set any IP. My solution was to reroute the traffic inbound to localhost and vice-versa, but there must be a django flag that allows that change (I'm a cherrypy person, it's a settings option there). And what was running on port 5000? I'm not sure, but there are a surprising number of ports held on this machine. I'm usually up in the 7-8000s.

I want to be sure it was clear, once I rerouted the port traffic, i was able to log in. Then once logged in (and authenticated with a token and all that jazz), I encountered the erorrs that required hacking the django code (which im not shy about doing, it was certainly not a long term solution, just a way to see how much further I could get. I'll definitely let you know how things go with the fixes, it's incredibly good timing b/c I work with a pack of data crazed Biologists who would be so psyched to be able to run their own workflows.

thanks again--jem