Running DDV2 in Windows

[photomedia]

I've been trying to get DDV2 running on Windows, but after installing Python, PIP, and GIT (these should be identified in the getting started documentation as required), I still get this error when trying to run any command:

Setting up Python... Running in: C:\Python\Scripts Traceback (most recent call last): File "DDV2.py", line 46, in from DDV.ChainParser import ChainParser File "C:\Python\lib\site-packages\DDV\ChainParser.py", line 10, in from DNASkittleUtils.DDVUtils import first_word, Batch, ReverseComplement, BlankIterator, editable_str ImportError: cannot import name 'editable_str'

[josiahseaman]

Hey, thanks for bringing this up! That is definitely because I forgot to update the dependency to DNASkittleUtils to 1.0.8

https://github.com/josiahseaman/DNASkittleUtils/releases/tag/1.0.8

pip install git+https://github.com/josiahseaman/DNASkittleUtils.git@1.0.8 --upgrade should fix the problem. The --upgrade is necessary to force pip to ignore the version number. I'll look into this more. Please let me know if that fixes it.

[photomedia]

Thank you, I ran the command, and the error is different now, so progress :)

C:\Python\Scripts>python DDV2.py --fasta="C:\DATA\ddv2data\sequence.fasta" Setting up Python... Running in: C:\Python\Scripts Traceback (most recent call last): File "DDV2.py", line 48, in from DDV.AnnotatedAlignment import AnnotatedAlignment File "C:\Python\lib\site-packages\DDV\AnnotatedAlignment.py", line 4, in from ChainParser import ChainParser, scan_past_header ModuleNotFoundError: No module named 'ChainParser'

[josiahseaman]

Okay, I've pushed a fix to remove all relative imports from the project. I wasn't able to replicate the error on my machine however. What's your python version?

You'll likely need to pip uninstall and reinstall DDV. python-master is my development branch, so it would be: pip install git+https://github.com/josiahseaman/DDV.git@python-master for the new version. This is going to change the main script name to DDV/fluentdna.py instead of DDV/DDV2.py

I version bumped DNASkittleUtils just to be certain.

pip install git+https://github.com/josiahseaman/DNASkittleUtils.git@1.0.9 --upgrade

[photomedia]

OK, so now I get this:

C:\DATA\ddv2data>fluentdna.py --fasta="C:\DATA\ddv2data\sequence.fasta" Setting up Python... Running in: C:\Python\Scripts Creating Chromosome Output Directory... sequence Creating Large Image from Input Fasta... Read contigs : 0:00:00.044114 Initialized Image: 0:00:00.053137

98.557 % done: gi|384511964|ref|NC_017316.1| Enterococcus faecalis OG1RF, complete genome

Drew Nucleotides: 0:00:02.184971 -- Writing: sequence -- Output Image in: 0:00:03.179348 Done creating Large Image at C:\Python\lib\site-packages\DDV\www-data\dnadata\sequence\sequence.png Error while generating HTML:

Traceback (most recent call last): File "C:\Python\lib\site-packages\DDV\TileLayout.py", line 360, in generate_html copytree(os.path.join(os.getcwd(), 'html_template'), output_folder) # copies the whole template directory File "C:\Python\lib\site-packages\DNASkittleUtils\DDVUtils.py", line 66, in copytree for item in os.listdir(src): FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:\Python\Scripts\html_template' Creating Deep Zoom Structure from Generated Image... Traceback (most recent call last): File "C:\Python\Scripts\fluentdna.py", line 478, in main() File "C:\Python\Scripts\fluentdna.py", line 474, in main ddv(args) File "C:\Python\Scripts\fluentdna.py", line 161, in ddv create_tile_layout_viz_from_fasta(args, args.fasta, output_dir, args.output_name) File "C:\Python\Scripts\fluentdna.py", line 289, in create_tile_layout_viz_from_fasta create_deepzoom_stack(os.path.join(output_dir, layout_final_output_location), os.path.join(output_dir, 'GeneratedImages', "dzc_output.xml")) File "C:\Python\lib\site-packages\DDV\DDVUtils.py", line 59, in create_deepzoom_stack import deepzoom ModuleNotFoundError: No module named 'deepzoom'

I end up with a PNG, but I don't know if those error messages are supposed to be there, probably not :)

[josiahseaman]

We're in the home stretch now 👍 dd6231c05ec026abd7eb54968cbd5feef3f2d05c just changes it to import DDV.deepzoom

pip install git+https://github.com/josiahseaman/DDV.git@python-master --upgrade

Or you could just change that by hand. Looking through all imports in the project, I can't spot any others that would need to be changed.

[photomedia]

I ran pip install git+https://github.com/josiahseaman/DDV.git@python-master --upgrade

but I still get this:

C:\DATA\ddv2data>fluentdna.py --fasta="C:\DATA\ddv2data\sequence.fasta" Setting up Python... Running in: C:\Python\Scripts Creating Chromosome Output Directory... sequence Creating Large Image from Input Fasta... Read contigs : 0:00:00.033106 Initialized Image: 0:00:00.052572

98.557 % done: gi|384511964|ref|NC_017316.1| Enterococcus faecalis OG1RF, complete genome

Drew Nucleotides: 0:00:02.250522 -- Writing: sequence -- Output Image in: 0:00:03.235326 Done creating Large Image at C:\Python\lib\site-packages\DDV\www-data\dnadata\sequence\sequence.png Error while generating HTML:

Traceback (most recent call last): File "C:\Python\lib\site-packages\DDV\TileLayout.py", line 360, in generate_html copytree(os.path.join(os.getcwd(), 'html_template'), output_folder) # copies the whole template directory File "C:\Python\lib\site-packages\DNASkittleUtils\DDVUtils.py", line 66, in copytree for item in os.listdir(src): FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:\Python\Scripts\html_template' Creating Deep Zoom Structure from Generated Image... Traceback (most recent call last): File "C:\Python\Scripts\fluentdna.py", line 478, in main() File "C:\Python\Scripts\fluentdna.py", line 474, in main ddv(args) File "C:\Python\Scripts\fluentdna.py", line 161, in ddv create_tile_layout_viz_from_fasta(args, args.fasta, output_dir, args.output_name) File "C:\Python\Scripts\fluentdna.py", line 289, in create_tile_layout_viz_from_fasta create_deepzoom_stack(os.path.join(output_dir, layout_final_output_location), os.path.join(output_dir, 'GeneratedImages', "dzc_output.xml")) File "C:\Python\lib\site-packages\DDV\DDVUtils.py", line 64, in create_deepzoom_stack creator = deepzoom.ImageCreator(tile_size=dz_params['tile_size'], NameError: name 'deepzoom' is not defined

[josiahseaman]

TL;DR Summary: I pushed another fix to master, see if that does any better. The test we want to see pass is after you generate a webpage you should be able to type python fluentdna.py --runserver and point chome at localhost:8000 and get a full webpage.

I'm sorry I haven't been able to reproduce this path problem on my machine, but I do think we're in the home stretch. The last fix is easy, it's just import DDV.deepzoom as deepzom to get the names to match. But the error with html_template is much more difficult...

os.getcwd() "get current working directory is returning the python environment, rather than the DDV module. That makes a certain sense.

https://stackoverflow.com/questions/247770/retrieving-python-module-path

import DDV
            module_path = os.path.dirname(DDV.__file__)
            copytree(os.path.join(module_path, 'html_template'), output_folder)

Fixes that problem by referencing the absolute path of the module. Hopefully, that doesn't trip on any permissioning or sandboxes to use absolute file paths.

[photomedia]

I still get the same error "deepzoom not defined" as before, after upgrading. This is the output for the runserver command:

C:\Python\Scripts>python fluentdna.py --runserver Setting up Python... Running in: C:\Python\Scripts Setting up HTTP Server based from C:\Python\lib\site-packages\DDV\www-data\dnadata Serving at 127.0.0.1 on port 8000 127.0.0.1 - - [24/Jan/2018 12:29:47] "GET /sequence/ HTTP/1.1" 200 - 127.0.0.1 - - [24/Jan/2018 12:29:48] "GET /sequence/seadragon.css HTTP/1.1" 200 - 127.0.0.1 - - [24/Jan/2018 12:29:48] "GET /sequence/openseadragon.js HTTP/1.1" 200 - 127.0.0.1 - - [24/Jan/2018 12:29:48] "GET /sequence/jquery-1.7.min.js HTTP/1.1" 200 - 127.0.0.1 - - [24/Jan/2018 12:29:48] "GET /sequence/openseadragon-scalebar.js HTTP/1.1" 200 - 127.0.0.1 - - [24/Jan/2018 12:29:48] "GET /sequence/nucleotideNumber.js HTTP/1.1" 200 - 127.0.0.1 - - [24/Jan/2018 12:29:48] "GET /sequence/img/LEGEND-A.png HTTP/1.1" 200 - 127.0.0.1 - - [24/Jan/2018 12:29:48] "GET /sequence/img/LEGEND-T.png HTTP/1.1" 200 - 127.0.0.1 - - [24/Jan/2018 12:29:48] "GET /sequence/img/LEGEND-G.png HTTP/1.1" 200 - 127.0.0.1 - - [24/Jan/2018 12:29:48] "GET /sequence/img/LEGEND-C.png HTTP/1.1" 200 - 127.0.0.1 - - [24/Jan/2018 12:29:48] "GET /sequence/img/LEGEND-N.png HTTP/1.1" 200 - 127.0.0.1 - - [24/Jan/2018 12:29:48] "GET /sequence/img/LEGEND-bg.png HTTP/1.1" 200 - 127.0.0.1 - - [24/Jan/2018 12:29:49] code 404, message File not found 127.0.0.1 - - [24/Jan/2018 12:29:49] "GET /sequence/GeneratedImages/dzc_output.xml HTTP/1.1" 404 - 127.0.0.1 - - [24/Jan/2018 12:29:49] "GET /sequence/img/zoomout_pressed.png HTTP/1.1" 200 - 127.0.0.1 - - [24/Jan/2018 12:29:49] "GET /sequence/img/home_grouphover.png HTTP/1.1" 200 - 127.0.0.1 - - [24/Jan/2018 12:29:49] "GET /sequence/img/fullpage_pressed.png HTTP/1.1" 200 - 127.0.0.1 - - [24/Jan/2018 12:29:49] "GET /sequence/img/zoomout_hover.png HTTP/1.1" 200 - 127.0.0.1 - - [24/Jan/2018 12:29:49] "GET /sequence/img/fullpage_grouphover.png HTTP/1.1" 200 - 127.0.0.1 - - [24/Jan/2018 12:29:49] "GET /sequence/img/fullpage_rest.png HTTP/1.1" 200 - 127.0.0.1 - - [24/Jan/2018 12:29:49] "GET /sequence/img/fullpage_hover.png HTTP/1.1" 200 - 127.0.0.1 - - [24/Jan/2018 12:29:49] "GET /sequence/img/home_rest.png HTTP/1.1" 200 - 127.0.0.1 - - [24/Jan/2018 12:29:49] "GET /sequence/img/home_pressed.png HTTP/1.1" 200 - 127.0.0.1 - - [24/Jan/2018 12:29:49] "GET /sequence/img/home_hover.png HTTP/1.1" 200 - 127.0.0.1 - - [24/Jan/2018 12:29:49] "GET /sequence/img/zoomin_rest.png HTTP/1.1" 200 - 127.0.0.1 - - [24/Jan/2018 12:29:49] "GET /sequence/img/zoomin_grouphover.png HTTP/1.1" 200 - 127.0.0.1 - - [24/Jan/2018 12:29:49] "GET /sequence/img/zoomin_hover.png HTTP/1.1" 200 - 127.0.0.1 - - [24/Jan/2018 12:29:49] "GET /sequence/img/zoomin_pressed.png HTTP/1.1" 200 - 127.0.0.1 - - [24/Jan/2018 12:29:49] "GET /sequence/img/zoomout_rest.png HTTP/1.1" 200 - 127.0.0.1 - - [24/Jan/2018 12:29:49] "GET /sequence/img/zoomout_grouphover.png HTTP/1.1" 200 -

So that sort of works, but it is unable to display the visualization, because the image stack is not found.

I think it is worthwhile to get the "quickstart" instructions on Windows working as smoothly as possible, because spending a lot of time on startup errors is something that most biologists wouldn't have the time for.

[josiahseaman]

I couldn't agree more. My intention was to make sure it installed on any computer with a single command. The two errors were actually caused by an embarrassingly simple typo. I found a way to reproduce it and I've pushed a fix to the pip branch. I've updated the readme to reflect that you need to provide the full path to DDV/fluentdna.py Thanks for all your help.

Here's an end to end example:

C:\Users\josiah>cd env_test
C:\Users\josiah\env_test>Scripts\activate
(env_test) C:\Users\josiah\env_test>pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
pip (9.0.1)
setuptools (38.4.0)
wheel (0.30.0)

(env_test) C:\Users\josiah\env_test>pip install --process-dependency-links git+https://github.com/josiahseaman/DDV.git@pip

Collecting git+https://github.com/josiahseaman/DDV.git@pip
  Cloning https://github.com/josiahseaman/DDV.git (to pip) to c:\users\josiah\appdata\local\temp\pip-blv5zz-build
  DEPRECATION: Dependency Links processing has been deprecated and will be removed in a future release.
Collecting Pillow>=3.2.0 (from DDV==2.2.1)
  Using cached Pillow-5.0.0-cp27-cp27m-win_amd64.whl
Collecting six>=1.10.0 (from DDV==2.2.1)
  Using cached six-1.11.0-py2.py3-none-any.whl
Collecting psutil>=4.3.1 (from DDV==2.2.1)
  Using cached psutil-5.4.3-cp27-none-win_amd64.whl
Collecting blist>=1.3.6 (from DDV==2.2.1)
Collecting natsort>=5.1.1 (from DDV==2.2.1)
  Using cached natsort-5.1.1-py2.py3-none-any.whl
Collecting DNASkittleUtils>=1.0.9 (from DDV==2.2.1)
  Cloning https://github.com/josiahseaman/DNASkittleUtils.git (to 1.0.9) to c:\users\josiah\appdata\local\temp\pip-build-fvmbco\dnaskittleutils
Installing collected packages: Pillow, six, psutil, blist, natsort, DNASkittleUtils, DDV
  Running setup.py install for DNASkittleUtils ... done
  Running setup.py install for DDV ... done
Successfully installed DDV-2.2.1 DNASkittleUtils-1.0.9 Pillow-5.0.0 blist-1.3.6 natsort-5.1.1 psutil-5.4.3 six-1.11.0

(env_test) C:\Users\josiah\env_test>python C:\Users\josiah\env_test\Lib\site-packages\DDV\fluentdna.py --fasta="D:\Genomes\Arabidopsis 10\TAIR10_chr5.fas"
Setting up Python...
Running in: C:\Users\josiah\env_test\Lib\site-packages\DDV
Creating Chromosome Output Directory... TAIR10_chr5
Creating Large Image from Input Fasta...
Read contigs : 0:00:00.279000
Initialized Image: 0:00:00.333000

 99.720 % done: Chr5 CHROMOSOME dumped from ADB: Jun/20/09 14:54; last updated: 2009-02-02

Drew Nucleotides: 0:00:13.352000
-- Writing: TAIR10_chr5 --
Output Image in: 0:00:22.577000
Done creating Large Image at  C:\Users\josiah\env_test\lib\site-packages\DDV\www-data\dnadata\TAIR10_chr5\TAIR10_chr5.png
Creating Deep Zoom Structure from Generated Image...
Done creating Deep Zoom Structure.

(env_test) C:\Users\josiah\env_test>python C:\Users\josiah\env_test\Lib\site-packages\DDV\fluentdna.py --runserver
Setting up Python...
Running in: C:\Users\josiah\env_test\Lib\site-packages\DDV
Setting up HTTP Server based from C:\Users\josiah\env_test\lib\site-packages\DDV\www-data\dnadata
Serving at 127.0.0.1 on port 8000
127.0.0.1 - - [25/Jan/2018 00:29:03] "GET /TAIR10_chr5/ HTTP/1.1" 200 -
127.0.0.1 - - [25/Jan/2018 00:29:03] "GET /TAIR10_chr5/openseadragon.js HTTP/1.1" 200 -
127.0.0.1 - - [25/Jan/2018 00:29:03] "GET /TAIR10_chr5/jquery-1.7.min.js HTTP/1.1" 200 -
127.0.0.1 - - [25/Jan/2018 00:29:03] "GET /TAIR10_chr5/openseadragon-scalebar.js HTTP/1.1" 200 -
127.0.0.1 - - [25/Jan/2018 00:29:03] "GET /TAIR10_chr5/nucleotideNumber.js HTTP/1.1" 200 -
127.0.0.1 - - [25/Jan/2018 00:29:03] "GET /TAIR10_chr5/seadragon.css HTTP/1.1" 200 -
127.0.0.1 - - [25/Jan/2018 00:29:03] "GET /TAIR10_chr5/img/LEGEND-A.png HTTP/1.1" 200 -
127.0.0.1 - - [25/Jan/2018 00:29:03] "GET /TAIR10_chr5/img/LEGEND-T.png HTTP/1.1" 200 -
127.0.0.1 - - [25/Jan/2018 00:29:03] "GET /TAIR10_chr5/img/LEGEND-G.png HTTP/1.1" 200 -

[photomedia]

Thank you, after running upgrade, I was able to generate visualizations on Windows 10 with a touchscreen. I had to install: PIP, Python, GIT, and Visual Studio (for a required runtime library?). It is a great accomplishment to port DDV to a multi-platform environment.

[josiahseaman]

Thanks. It's good to know that the dependencies work from scratch. I was about to open a second issue about the Visual Studio dependency in Python 3.5+. Can you confirm you're running Python 3.6? That's the only place I ran into that issue. python --version

Pip install is the preferred method for developers. For non-developers, I'd probably point them to the unzip - install here: https://github.com/josiahseaman/FluentDNA/releases/tag/DDV_2.0_Pre
I need to update that with the latest features and compile for Mac as well.

[photomedia]

Yes, I was running with Python 3.6.4.