Python script works on command line but not in Wooey

[razermouse]

I've got a non-technical employee that needs access to Python scripts that we need to use to conduct our business. I've gotten Wooey working but one of the scripts breaks. I love this concept - my employee doesn't understand Python, Linux, or any other myriad of aspects involved with running this script. So, if I can get this working, it'll be awesome.

The script is called dnstwist.py (https://github.com/elceef/dnstwist/blob/master/dnstwist.py). In the Wooey console, it generates the following errors:

Traceback (most recent call last): File "/home/ubuntu/twisty/twisty/user_uploads/wooey_scripts/dnstwist.py", line 1009, in main() File "/home/ubuntu/twisty/twisty/user_uploads/wooey_scripts/dnstwist.py", line 1003, in main p_cli(generate_cli(domains)) File "/home/ubuntu/twisty/twisty/user_uploads/wooey_scripts/dnstwist.py", line 120, in p_cli sys.stdout.write(data) UnicodeEncodeError: 'ascii' codec can't encode characters in position 3628-3629: ordinal not in range(128)

I'm pretty sure it has to do with how the data is formatted in 3 columns in the output but I don't know how to fix it. Any help would be greatly appreciated.

[Chris7]

Hi @razermouse,

That is because the host is configured to use ascii but unicode is being printed. You could use python3 or maybe set the system encoding to utf-8. This may help: https://perlgeek.de/en/article/set-up-a-clean-utf8-environment

[razermouse]

I looked at the python script and he actually uses IDNA.

IDNA specifies how the conversion between names written in non-ASCII characters and their ASCII-based representation is performed. So, for example: ƿàypàl.com is a valid domain.

I tried to start Wooey using Python3 and it broke. I also tried the system encoding mentioned in the article but if it's not using the IDNA encoding then it doesn't list as many potential domains. Anyway, maybe I'll try to see if I can get Python3 working but I don't think that will fix it.

[razermouse]

Does anyone have any ideas on how I could fix this?

[Chris7]

How would i use IDNA to test the script?

[razermouse]

Hi Chris,

I can't figure out what library the author calls to use the encode/decode function.

This is the source for the script itself: https://github.com/elceef/dnstwist

I did find a couple of resources about idna:

https://pypi.org/project/idna/ https://docs.python.org/2.4/lib/module-encodings.idna.html

[Chris7]

I think it's how the machine is configured still. Here's reproducing the error when the LANG setting is incorrect:

$ export LANG=NONE
$ python
Python 2.7.15rc1 (default, Nov 12 2018, 14:31:15) 
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import idna
>>> print idna.decode('xn--eckwd4c7c.xn--zckzah')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-3: ordinal not in range(128)
>>> 
$ export LANG=en_US.UTF-8
$ python
Python 2.7.15rc1 (default, Nov 12 2018, 14:31:15) 
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import idna
>>> print idna.decode('xn--eckwd4c7c.xn--zckzah')
ドメイン.テスト

[razermouse]

Ok, I emailed the author of dnstwist.py and he says to run it with Python 3.

Is it possible for me to switch to Python 3 easily or (this is what I suspect) do I have to reinstall Wooey using Python 3?

[Chris7]

Wooey is python2/3 compatible, so it should be easy to do. If you haven't customized anything, you should be able to simply install wooey in a python3 environment and everything will just work.