EzhilFileExecuter get_output returns empty data

[kracekumar]

I have django view which calls following function to execute the code.

def execute(code):
    with tempfile.NamedTemporaryFile() as tmp_file:
        tmp_file.write(code.encode('utf-8'))
        tmp_file.flush()
        print("Input file: {0}".format(tmp_file.name))
        TIMEOUT = getattr(settings, 'TIMEOUT', 10)
        obj = EzhilFileExecuter(file_input=tmp_file.name, redirectop=True,
                                TIMEOUT=TIMEOUT, debug=False)
        obj.run()
        res = obj.get_output()
        print(res)
        return res

The view properly decodes the json data and output is

Input file: /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmp7J6mbD
['/var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpLn5eUL.output', '/var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/Tezhil_21358.out', '']
.... terminate!!! ....

The code value is u'\nz = dict()\n\u0bb5\u0bc8(z, "name", "Tamil")\n\u0baa\u0ba4\u0bbf\u0baa\u0bcd\u0baa\u0bbf keys(z)\n'

The api is called using requests.

In [130]: r = requests.post("http://127.0.0.1:8000/api/", json={'code': code})

In [131]: code
Out[131]:u'\nz = dict()\n\u0bb5\u0bc8(z, "name", "Tamil")\n\u0baa\u0ba4\u0bbf\u0baa\u0bcd\u0baa\u0bbf keys(z)\n'

Here is the output when debug=True

Input file: /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF
ezhil file executer
EzhilFileExecuter - entering the redirect mode
begin redirect mode
timeout non-zero
in busy loop : 0 , 10
SLEEP
['/var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpDLuEoA.output', '/var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/Tezhil_39332.out', 'init/EzhilLex\n(u\'-> opening file %s\', \'/var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF\')\npost file open\nStart of Ezhil lexer - begin tokenize\n\nz = dict()\n\xe0\xae\xb5\xe0\xaf\x88(z, "name", "Tamil")\n\xe0\xae\xaa\xe0\xae\xa4\xe0\xae\xbf\xe0\xae\xaa\xe0\xaf\x8d\xe0\xae\xaa\xe0\xae\xbf keys(z)\n\n(0, u\'\\n\')\n(1, u\'z\')\n(u\'get_lexeme\', u\'z\', 1)\n(u\'Lexer token = \', <ezhil.ezhil_scanner.EzhilLexeme instance at 0x10604e200>)\n(2, u\' \')\n(3, u\'=\')\n(u\'get_lexeme\', u\'=\', 3)\n(u\'Lexer token = \', <ezhil.ezhil_scanner.EzhilLexeme instance at 0x10604e248>)\n(4, u\' \')\n(5, u\'d\')\n(u\'get_lexeme\', u\'dict\', 5)\n(u\'Lexer token = \', <ezhil.ezhil_scanner.EzhilLexeme instance at 0x10604e2d8>)\n(9, u\'(\')\n(u\'get_lexeme\', u\'(\', 9)\n(u\'Lexer token = \', <ezhil.ezhil_scanner.EzhilLexeme instance at 0x10604e290>)\n(10, u\')\')\n(u\'get_lexeme\', u\')\', 10)\n(u\'Lexer token = \', <ezhil.ezhil_scanner.EzhilLexeme instance at 0x10604e368>)\n(11, u\'\\n\')\n(12, u\'\\u0bb5\')\n(u\'get_lexeme\', u\'\\u0bb5\\u0bc8\', 12)\n(u\'Lexer token = \', <ezhil.ezhil_scanner.EzhilLexeme instance at 0x10604e320>)\n(14, u\'(\')\n(u\'get_lexeme\', u\'(\', 14)\n(u\'Lexer token = \', <ezhil.ezhil_scanner.EzhilLexeme instance at 0x10604e3b0>)\n(15, u\'z\')\n(u\'get_lexeme\', u\'z\', 15)\n(u\'Lexer token = \', <ezhil.ezhil_scanner.EzhilLexeme instance at 0x10604e3f8>)\n(16, u\',\')\n(u\'get_lexeme\', u\',\', 16)\n(u\'Lexer token = \', <ezhil.ezhil_scanner.EzhilLexeme instance at 0x10604e440>)\n(17, u\' \')\n(18, u\'"\')\n(u\'get_lexeme\', u\'"name"\', 18)\n(u\'Lexer token = \', <ezhil.ezhil_scanner.EzhilLexeme instance at 0x10604e488>)\n(24, u\',\')\n(u\'get_lexeme\', u\',\', 24)\n(u\'Lexer token = \', <ezhil.ezhil_scanner.EzhilLexeme instance at 0x10604e4d0>)\n(25, u\' \')\n(26, u\'"\')\n(u\'get_lexeme\', u\'"Tamil"\', 26)\n(u\'Lexer token = \', <ezhil.ezhil_scanner.EzhilLexeme instance at 0x10604e518>)\n(33, u\')\')\n(u\'get_lexeme\', u\')\', 33)\n(u\'Lexer token = \', <ezhil.ezhil_scanner.EzhilLexeme instance at 0x10604e560>)\n(34, u\'\\n\')\n(35, u\'\\u0baa\')\n(u\'get_lexeme\', u\'\\u0baa\\u0ba4\\u0bbf\\u0baa\\u0bcd\\u0baa\\u0bbf\', 35)\n(u\'Lexer token = \', <ezhil.ezhil_scanner.EzhilLexeme instance at 0x10604e5a8>)\n(42, u\' \')\n(43, u\'k\')\n(u\'get_lexeme\', u\'keys\', 43)\n(u\'Lexer token = \', <ezhil.ezhil_scanner.EzhilLexeme instance at 0x10604e5f0>)\n(47, u\'(\')\n(u\'get_lexeme\', u\'(\', 47)\n(u\'Lexer token = \', <ezhil.ezhil_scanner.EzhilLexeme instance at 0x10604e638>)\n(48, u\'z\')\n(u\'get_lexeme\', u\'z\', 48)\n(u\'Lexer token = \', <ezhil.ezhil_scanner.EzhilLexeme instance at 0x10604e680>)\n(49, u\')\')\n(u\'get_lexeme\', u\')\', 49)\n(u\'Lexer token = \', <ezhil.ezhil_scanner.EzhilLexeme instance at 0x10604e6c8>)\n(50, u\'\\n\')\nbefore reverse\n  [ - 0] Line=5, Col=1 in File   \n ) [) - 9] Line=4, Col=15 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n z [z - 1] Line=4, Col=14 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n ( [( - 8] Line=4, Col=13 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n keys [keys - 1] Line=4, Col=9 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n \xe0\xae\xaa\xe0\xae\xa4\xe0\xae\xbf\xe0\xae\xaa\xe0\xaf\x8d\xe0\xae\xaa\xe0\xae\xbf [\xe0\xae\xaa\xe0\xae\xa4\xe0\xae\xbf\xe0\xae\xaa\xe0\xaf\x8d\xe0\xae\xaa\xe0\xae\xbf - 3] Line=4, Col=1 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n ) [) - 9] Line=3, Col=22 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n Tamil [Tamil - 21] Line=3, Col=15 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n , [, - 10] Line=3, Col=13 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n name [name - 21] Line=3, Col=7 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n , [, - 10] Line=3, Col=5 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n z [z - 1] Line=3, Col=4 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n ( [( - 8] Line=3, Col=3 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n \xe0\xae\xb5\xe0\xaf\x88 [\xe0\xae\xb5\xe0\xaf\x88 - 1] Line=3, Col=1 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n ) [) - 9] Line=2, Col=10 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n ( [( - 8] Line=2, Col=9 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n dict [dict - 1] Line=2, Col=5 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n = [= - 11] Line=2, Col=3 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n z [z - 1] Line=2, Col=1 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF \nafter reverse\n z [z - 1] Line=2, Col=1 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n = [= - 11] Line=2, Col=3 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n dict [dict - 1] Line=2, Col=5 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n ( [( - 8] Line=2, Col=9 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n ) [) - 9] Line=2, Col=10 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n \xe0\xae\xb5\xe0\xaf\x88 [\xe0\xae\xb5\xe0\xaf\x88 - 1] Line=3, Col=1 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n ( [( - 8] Line=3, Col=3 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n z [z - 1] Line=3, Col=4 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n , [, - 10] Line=3, Col=5 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n name [name - 21] Line=3, Col=7 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n , [, - 10] Line=3, Col=13 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n Tamil [Tamil - 21] Line=3, Col=15 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n ) [) - 9] Line=3, Col=22 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n \xe0\xae\xaa\xe0\xae\xa4\xe0\xae\xbf\xe0\xae\xaa\xe0\xaf\x8d\xe0\xae\xaa\xe0\xae\xbf [\xe0\xae\xaa\xe0\xae\xa4\xe0\xae\xbf\xe0\xae\xaa\xe0\xaf\x8d\xe0\xae\xaa\xe0\xae\xbf - 3] Line=4, Col=1 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n keys [keys - 1] Line=4, Col=9 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n ( [( - 8] Line=4, Col=13 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n z [z - 1] Line=4, Col=14 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n ) [) - 9] Line=4, Col=15 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n  [ - 0] Line=5, Col=1 in File  \n####### dump tokens ########\n z [z - 1] Line=2, Col=1 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n = [= - 11] Line=2, Col=3 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n dict [dict - 1] Line=2, Col=5 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n ( [( - 8] Line=2, Col=9 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n ) [) - 9] Line=2, Col=10 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n \xe0\xae\xb5\xe0\xaf\x88 [\xe0\xae\xb5\xe0\xaf\x88 - 1] Line=3, Col=1 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n ( [( - 8] Line=3, Col=3 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n z [z - 1] Line=3, Col=4 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n , [, - 10] Line=3, Col=5 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n name [name - 21] Line=3, Col=7 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n , [, - 10] Line=3, Col=13 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n Tamil [Tamil - 21] Line=3, Col=15 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n ) [) - 9] Line=3, Col=22 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n \xe0\xae\xaa\xe0\xae\xa4\xe0\xae\xbf\xe0\xae\xaa\xe0\xaf\x8d\xe0\xae\xaa\xe0\xae\xbf [\xe0\xae\xaa\xe0\xae\xa4\xe0\xae\xbf\xe0\xae\xaa\xe0\xaf\x8d\xe0\xae\xaa\xe0\xae\xbf - 3] Line=4, Col=1 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n keys [keys - 1] Line=4, Col=9 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n ( [( - 8] Line=4, Col=13 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n z [z - 1] Line=4, Col=14 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n ) [) - 9] Line=4, Col=15 in File /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpJS1GyF  \n  [ - 0] Line=5, Col=1 in File  \n##########################\n']
.... terminate!!! ....

But executing same code in interpreter produces proper output

>>> code= u'\nz = dict()\n\u0bb5\u0bc8(z, "name", "Tamil")\n\u0baa\u0ba4\u0bbf\u0baa\u0bcd\u0baa\u0bbf keys(z)\n'
>>> test(code)
['/var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmp0dpbxs.output', '/var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/Tezhil_32380.out', '']
>>> f = open('/tmp/foo.txt', 'w')
>>> f.write(code.encode('utf-8'))
>>> f.close()
>>> obj = EzhilFileExecuter(file_input=f.name, redirectop=True, TIMEOUT=10)
>>> obj.run()
>>> obj.get_output()
['/var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpeWfk39.output', '/var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/Tezhil_32655.out', "[u'name']\n"]

I am using Python 2.7.10, ezhil is installed by python setup.py develop, git+git@github.com:kracekumar/Ezhil-Lang.git@5738b1178355bcb886e4af4331932f3020fce625#egg=ezhil-master

Looks like something silly I am doing. Any help ?

[arcturusannamalai]

@kracekumar - This is totally a guess at this point; I suspect the issue maybe due to redirecting stdout within EzhilFileExecuter which Django may depend upon. This gets hairy and complicated really quickly.

We need to find a way to perform the ezhil evaluate operation within the Django call stack, and not worry about the infinite-loop cases, which Django timeouts can resolve.

Basically the alternative at this point is to write code like,

import ezhil

output = ezhil.eval( " ... code ... ")

return output

Ofcourse this function ezhil.eval() does not exist in the same form today, whereas we have ezhil.start(). So some amount of refactoring will help.