SyntaxError: invalid syntax with python 2.5, need 2.4/2.3 compatibility

[GoogleCodeExporter]

What steps will reproduce the problem?
1. ./scalp-0.4.py -l /var/log/apache2/access.log -f ./default_filter.xml -o
./scalp-output --html

What is the expected output? What do you see instead?
Expected : Unsure, never ran.
Actual : 

  File "./scalp-0.4.py", line 328
    with open(access) as log_file:
            ^
SyntaxError: invalid syntax

What version of the product are you using? On what operating system?
Scalp : 0.4
OS : Debian 4 with kernel 2.6.18-6-686
Python : 2.4.4

Please provide any additional information below.
md5sum of scalp : 90f87b11fccb21028c60634cc1c5f305

Original issue reported on code.google.com by martinus...@gmail.com on 19 Sep 2008 at 9:36

[GoogleCodeExporter]

The problem is that scalp has been developed using Python 2.5 and the with 
statement
is in the __future__ package of python 2.5
In order to fix this, you may replace this particular line with the following 
code:

stream = None
log_file = None
try:
    stream = open(access, 'r')
    log_file = stream.readlines()
except IOError:
    print "Cannot find open the file", access

This will be slightly slower but will do the work!
I will try to figure out how to integrate this compatibility snippet directly 
in the
scalp source code (milestone for 0.5)

Original comment by romain.g...@gmail.com on 19 Sep 2008 at 11:17

• Changed title: SyntaxError: invalid syntax with python 2.5, need 2.4 compatibility
• Changed state: Accepted
• Added labels: OpSys-All, Priority-Low
• Removed labels: Priority-Medium

[GoogleCodeExporter]

Btw, you should also remove the line:
  from __future__ import with_statement

Original comment by romain.g...@gmail.com on 19 Sep 2008 at 11:42

[GoogleCodeExporter]

Figured that one out quick :-)

But :

Cannot find the ElementTree in your python packages

To check :
# dpkg -l | grep elementtree
ii  python-elementtree                1.2.6-10                            
Light-weight toolkit for XML processing

And just to double check :
# find / -name "*ElementTree*"
/var/lib/python-support/python2.4/elementtree/ElementTree.pyc
/var/lib/python-support/python2.4/elementtree/ElementTree.py
/usr/share/python-support/python-elementtree/elementtree/ElementTree.py
/usr/lib/gedit-2/plugins/externaltools/ElementTree.pyc
/usr/lib/gedit-2/plugins/externaltools/ElementTree.py
/usr/lib/gedit-2/plugins/snippets/ElementTree.pyc
/usr/lib/gedit-2/plugins/snippets/ElementTree.py

Then, apt-cache shows me there is python-celementtree.  After installing that
package, I still get the same error.

p.s.
As a very minor bug "The directory %s doesn't exist,".  I have a feeling you 
did not
want a literal %s there :-)

I think the end of line 630 should be something like .... %s 
(preferences['odir'])

Original comment by martinus...@gmail.com on 19 Sep 2008 at 12:32

[GoogleCodeExporter]

Can't edit comment ....

It is line 622, not 630.  Was looking at line numbers after I modified it.

Original comment by martinus...@gmail.com on 19 Sep 2008 at 12:38

[GoogleCodeExporter]

Hmm, the ElementTree is kinda interesting, where does it install cElementTree? 
Usually it's somewhere like 
/usr/lib/python2.4/xml/etree/cElementTree.py 

You may try to add 2 more levels of try..except more with:

try:
    import ElementTree as etree
except:
    try:
        import cElementTree as etree
    except:
        print "Cannot find the ElementTree in your python packages"

Thanks for the directory thing, it will be fixed in the new version :)

Original comment by romain.g...@gmail.com on 19 Sep 2008 at 12:45

[GoogleCodeExporter]

Here is where cElementTree is installed :

# find / -iname "*cElementTree*"
/var/lib/dpkg/info/python-celementtree.md5sums
/var/lib/dpkg/info/python-celementtree.list
/var/cache/apt/archives/python-celementtree_1.0.5-8_i386.deb
/usr/share/doc/python-celementtree
/usr/lib/python2.3/site-packages/cElementTree-1.0.5_20051216-py2.3.egg-info
/usr/lib/python2.3/site-packages/cElementTree.so
/usr/lib/python2.4/site-packages/cElementTree-1.0.5_20051216-py2.4.egg-info
/usr/lib/python2.4/site-packages/cElementTree.so

Using :
import cElementTree as etree

works.

The script now runs, thank you.

Do you have something like GTUBE for spammassassin or EICAR for anti-virus, to 
test
the filters ?

Original comment by martinus...@gmail.com on 19 Sep 2008 at 1:07

[GoogleCodeExporter]

The filters (regexp) are not from me, but from the PHP-IDS project (http://php-
ids.org). They have their own test beds and among the best security researchers 
in 
order to create/optimize/correct the filters.

Otherwise, I will start creating test cases (especially due to the decoding 
part) 
soon. If you have any content you would like to see in such test cases, feel 
free to 
tell :)

Thanks again

Original comment by romain.g...@gmail.com on 19 Sep 2008 at 1:13

[GoogleCodeExporter]

I'm trying to use scalp-0.4 on CentOS 4.7 which has python 2.3.4-14.7. I get 
the 
following error message when I try to run it using:

./scalp-0.4.py -l /var/log/httpd_log -f /etc/default_filter.xml -o 
./scalp-output --
html

  File "./scalp-0.4.py", line 317
    total_nb_lines = sum(1 for line in open(access))
                             ^
SyntaxError: invalid syntax

Does this mean it will not work on python 2.3?

Thank you.
Mike

Original comment by osborn...@gmail.com on 20 Sep 2008 at 1:42

[GoogleCodeExporter]

I forgot to mention that I removed the line:

  from __future__ import with_statement

but I never found the references to:

stream = None
log_file = None

so I could change them.

Thank you.
Mike

Original comment by osborn...@gmail.com on 20 Sep 2008 at 1:46

[GoogleCodeExporter]

I will try to get the version working for 2.4/2.3 later, maybe a new version 
only for 
the old python versions.
I just need to finish/test features then, I will work on that.

Original comment by romain.g...@gmail.com on 20 Sep 2008 at 1:46

• Changed title: SyntaxError: invalid syntax with python 2.5, need 2.4/2.3 compatibility

[GoogleCodeExporter]

the stream and log_file variable are used by the snippet which replace the 
following 
problematic line under python 2.4/2.3:

with open(access) as log_file

Original comment by romain.g...@gmail.com on 20 Sep 2008 at 1:54

[GoogleCodeExporter]

I replaced the problematic line as indicated.
The code now looks like:
...
    321                 sampled_lines.sort()
    322
    323         loc, lines, nb_lines = 0, 0, 0
    324         old_diff = 0
    325         start = time.time()
    326         diff = []
    327 stream = None
    328 log_file = None
    329 try:
    330     stream = open(access, 'r')
    331     log_file = stream.readlines()
    332 except IOError:
    333     print "Cannot find open the file", access
    334                 for line in log_file:
    335                         lines += 1
    336                         if sample and lines not in sampled_lines:
    337                                 continue
    338                         if c_reg.match(line):
    339                                 out = c_reg.search(line)

...
I also removed the "future" line
and I get a new syntax error:
  File "./scalp-0.4.py", line 334
    for line in log_file:
    ^
SyntaxError: invalid syntax

System Debian etch
python 2.4.4-2 

Any ideas ? Thanx to everyone

Original comment by itx...@gmail.com on 28 Oct 2008 at 8:56

[GoogleCodeExporter]

Hey itxgrp,
With Python, you must give the same indentation of the code.
Here your code doesn't follow the indentation... and you should put the code 
right 
after 'log_file = stream.readlines()'

Which should be something that looks like that:

    321                 sampled_lines.sort()
    322
    323         loc, lines, nb_lines = 0, 0, 0
    324         old_diff = 0
    325         start = time.time()
    326         diff = []
    327         stream = None
    328         log_file = None
    329         try:
    330                 stream = open(access, 'r')
    331                 log_file = stream.readlines()
    332         except IOError:
    333                 print "Cannot find open the file", access
    33A                 sys.exit(0)
    334         for line in log_file:
    335                 lines += 1
    336                 if sample and lines not in sampled_lines:
    337                         continue
    338                 if c_reg.match(line):
    339                         out = c_reg.search(line)

Original comment by romain.g...@gmail.com on 28 Oct 2008 at 9:09

[GoogleCodeExporter]

I'm a php person not python. I getting the error 

-----------
Cannot find the ElementTree in your python packages
Traceback (most recent call last):
  File "./scalp-0.4.py", line 641, in ?
    main(len(sys.argv), sys.argv)
  File "./scalp-0.4.py", line 638, in main
    scalper(access, filters, preferences)
  File "./scalp-0.4.py", line 268, in scalper
    xml_filters = parse(filters)
  File "./scalp-0.4.py", line 128, in parse
    doc = etree.parse(xml_handler).getroot()
NameError: global name 'etree' is not defined
------------------

I not sure what is done by using the import cElementTree as etree statment

can anyone help

I assume I am using python < 2.5 and using scalp 0.4

Original comment by kendall....@gmail.com on 12 May 2009 at 2:03

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

It's a pity this project seems to have gone cold, it looks like a very useful 
tool,
unfortunately current Redhat versions come with Python 2.4 and the software is
crashing out with an error as many have already pointed out.  Would it be very
difficult to make this compatible with Python 2.4?

Original comment by rene.dok...@gmail.com on 23 Jun 2009 at 10:38

[GoogleCodeExporter]

for python 2.4 to fix the elementtree issue as cElementTree is 2.5 (also 
available from effbot if you want it http://effbot.org/zone/celementtree.htm)

change the 2nd elementtree import statement from:
    import xml.etree.ElementTree as etree
to
    from elementtreee import ElementTree as etree

Original comment by leonard....@swiftlogicsystems.com on 7 Apr 2011 at 1:42

[GoogleCodeExporter]

The comment 17 by leonard does the trick but it has an errata (the third "e" in 
elementtreee). So the line must look like:
   from elementtree import ElementTree as etree

Original comment by ivai...@gmail.com on 10 Jun 2011 at 8:21

[GoogleCodeExporter]

Wow, just noticing the last post is over 2 years old, so my luck is slim but 
just wondering if anybody has a version of Scalp that works with Python 2.4. I 
didn't get anywhere trying the suggestions above and no matter what always get:

  File "./scalp-0.4.py", line 327
    with open(access) as log_file:
            ^
SyntaxError: invalid syntax

I gave up.

Original comment by ske...@gmail.com on 4 Sep 2013 at 11:05