search

futuregrid / cloud-metrics

Project to create usage statistics from IaaS such as OpenStack, Eucalyptus, and Nimbus
2 stars 3 forks source link

parse_file > ccInstance_parser error #4

Closed lee212 closed 12 years ago

lee212 commented 12 years ago

Parsing stopped due the an unknown error. It is following:

Processing file is: 2012-02-15-18-42-35-cc.log /home/metric/BACKUP//2012-02-15-18-42-35-cc.log 99%Traceback (most recent call last): File "FGParser.py", line 921, in main() File "FGParser.py", line 918, in main read_all_log_files_and_store_to_db (args) File "FGParser.py", line 818, in read_all_log_files_and_store_to_db parse_file(path + "/" + filename, instances.add, args.linetypes, debug=False, progress=True) File "FGParser.py", line 576, in parse_file ccInstance_parser(rest, data) File "FGParser.py", line 470, in ccInstance_parser data['keyName'] = m.group(2).strip() AttributeError: 'NoneType' object has no attribute 'group'

/home/metric/BACKUP//2012-02-15-18-42-35-cc.log file can be accessed at metric@129.79.49.76:BACKUP. When you need access, please send your ssh public key to Hyungro Lee.

fugangwang commented 12 years ago

Looks like some print_ccInstance log entry has special format(or could be corrupted). You could add debug info to find out the entry to see if we missed something if another valid format is possible. Also you could add exception handling around the regex match so if it fails it does not proceed.

laszewsk commented 12 years ago

can you fix the error in the parser.

It was to be expected that some things will not work.

Do we have to modify the datastructure.

Maybe you can describe the cause for the problem

On Wed, Mar 7, 2012 at 10:30 AM, lee212 < reply@reply.github.com

wrote:

Parsing stopped due the an unknown error. It is following:

Processing file is: 2012-02-15-18-42-35-cc.log /home/metric/BACKUP//2012-02-15-18-42-35-cc.log 99%Traceback (most recent call last): File "FGParser.py", line 921, in main() File "FGParser.py", line 918, in main read_all_log_files_and_store_to_db (args) File "FGParser.py", line 818, in read_all_log_files_and_store_to_db parse_file(path + "/" + filename, instances.add, args.linetypes, debug=False, progress=True) File "FGParser.py", line 576, in parse_file ccInstance_parser(rest, data) File "FGParser.py", line 470, in ccInstance_parser data['keyName'] = m.group(2).strip() AttributeError: 'NoneType' object has no attribute 'group'

/home/metric/BACKUP//2012-02-15-18-42-35-cc.log file can be accessed at metric@129.79.49.76:BACKUP. When you need access, please send your ssh public key to Hyungro Lee.

Reply to this email directly or view it on GitHub: https://github.com/futuregrid/eucalyptus-cloud-metrics/issues/4

lee212 commented 12 years ago

The only thing that I can do right now is adding try: except: statements between the error part but I can't think of what side effect will be. It seems ccInstance_parser function gets 'rest' parameter as a input and stores parsed data into the 'data' parameter but I don't know what happens if I return null 'data' when an exception occurs. So, to me, it may take some time to fix the problem. I guess the error is caused by a corrupted log line.

laszewsk commented 12 years ago

thats what i assumed, why are you not showing us the line that cause the problem. I am sure you have much more time available to identify the problem and the line than either Fugang or me.

We like to help, but do not have the time to do the investigation.

On Wed, Mar 7, 2012 at 1:53 PM, lee212 < reply@reply.github.com

wrote:

The only thing that I can do right now is adding try: except: statements between the error part but I can't think of what side effect will be. It seems ccInstance_parser function gets 'rest' parameter as a input and stores parsed data into the 'data' parameter but I don't know what happens if I return null 'data' when an exception occurs. So, to me, it may take some time to fix the problem. I guess the error is caused by a corrupted log line.

Reply to this email directly or view it on GitHub:

https://github.com/futuregrid/eucalyptus-cloud-metrics/issues/4#issuecomment-4374525

lee212 commented 12 years ago

I understand Fugang and you are busy. Yes, I can fix the problem which seems pretty obvious and easy to eliminate. When I post this issue, I wanted to report it first and to find some time later to fix it. Also, I was a bit worried about to modify the functions which I didn't implement. Now, I will go ahead and find some time to fix it. And I will report here what the corrupted line is and how I fixed.

Thanks,

laszewsk commented 12 years ago

ok. you alway could in this case do a branch

also as entioned, you could share with us the root cause of the problem so that we assess with you the solution.

On Thu, Mar 8, 2012 at 1:34 AM, lee212 < reply@reply.github.com

wrote:

I understand Fugang and you are busy. Yes, I can fix the problem which seems pretty obvious and easy to eliminate. When I post this issue, I wanted to report it first and to find some time later to fix it. Also, I was a bit worried about to modify the functions which I didn't implement. I will go ahead and find some time to fix it. And I will report here what the corrupted line is and how I fixed.

Thanks,

Reply to this email directly or view it on GitHub:

https://github.com/futuregrid/eucalyptus-cloud-metrics/issues/4#issuecomment-4385671

lee212 commented 12 years ago

When a corrupted line occurs, ccInstance_parser ignore the line by exception handling (try: except:)