search

hammackj / risu

Risu is Nessus parser, that converts the generated reports into a ActiveRecord database, this allows for easy report generation and vulnerability verification.
http://hammackj.github.io/risu
MIT License
63 stars 20 forks source link

RISU Parsing 3.5mb Nessus 5 file takes over 2 hours #54

Closed jammcg closed 12 years ago

jammcg commented 12 years ago

Parsing a 3.5MB Nessus 5 files to an sqlite3.db takes more than 2 hours to run.

I dont think that I will be running any 50MB Nessus files anytime soon.

Any assitance would be appreciated. (BT5 R1 64bit, GEM 1.8.21, Ruby 1.9.3, sqlite 3.6.22)

hammackj commented 12 years ago

Could you provide a gem list please. I haven;t been able to recreate this. I have had reports of files taking extreme times like you mention but when I get the report they parse in several seconds.

I think it might be a issue with libxml.

jammcg commented 12 years ago

What's the best way to provide the gem list for you?

​​​​​ Kind regards

James McGrath

jammcg commented 12 years ago

Nevermind :) Local Gems list as requested:

actionmailer (3.2.3) actionpack (3.2.3) activemodel (3.2.3) activerecord (3.2.3) activeresource (3.2.3) activesupport (3.2.3) arel (3.0.2) Ascii85 (1.0.1) builder (3.0.0) bundler (1.1.3) erubis (2.7.0) gruff (0.3.6) hike (1.2.1) i18n (0.6.0) journey (1.0.3) json (1.6.6) libxml-ruby (2.3.2) mail (2.4.4) mime-types (1.18) multi_json (1.2.0) mysql (2.8.1) pdf-reader (1.1.0) polyglot (0.3.3) prawn (0.12.0) rack (1.4.1) rack-cache (1.2) rack-ssl (1.3.2) rack-test (0.6.1) rails (3.2.3) railties (3.2.3) rake (0.9.2.2) rdoc (3.12) risu (1.5.0) rmagick (2.13.1) ruby-rc4 (0.1.5) rubygems-bundler (0.9.0) rvm (1.11.3.3) simplecov (0.6.1) simplecov-html (0.5.3) sprockets (2.1.2) sqlite3 (1.3.5) thor (0.14.6) tilt (1.3.3) treetop (1.4.10) ttfunk (1.0.3) tzinfo (0.3.33)

hammackj commented 12 years ago

Are you still having this issue? If so could you try ruby 1.9.2, I am unable to recreate this in any of my VMs on 1.9.3 and I run everything on 1.9.2 just fine

abenson commented 12 years ago

I have been able to create the issue, 556k audit file took about 10.5 minutes.

[*] Finished parsing Internal_Audit.nessus. Parse took 634.20 seconds

Ruby is ruby-1.9.3-p203.

Gems are as follows.


*** LOCAL GEMS ***

actionmailer (3.2.6)
actionpack (3.2.6)
activemodel (3.2.6)
activerecord (3.2.6)
activeresource (3.2.6)
activesupport (3.2.6)
arel (3.0.2)
Ascii85 (1.0.1)
builder (3.0.0)
bundler (1.1.4)
erubis (2.7.0)
gruff (0.3.6)
hike (1.2.1)
i18n (0.6.0)
journey (1.0.4)
json (1.7.3 java)
libxml-ruby (2.3.2)
mail (2.4.4)
mime-types (1.19)
multi_json (1.3.6)
mysql (2.8.1)
pdf-reader (1.1.1)
polyglot (0.3.3)
prawn (0.12.0)
rack (1.4.1)
rack-cache (1.2)
rack-ssl (1.3.2)
rack-test (0.6.1)
rails (3.2.6)
railties (3.2.6)
rake (0.9.2.2)
rdoc (3.12)
risu (1.5.0)
rmagick (2.13.1)
ruby-rc4 (0.1.5)
rubygems-bundler (1.0.3, 1.0.2)
rvm (1.11.3.4, 1.11.3.3)
simplecov (0.6.4)
simplecov-html (0.5.3)
sprockets (2.1.3)
sqlite3 (1.3.6)
thor (0.15.3)
tilt (1.3.3)
treetop (1.4.10)
ttfunk (1.0.3)
tzinfo (0.3.33)
hammackj commented 12 years ago

Are you using SQLite? I tracked the issue down in SQLite.

It has to do with SQLite waiting for the OS to write each insert to disk.

I have a possible fix coming soon.

Sent from my Phone

On Jun 22, 2012, at 7:15 PM, Andrew Benson reply@reply.github.com wrote:

I have been able to create the issue, 556k audit file took about 10.5 minutes.

[*] Finished parsing Internal_Audit.nessus. Parse took 634.20 seconds

Ruby is ruby-1.9.3-p203.

Gems are as follows.


*** LOCAL GEMS ***

actionmailer (3.2.6)
actionpack (3.2.6)
activemodel (3.2.6)
activerecord (3.2.6)
activeresource (3.2.6)
activesupport (3.2.6)
arel (3.0.2)
Ascii85 (1.0.1)
builder (3.0.0)
bundler (1.1.4)
erubis (2.7.0)
gruff (0.3.6)
hike (1.2.1)
i18n (0.6.0)
journey (1.0.4)
json (1.7.3 java)
libxml-ruby (2.3.2)
mail (2.4.4)
mime-types (1.19)
multi_json (1.3.6)
mysql (2.8.1)
pdf-reader (1.1.1)
polyglot (0.3.3)
prawn (0.12.0)
rack (1.4.1)
rack-cache (1.2)
rack-ssl (1.3.2)
rack-test (0.6.1)
rails (3.2.6)
railties (3.2.6)
rake (0.9.2.2)
rdoc (3.12)
risu (1.5.0)
rmagick (2.13.1)
ruby-rc4 (0.1.5)
rubygems-bundler (1.0.3, 1.0.2)
rvm (1.11.3.4, 1.11.3.3)
simplecov (0.6.4)
simplecov-html (0.5.3)
sprockets (2.1.3)
sqlite3 (1.3.6)
thor (0.15.3)
tilt (1.3.3)
treetop (1.4.10)
ttfunk (1.0.3)
tzinfo (0.3.33)

Reply to this email directly or view it on GitHub: https://github.com/hammackj/risu/issues/54#issuecomment-6520985

hammackj commented 12 years ago

Still trying to fix this issue, going to try and fix it in 1.5.2. Going to release 1.5.1 in a few days.

hammackj commented 12 years ago

I put some PRAGMAs in that should speed this up. This was released in v1.5.2. Please reopen if it doesn't solve your problem.