Bug: navi update fails with db lock

packetchaos / navi

A Command-line tool which leverages the Tenable.io API to reduce the time it takes to get information that is common during remediation or a troubleshooting event

GNU General Public License v3.0

72 stars 26 forks source link

Bug: navi update fails with db lock #2

Closed jbpratt closed 4 years ago

jbpratt commented 4 years ago

I have failed twice on doing an update due to the DB lock, imo this should probably be handled and retried(?) if possible. It takes right under 9 minutes for me to run the below (only 1 days worth if the day flag is honored here) so failing when processing the chunks is a bit annoying. I will dig in tomorrow and see what I can think of. Would love any thoughts @packetchaos

I have not yet been able to successfully export all data yet but will try again in the morning.

root@edda8842edc8:/usr/src/app# Navi update --days 1
Hey Listen!
Requesting Vulnerability Export with ID : 700b8073-a5c7-4430-8db3-9da4e5c7d04a
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : FINISHED
Parsing Chunk 2 ...Finished
Parsing Chunk 3 ...Finished

 Have you entered your keys?

Error:  database is locked

packetchaos commented 4 years ago

How many assets do you have in your IO instance?

I have't been able to test on large deployments...

On Mon, Dec 9, 2019 at 2:00 PM jbpratt notifications@github.com wrote:

I have failed twice on doing an update due to the DB lock, imo this should probably be handled and retried(?) if possible. It takes right under 9 minutes for me to run the below (only 1 days worth if the day flag is honored here) so failing when processing the chunks is a bit annoying. I will dig in tomorrow and see what I can think of. Would love any thoughts @packetchaos https://github.com/packetchaos

I have not yet been able to successfully export all data yet but will try again in the morning.

root@edda8842edc8:/usr/src/app# Navi update --days 1 Hey Listen! Requesting Vulnerability Export with ID : 700b8073-a5c7-4430-8db3-9da4e5c7d04a Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : FINISHED Parsing Chunk 2 ...Finished Parsing Chunk 3 ...Finished

Have you entered your keys?

Error: database is locked

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/packetchaos/Navi/issues/2?email_source=notifications&email_token=AHWPVTOXNTDBGT6G7OCYQLLQX2WVBA5CNFSM4JYSCZZKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4H7IJQMA, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHWPVTPLYWLUZIMVAYS4Q23QX2WVBANCNFSM4JYSCZZA .

packetchaos commented 4 years ago

I found a bug in the code after your pull request. I just fixed the bug and pushed a new container. Pull the new one down and try using Navi update -vulns 1 and see if it finishes, then do Navi update -assets 1 and see if it finishes.

This will be helpful deciding where the problem resides.

Thanks,

On Mon, Dec 9, 2019 at 2:15 PM silent shadow cyberdice113@gmail.com wrote:

How many assets do you have in your IO instance?

I have't been able to test on large deployments...

On Mon, Dec 9, 2019 at 2:00 PM jbpratt notifications@github.com wrote:

I have failed twice on doing an update due to the DB lock, imo this should probably be handled and retried(?) if possible. It takes right under 9 minutes for me to run the below (only 1 days worth if the day flag is honored here) so failing when processing the chunks is a bit annoying. I will dig in tomorrow and see what I can think of. Would love any thoughts @packetchaos https://github.com/packetchaos

I have not yet been able to successfully export all data yet but will try again in the morning.

root@edda8842edc8:/usr/src/app# Navi update --days 1 Hey Listen! Requesting Vulnerability Export with ID : 700b8073-a5c7-4430-8db3-9da4e5c7d04a Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : QUEUED Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : PROCESSING Status : FINISHED Parsing Chunk 2 ...Finished Parsing Chunk 3 ...Finished

Have you entered your keys?

Error: database is locked

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/packetchaos/Navi/issues/2?email_source=notifications&email_token=AHWPVTOXNTDBGT6G7OCYQLLQX2WVBA5CNFSM4JYSCZZKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4H7IJQMA, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHWPVTPLYWLUZIMVAYS4Q23QX2WVBANCNFSM4JYSCZZA .

jbpratt commented 4 years ago

I ~think~ know our license is up to 30,000, which I ~know~ think we are close to. I will pull and test again tomorrow morning once I am back in the office.

Thanks for the help.

jbpratt commented 4 years ago

~Both~ One of those finished much quicker (only 8 seconds for assets and -----). ~I will try a full export soon. I will continue to monitor for this bug as well.~ as I was typing this and running for vulns I hit the database lock again. I will be investigating.

Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : FINISHED
Parsing Chunk 2 ...Finished
Parsing Chunk 3 ...Finished

 Have you entered your keys?

Error:  database is locked

real    8m2.766s
user    0m2.190s
sys     0m0.680s

packetchaos commented 4 years ago

Can you try using Navi python Package instead of the container. I think there might be an issue with the size of data you are pulling down and the way the container is built.

jbpratt commented 4 years ago

@packetchaos thanks for working with me. Hmm nope, seems to not be a constraint on the container, still fails on just a simple download. Is the database being locked during a write that isn't releasing or are we trying to write at the same time as a separate process? Haven't sat down to search through yet and find where this is occurring at. Probably around the same time as the above attempt took.

➜  work Navi update -vulns --days 1
Hey Listen!
Requesting Vulnerability Export with ID : 7f7aa0ab-7251-4a02-9a27-ba98bb17216c
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : QUEUED
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : FINISHED
Parsing Chunk 2 ...Finished
Parsing Chunk 3 ...Finished

 Have you entered your keys?

Error:  database is locked

Edit: Already wrote a Go program to do this, prior to finding the library, so it is not like this issue blocking me. Just would like to help improve the tool as it will be useful for me.

packetchaos commented 4 years ago

I'm going to work on the logic; I'm using the same connection while adding data to the database, but I think the database isn't finishing the process before I try to add a new set of data. I'm working on a patch to see if it resolves the issue. I've only tested this on accounts with 5000 Assets so I'm hoping you can continue to test with me as I solve this issue. Since, it works on my smaller container.

I should have a patch this week, by Monday at the latest.

On Tue, Dec 10, 2019 at 12:06 PM jbpratt notifications@github.com wrote:

@packetchaos https://github.com/packetchaos thanks for working with me. Hmm nope, seems to not be a constraint on the container, still fails on just a simple download. Is the database being locked during a write that isn't releasing or are we trying to write at the same time as a separate process? Haven't sat down to search through yet and find where this is occurring at. Probably around the same time as the above attempt took.

➜ work Navi update -vulns --days 1

Hey Listen!

Requesting Vulnerability Export with ID : 7f7aa0ab-7251-4a02-9a27-ba98bb17216c

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : PROCESSING

Status : PROCESSING

Status : PROCESSING

Status : PROCESSING

Status : PROCESSING

Status : PROCESSING

Status : PROCESSING

Status : FINISHED

Parsing Chunk 2 ...Finished

Parsing Chunk 3 ...Finished

Have you entered your keys?

Error: database is locked

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/packetchaos/Navi/issues/2?email_source=notifications&email_token=AHWPVTOPXD666KOKGY5CAZTQX7SCJA5CNFSM4JYSCZZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGQMICA#issuecomment-564184072, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHWPVTIW7NYUGPWWKKLFCMLQX7SCJANCNFSM4JYSCZZA .

packetchaos commented 4 years ago

@jbpratt If you have time can you try this version. I took a slightly different approach, but I don't have a container large enough to test with.

docker pull silentninja/navi:506

https://hub.docker.com/layers/silentninja/navi/506/images/sha256-91ecbd43e80a9d7042003c3c4d81feb7203e11b5497fdbb665d5c9c286853e49

note: I changed Navi to navi(lowercase)....

On Tue, Dec 10, 2019 at 1:53 PM silent shadow cyberdice113@gmail.com wrote:

I'm going to work on the logic; I'm using the same connection while adding data to the database, but I think the database isn't finishing the process before I try to add a new set of data. I'm working on a patch to see if it resolves the issue. I've only tested this on accounts with 5000 Assets so I'm hoping you can continue to test with me as I solve this issue. Since, it works on my smaller container.

I should have a patch this week, by Monday at the latest.

On Tue, Dec 10, 2019 at 12:06 PM jbpratt notifications@github.com wrote:

@packetchaos https://github.com/packetchaos thanks for working with me. Hmm nope, seems to not be a constraint on the container, still fails on just a simple download. Is the database being locked during a write that isn't releasing or are we trying to write at the same time as a separate process? Haven't sat down to search through yet and find where this is occurring at. Probably around the same time as the above attempt took.

➜ work Navi update -vulns --days 1

Hey Listen!

Requesting Vulnerability Export with ID : 7f7aa0ab-7251-4a02-9a27-ba98bb17216c

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : QUEUED

Status : PROCESSING

Status : PROCESSING

Status : PROCESSING

Status : PROCESSING

Status : PROCESSING

Status : PROCESSING

Status : PROCESSING

Status : FINISHED

Parsing Chunk 2 ...Finished

Parsing Chunk 3 ...Finished

Have you entered your keys?

Error: database is locked

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/packetchaos/Navi/issues/2?email_source=notifications&email_token=AHWPVTOPXD666KOKGY5CAZTQX7SCJA5CNFSM4JYSCZZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGQMICA#issuecomment-564184072, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHWPVTIW7NYUGPWWKKLFCMLQX7SCJANCNFSM4JYSCZZA .

jbpratt commented 4 years ago

@packetchaos Were there code changes to this build or just adjusting how the container is built?

packetchaos commented 4 years ago

There are code changes. I use multiple db connections in this version.

On Wed, Dec 11, 2019, 5:37 AM jbpratt notifications@github.com wrote:

@packetchaos https://github.com/packetchaos Were there code changes to this build or just adjusting how the container is built?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/packetchaos/Navi/issues/2?email_source=notifications&email_token=AHWPVTLSOW4ZVDDG2DO5B4DQYDNKDA5CNFSM4JYSCZZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGS6TAY#issuecomment-564521347, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHWPVTKMBMGP6WBKTV4LVHLQYDNKDANCNFSM4JYSCZZA .

jbpratt commented 4 years ago

Can you publish the code as a branch? Sorry, just need to look at it before running.

packetchaos commented 4 years ago

I just published the branch "dblock". This is what the container was built off of.

On Wed, Dec 11, 2019 at 5:53 AM jbpratt notifications@github.com wrote:

Can you publish the code as a branch? Sorry, just need to look at it before running.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/packetchaos/Navi/issues/2?email_source=notifications&email_token=AHWPVTK6NE46NSCXPZQEYS3QYDPC7A5CNFSM4JYSCZZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGS7ZWI#issuecomment-564526297, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHWPVTMMRUNMX7JFFFCN6X3QYDPC7ANCNFSM4JYSCZZA .

jbpratt commented 4 years ago

I just published the branch "dblock". This is what the container was built off of. … On Wed, Dec 11, 2019 at 5:53 AM jbpratt @.***> wrote: Can you publish the code as a branch? Sorry, just need to look at it before running. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2?email_source=notifications&email_token=AHWPVTK6NE46NSCXPZQEYS3QYDPC7A5CNFSM4JYSCZZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGS7ZWI#issuecomment-564526297>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHWPVTMMRUNMX7JFFFCN6X3QYDPC7ANCNFSM4JYSCZZA .

Thank you, Ill start this in a bit!

jbpratt commented 4 years ago

@packetchaos Right after entering keys and tried updating, it failed:

root@167a3b7637c9:/usr/src/app# navi update
Hey Listen!

The object you tried to create already exists

If you are Updating tags via groups it is not supported right now, delete your group using the delete command

Traceback (most recent call last):
  File "/usr/local/bin/navi", line 11, in <module>
    load_entry_point('navi-Pro==5.0.5', 'console_scripts', 'navi')()
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/navi_Pro-5.0.5-py3.6.egg/navi/plugins/update.py", line 19, in update
  File "/usr/local/lib/python3.6/dist-packages/navi_Pro-5.0.5-py3.6.egg/navi/plugins/vuln_export.py", line 25, in vuln_export
TypeError: 'NoneType' object is not subscriptable

Edit: I think a bug was introduced in this update, it seems that all commands I have tried are at least attempting to do something with groups.

packetchaos commented 4 years ago

Okay I just repacked the code into the container...I just pushed the new container with the latest corrections. Sorry for the hassle, I've been making silly mistakes today.

On Wed, Dec 11, 2019 at 7:54 AM jbpratt notifications@github.com wrote:

@packetchaos https://github.com/packetchaos Right after entering keys and tried updating, it failed:

root@167a3b7637c9:/usr/src/app# navi update Hey Listen!

The object you tried to create already exists

If you are Updating tags via groups it is not supported right now, delete your group using the delete command

Traceback (most recent call last): File "/usr/local/bin/navi", line 11, in load_entry_point('navi-Pro==5.0.5', 'console_scripts', 'navi')() File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 764, in call return self.main(args, kwargs) File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 555, in invoke return callback(args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/navi_Pro-5.0.5-py3.6.egg/navi/plugins/update.py", line 19, in update File "/usr/local/lib/python3.6/dist-packages/navi_Pro-5.0.5-py3.6.egg/navi/plugins/vuln_export.py", line 25, in vuln_export TypeError: 'NoneType' object is not subscriptable

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/packetchaos/Navi/issues/2?email_source=notifications&email_token=AHWPVTJFQQ72FOEB3HA4I3TQYD5LXA5CNFSM4JYSCZZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGTNLWY#issuecomment-564581851, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHWPVTNKCWKF4E4V4MW6YIDQYD5LXANCNFSM4JYSCZZA .

jbpratt commented 4 years ago

Attempted this morning an something occurred while the process was running and froze after 20 minutes, without me noticing for another 40 minutes. I will give it another go tomorrow.

packetchaos commented 4 years ago

Okay. Sorry for the lost time. Try limiting the download again to 1 day.

I configured autobuilds off the dblock branch so I can iterate over problems found. I added more visibility to the update command and I lengthened the check-in time to Tenable.io. When you see "Processing" this is coming from the T.io API. So I can't show progress bar when the download will be ready. However, I added how many chunks were found and let you know what number you are on. EX: "Parsing chunk 1 of 100".

I just pushed this new code out; which updated the container. silentninja/navi:506

On Wed, Dec 11, 2019 at 2:07 PM jbpratt notifications@github.com wrote:

Attempted this morning an something occurred while the process was running and froze after 20 minutes, without me noticing for another 40 minutes. I will give it another go tomorrow.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/packetchaos/Navi/issues/2?email_source=notifications&email_token=AHWPVTMZ25S6ZIEAWTVLA43QYFJA3A5CNFSM4JYSCZZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGUSJLY#issuecomment-564733103, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHWPVTLLZWLNAXLNLXQQSBTQYFJA3ANCNFSM4JYSCZZA .

packetchaos commented 4 years ago

@jbpratt,

I'm working on Navi the next few days. I'm curious if you were able to download your data without a DB lock? Since I can't test it out I'm not sure what to address. There could be more than one problem. One being the DB lock but the other being the container's ability to scale to the data size required. Let me know the results of your tests so I can try to resolve them in the next few days.

Thanks again for your help!

On Wed, Dec 11, 2019 at 4:23 PM silent shadow cyberdice113@gmail.com wrote:

Okay. Sorry for the lost time. Try limiting the download again to 1 day.

I configured autobuilds off the dblock branch so I can iterate over problems found. I added more visibility to the update command and I lengthened the check-in time to Tenable.io. When you see "Processing" this is coming from the T.io API. So I can't show progress bar when the download will be ready. However, I added how many chunks were found and let you know what number you are on. EX: "Parsing chunk 1 of 100".

I just pushed this new code out; which updated the container. silentninja/navi:506

On Wed, Dec 11, 2019 at 2:07 PM jbpratt notifications@github.com wrote:

Attempted this morning an something occurred while the process was running and froze after 20 minutes, without me noticing for another 40 minutes. I will give it another go tomorrow.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/packetchaos/Navi/issues/2?email_source=notifications&email_token=AHWPVTMZ25S6ZIEAWTVLA43QYFJA3A5CNFSM4JYSCZZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGUSJLY#issuecomment-564733103, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHWPVTLLZWLNAXLNLXQQSBTQYFJA3ANCNFSM4JYSCZZA .

jbpratt commented 4 years ago

I will try again in just a bit and see how it goes. Outside of a progress bar, it may be better to just have a running timer rather than the PROCESSING printing every time, but this isn’t a big deal.

On Fri, Dec 13, 2019 at 11:18 AM Casey Reid notifications@github.com wrote:

@jbpratt,

I'm working on Navi the next few days. I'm curious if you were able to download your data without a DB lock? Since I can't test it out I'm not sure what to address. There could be more than one problem. One being the DB lock but the other being the container's ability to scale to the data size required. Let me know the results of your tests so I can try to resolve them in the next few days.

Thanks again for your help!

On Wed, Dec 11, 2019 at 4:23 PM silent shadow cyberdice113@gmail.com wrote:

Okay. Sorry for the lost time. Try limiting the download again to 1 day.

I configured autobuilds off the dblock branch so I can iterate over problems found. I added more visibility to the update command and I lengthened the check-in time to Tenable.io. When you see "Processing" this is coming from the T.io API. So I can't show progress bar when the download will be ready. However, I added how many chunks were found and let you know what number you are on. EX: "Parsing chunk 1 of 100".

I just pushed this new code out; which updated the container. silentninja/navi:506

On Wed, Dec 11, 2019 at 2:07 PM jbpratt notifications@github.com wrote:

Attempted this morning an something occurred while the process was running and froze after 20 minutes, without me noticing for another 40 minutes. I will give it another go tomorrow.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub < https://github.com/packetchaos/Navi/issues/2?email_source=notifications&email_token=AHWPVTMZ25S6ZIEAWTVLA43QYFJA3A5CNFSM4JYSCZZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGUSJLY#issuecomment-564733103 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AHWPVTLLZWLNAXLNLXQQSBTQYFJA3ANCNFSM4JYSCZZA

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/packetchaos/Navi/issues/2?email_source=notifications&email_token=AHBA5HNCBR6ABCR2G73BEWTQYO7W3A5CNFSM4JYSCZZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEG2T4WI#issuecomment-565526105, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHBA5HNB4QZDXBLAT25LTO3QYO7W3ANCNFSM4JYSCZZA .

jbpratt commented 4 years ago

I didn’t get the chance to run it today, my apologies

On Fri, Dec 13, 2019 at 12:07 PM Brady Pratt jbpratt78@gmail.com wrote:

I will try again in just a bit and see how it goes. Outside of a progress bar, it may be better to just have a running timer rather than the PROCESSING printing every time, but this isn’t a big deal.

On Fri, Dec 13, 2019 at 11:18 AM Casey Reid notifications@github.com wrote:

@jbpratt,

I'm working on Navi the next few days. I'm curious if you were able to download your data without a DB lock? Since I can't test it out I'm not sure what to address. There could be more than one problem. One being the DB lock but the other being the container's ability to scale to the data size required. Let me know the results of your tests so I can try to resolve them in the next few days.

Thanks again for your help!

On Wed, Dec 11, 2019 at 4:23 PM silent shadow cyberdice113@gmail.com wrote:

Okay. Sorry for the lost time. Try limiting the download again to 1 day.

I configured autobuilds off the dblock branch so I can iterate over problems found. I added more visibility to the update command and I lengthened the check-in time to Tenable.io. When you see "Processing" this is coming from the T.io API. So I can't show progress bar when the download will be ready. However, I added how many chunks were found and let you know what number you are on. EX: "Parsing chunk 1 of 100".

I just pushed this new code out; which updated the container. silentninja/navi:506

On Wed, Dec 11, 2019 at 2:07 PM jbpratt notifications@github.com wrote:

Attempted this morning an something occurred while the process was running and froze after 20 minutes, without me noticing for another 40 minutes. I will give it another go tomorrow.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub < https://github.com/packetchaos/Navi/issues/2?email_source=notifications&email_token=AHWPVTMZ25S6ZIEAWTVLA43QYFJA3A5CNFSM4JYSCZZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGUSJLY#issuecomment-564733103 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AHWPVTLLZWLNAXLNLXQQSBTQYFJA3ANCNFSM4JYSCZZA

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/packetchaos/Navi/issues/2?email_source=notifications&email_token=AHBA5HNCBR6ABCR2G73BEWTQYO7W3A5CNFSM4JYSCZZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEG2T4WI#issuecomment-565526105, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHBA5HNB4QZDXBLAT25LTO3QYO7W3ANCNFSM4JYSCZZA .

jbpratt commented 4 years ago

From this mornings attempt:

➜  Navi git:(dblock) docker run -it -p 8000:8000 silentninja/navi:506 bash
root@fc87fb588003:/usr/src/app# navi keys
Hey Listen!
Hey you don't have any Keys!
Please provide your Access Key :
Please provide your Secret Key :
root@fc87fb588003:/usr/src/app# navi update --days 1

Hey Listen!
Requesting Vulnerability Export with ID : 7d94c1dc-1358-42b1-af03-6af82607fc10
Status : QUEUED
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : PROCESSING
Status : FINISHED
Parsing Chunk 2 of 2 ...Finished
Parsing Chunk 3 of 2 ...Finished

 Have you entered your keys?

Error:  database is locked

Seems the chunk iterator is ahead by one. Still got the same db lock though and this was just for a single day.

packetchaos commented 4 years ago

Okay. I think I figured out the problem; hopefully. I changed the logic quite a bit this time around. I create a new DB connection for every insert rather than do multiple inserts under one persistent connection. This will be a bit slower, but if it works I can address the slowness later.

I just pushed out the new code to the dblock branch and a new container is live: "silentninja/navi:506" If you have time to test today, that would be great.

Thanks,

On Mon, Dec 16, 2019 at 8:18 AM jbpratt notifications@github.com wrote:

From this mornings attempt:

➜ Navi git:(dblock) docker run -it -p 8000:8000 silentninja/navi:506 bash

root@fc87fb588003:/usr/src/app# navi keys

Hey Listen!

Hey you don't have any Keys!

Please provide your Access Key :

Please provide your Secret Key :

root@fc87fb588003:/usr/src/app# navi update --days 1

Hey Listen!

Requesting Vulnerability Export with ID : 7d94c1dc-1358-42b1-af03-6af82607fc10

Status : QUEUED

Status : PROCESSING

Status : PROCESSING

Status : PROCESSING

Status : PROCESSING

Status : PROCESSING

Status : PROCESSING

Status : PROCESSING

Status : FINISHED

Parsing Chunk 2 of 2 ...Finished

Parsing Chunk 3 of 2 ...Finished

Have you entered your keys?

Error: database is locked

Seems the chunk iterator is ahead by one. Still got the same db lock though and this was just for a single day.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/packetchaos/Navi/issues/2?email_source=notifications&email_token=AHWPVTIB6MZH7I7O3DVW4OLQY6L3DA5CNFSM4JYSCZZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEG7BH7Q#issuecomment-566105086, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHWPVTKFCRWDK234SDH6JFTQY6L3DANCNFSM4JYSCZZA .

jbpratt commented 4 years ago

After running this morning, navi update -assets --days 1 finished in 20 seconds yet navi update -vulns --days 1 was still going at real 19m40.287s when I killed the process. Not sure what the problem is.

packetchaos commented 4 years ago

If you have a lot of plugins it could take some time. Right now it is single threaded. I would be interested to see how long it takes to finish. If I got past the DB locks I can work to improve the speed. Would you mind seeing if it finishes successfully?

On Tue, Dec 17, 2019 at 9:31 AM jbpratt notifications@github.com wrote:

After running this morning, navi update -assets --days 1 finished in 20 seconds yet navi update -vulns --days 1 was still going at real 19m40.287s when I killed the process. Not sure what the problem is.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/packetchaos/Navi/issues/2?email_source=notifications&email_token=AHWPVTLCEZXNAEHSYTGHE73QZD5ERA5CNFSM4JYSCZZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHDEEHI#issuecomment-566641181, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHWPVTJP2H3PLCOESYQ6EPTQZD5ERANCNFSM4JYSCZZA .

jbpratt commented 4 years ago

Okay, I have finished exporting vulns. The process seemingly died around 83 minutes the first time, but finished just shy of 56 minutes this time.

packetchaos commented 4 years ago

Okay. This is good information. I'm working on code to increase the speed substantially. Thank you again for testing this out.

On Tue, Dec 17, 2019 at 12:31 PM jbpratt notifications@github.com wrote:

Okay, I have finished exporting vulns. The process seemingly died around 83 minutes the first time, but finished just shy of 56 minutes this time.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/packetchaos/Navi/issues/2?email_source=notifications&email_token=AHWPVTLR3G7JXNQUNWCCZOTQZESJBA5CNFSM4JYSCZZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHDV5EI#issuecomment-566714001, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHWPVTPPU5CXNNR5TGQQDX3QZESJBANCNFSM4JYSCZZA .

packetchaos commented 4 years ago

@jbpratt I pushed out a new version today with some timing statistics. I put a timer on each section of the process.

Tenable.io Processing
Download Time
Processing and saving the data

I'm interested in verifying the bottle neck.

Would you mind testing?

On Tue, Dec 17, 2019 at 1:33 PM silent shadow cyberdice113@gmail.com wrote:

Okay. This is good information. I'm working on code to increase the speed substantially. Thank you again for testing this out.

On Tue, Dec 17, 2019 at 12:31 PM jbpratt notifications@github.com wrote:

Okay, I have finished exporting vulns. The process seemingly died around 83 minutes the first time, but finished just shy of 56 minutes this time.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/packetchaos/Navi/issues/2?email_source=notifications&email_token=AHWPVTLR3G7JXNQUNWCCZOTQZESJBA5CNFSM4JYSCZZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHDV5EI#issuecomment-566714001, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHWPVTPPU5CXNNR5TGQQDX3QZESJBANCNFSM4JYSCZZA .

packetchaos commented 4 years ago

@jbpratt. I finally figured out the problem. I pushed out the newest container resolving the DB lock issue and reduced the time by 4x. I tested it on 25,000 assets and 50,000 vulns and it took 33 minutes oppose to 2 hours.

I'm going to close this ticket and merge the dblock code with the main branch this weekend. I will be working on mult-threading the export to increase the speed some more, but the db is no longer the bottle neck

Thank you for your help!

On Tue, Dec 17, 2019 at 10:15 PM silent shadow cyberdice113@gmail.com wrote:

@jbpratt I pushed out a new version today with some timing statistics. I put a timer on each section of the process.

Tenable.io Processing

Download Time

Processing and saving the data

I'm interested in verifying the bottle neck.

Would you mind testing?

On Tue, Dec 17, 2019 at 1:33 PM silent shadow cyberdice113@gmail.com wrote:

Okay. This is good information. I'm working on code to increase the speed substantially. Thank you again for testing this out.

On Tue, Dec 17, 2019 at 12:31 PM jbpratt notifications@github.com wrote:

Okay, I have finished exporting vulns. The process seemingly died around 83 minutes the first time, but finished just shy of 56 minutes this time.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/packetchaos/Navi/issues/2?email_source=notifications&email_token=AHWPVTLR3G7JXNQUNWCCZOTQZESJBA5CNFSM4JYSCZZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHDV5EI#issuecomment-566714001, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHWPVTPPU5CXNNR5TGQQDX3QZESJBANCNFSM4JYSCZZA .

jbpratt commented 4 years ago

@packetchaos awesome job. Sorry I was not able to test the last few days, we were busy closing things out and I'm on vacation now. Excited to test it out when I get back. Thank you.