search

greenbone / ospd-openvas

ospd-openvas is an OSP server implementation to allow GVM to remotely control an OpenVAS Scanner
GNU Affero General Public License v3.0
67 stars 58 forks source link

[20.8.0] Unexpected interrupted return code #335

Closed tuxmaster5000 closed 3 years ago

tuxmaster5000 commented 4 years ago

The scanner self will exits fine, but ospd-openvas will interpret it as an error. ospd-openvas log:

(ospd.ospd) aae6b036-7a1c-47d4-8866-0e07c8799b48: Host scan finished. (ospd.ospd) aae6b036-7a1c-47d4-8866-0e07c8799b48: Scan interrupted. (ospd.ospd) aae6b036-7a1c-47d4-8866-0e07c8799b48: Scan stopped with errors. (ospd.ospd) aae6b036-7a1c-47d4-8866-0e07c8799b48: Scan interrupted.

openvas log:

Vulnerability scan d667271d-11ef-4e68-bc0c-39fa0659e778 finished for host XXXXX in 993.56 seconds Vulnerability scan d667271d-11ef-4e68-bc0c-39fa0659e778 finished in 1000 seconds: 1 hosts

No error was logged by openvas.

openvas --version

OpenVAS 20.8.0 gvm-libs 20.8.0

ospd version:

20.8.1

Most new code since 2005: (C) 2020 Greenbone Networks GmbH Nessus origin: (C) 2004 Renaud Deraison deraison@nessus.org License GPLv2: GNU GPL version 2 This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

jjnicola commented 4 years ago

Hi @tuxmaster5000,

could you provide please some more information about the scanner configuration? Is it a full & fast scan config? Some special option for scanner? does the target has something special (behind a firewall)? Did you see in the report or in the log any plugin timeout error / closed port ?

tuxmaster5000 commented 4 years ago

No I don't use any special options, it is an simple full & fast scan. I only use the default setting for the scanner. The only noticeable is, that it will happens not on all hosts. An really error is not show in the report under the error page. Only the "universal error":

Scan process failure.   Scan process failure.   Task interrupted unexpectedly

When I let run openvas in debug mode, I only see the typical error/warning messages about "Not launching because a mandatory key is missing (this is not an error)" on some tests. But the string "Scan process failure" don't occur in the debug log of openvas.

Let an grep run over the debug log don't shown any real error. Only thinks like:

lib nasl: DEBUG:2020-09-11 05h57.22 utc:323: Request => /_errors/ sd main:MESSAGE:2020-09-11 06h17.16 utc:2302: Plugin Policy/policy_BSI-TR-03116-4_error.nasl is deprecated. It will neither be loaded nor launched. sd main:MESSAGE:2020-09-11 06h27.36 utc:2330: Launching 2014/gb_ajenti_respond_error_mult_xss_vuln.nasl (1.3.6.1.4.1.25623.1.0.804654) against XXXX [30826] sd main:MESSAGE:2020-09-11 06h27.37 utc:2330: 2014/gb_ajenti_respond_error_mult_xss_vuln.nasl (1.3.6.1.4.1.25623.1.0.804654) [30826] finished its job in 0.147 seconds

When an real error will occur, I think openvas will log it in the debug mode.

rolf-d2i commented 3 years ago

I am experiencing something similar with ospdf-openvas. Looks like the same error as above. I don't see how the error below is related to a configuration error. I havent looked into this in detail but set_scan_total_hosts looks like it has missing method being called from somewhere which I don't think is related to configuration errors but source code issues.

OSPD[30973] 2020-10-20 08:30:06,334: ERROR: (ospd.ospd) While scanning: 164a83fe-17e9-42f9-b502-081e3c5b96b4 Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/ospd-20.8.1-py3.7.egg/ospd/ospd.py", line 571, in start_scan self.exec_scan(scan_id) File "/usr/local/lib/python3.7/dist-packages/ospd_openvas-20.8.0-py3.7.egg/ospd_openvas/daemon.py", line 1351, in exec_scan self.report_openvas_results(kbdb, scan_id, "") File "/usr/local/lib/python3.7/dist-packages/ospd_openvas-20.8.0-py3.7.egg/ospd_openvas/daemon.py", line 1123, in report_openvas_results self.set_scan_total_hosts(scan_id, count_total) AttributeError: 'OSPDopenvas' object has no attribute 'set_scan_total_hosts' OSPD[30973] 2020-10-20 08:30:06,336: INFO: (ospd.ospd) 164a83fe-17e9-42f9-b502-081e3c5b96b4: Scan interrupted.

CipherMonger commented 3 years ago

Same issue here as well. Details are as follows:

The report shows some NVT timeouts followed by "Scan Process Failure" and "Task interrupted unexpectedly", but the openvas.log output says:

sd main:MESSAGE:2020-10-26 21h42.55 utc:2791: Vulnerability scan fb1569fe-58ae-443e-8677-1379516cfbc8 finished in 26825 seconds: 2536 hosts

No indication of errors in the log.

rolf-d2i commented 3 years ago

I managed to fix my error by patching the source code in ospd_openvas/daemon.py replacing the except statement on line 1124 with " except:". After patching the source code I could actually scan some targets.

tuxmaster5000 commented 3 years ago

@rolf-d2i , can you post an patch for it? Because on line 1124 I don't see an except statement:

1120             )
1121 
1122         return total_results
1123 
1124     def report_openvas_timestamp_scan_host(
1125         self, scan_db: ScanDB, scan_id: str, host: str
1126     ):
1127         """ Get start and end timestamp of a host scan from redis kb. """
1128         timestamp = scan_db.get_host_scan_end_time()
1129         if timestamp:
1130             self.add_scan_log(
1131                 scan_id, host=host, name='HOST_END', value=timestamp
rolf-d2i commented 3 years ago

The issue i found is related to release 20.8.1 code, processing HOSTS_COUNT (starting on line 1119 in daemon.py), which doesn't work in that branch and version. Ignoring errors from this part of the code was good enough for me, it is more of a hack than a patch.

        # To update total host count
        if msg[0] == 'HOSTS_COUNT':
            try:
                count_total = int(msg[4])
                self.set_scan_total_hosts(scan_id, count_total)
            except:
                logger.debug('Error processing total host count')
bjoernricks commented 3 years ago

The set_scan_total_hosts is only used in the ospd-openvas-20.08 branch and not included in any release yet. So if you use an unreleased version from the ospd-openvas-20.08 branch you need to use the ospd-20.08 branch for ospd too.

The patch from @rolf-d2i is wrong. It wont work and just resulting in a different issue.

rolf-d2i commented 3 years ago

No this is wrong @bjoernrick. I was building OpenVAS in a docker container (see below). The 20.08 is supposed to be stable released version (when looking at the documentation and github), if it is not a stable released version then the clone command is picking up the wrong version of OpenVAS for some reason.

   RUN git clone -b ospd-openvas-20.08 https://github.com/greenbone/ospd-openvas.git /root/source/ospd-openvas
jjnicola commented 3 years ago

Hi @rolf-d2i , as @bjoernricks explained, your issue is not related with the original one reported by @tuxmaster5000. The patch generating your issue, was added with the PR #343, while this issue was reported before that. You are using the current state of the 20.08 branch which includes non released patches. Please, use the last release for your installation. Regards

rolf-d2i commented 3 years ago

As I mentioned before @jjnicola . I have built the version from source using what is documented as the latest stable version and encountering "Unexpected interrupted return issues". If there is an issue with the latest stable version then there should be at least a new issue when building the latest stable version. I was building everything from scratch using the same version. I don't think there is anything in the documentation stating which 20.08.X version is considered stable or if there are unstable 20.08.X versions.

bjoernricks commented 3 years ago

@rolf-d2i again you are mixing things up. A release can be found at the releases page of a repository at GitHub e.g. https://github.com/greenbone/ospd-openvas/releases Each release refers to a git tag. If you want to use the latest release you have to use https://github.com/greenbone/ospd-openvas/tree/v20.8.0 for ospd-openvas and https://github.com/greenbone/ospd/tree/v20.8.1 for ospd. New bugfix releases will be created from the release branches e.g. from https://github.com/greenbone/ospd/tree/ospd-20.08 If you are using a release branch of ospd-openvas you must use the release branch of ospd too. You can't combine the 20.8.1 release of ospd with the ospd-openvas 20.8 branch. The lastest stable version is not contained in a release branch. The release branch contains the next going to be released version.

bjoernricks commented 3 years ago

And also again your issue has nothing to do with the original reported one here.

rolf-d2i commented 3 years ago

@bjoernricks as I mentioned before what versions should be used is poorly documented. Certainly the page "https://community.greenbone.net/t/gvm-20-08-stable-initial-release-2020-08-12/6312" doesn't clearly say anything about 20.08.X versions. It is unexpected behaviour that 20.08.01 release breaks with 20.08.00, you would commonly expect major.minor.patch as version numbering. The page lists for instance OSPd 20.8.1 together with 20.8.0 components implying you should use the latest 20.08.X version. This is honestly a very confusing version handling scheme/git management policy you are using.

bjoernricks commented 3 years ago

Yes this might be confusing from the outside and this specific error will not happen very often. But to be clear even if we would use semantic versioning (which we don't do) it wouldn't indicate that the unreleased ospd-openvas need an unreleased version of ospd or that a 21.1.0 versions of ospd-openvas doesn't work with 20.11.4. This is a task for the dependency management. The dependencies are set correctly for poetry but there seems to be a missing change in the setup.py file.

In most situations it should be fine to mix the latest releases with using release branches. But personally I would never do that and I would not advise to do that. I don't even advise users to build from git checkouts. Even the release announcement says

GVM is published as regularly updated and tested source code releases.

So my general advise is to use the latest released versions. If you are familiar with fixing issues by yourself especially with debugging error messages like the one you got it is fine to build from git.

zenire commented 3 years ago

Looks like this issue occurs after the scan itself has finished. For some reason the process stays hanging while status is Done/Finished and progress is 100%. I get this error after the NMAP scan and NASL tests are done.

zenire commented 3 years ago

@tuxmaster5000 @CipherMonger Could you share if you set or share your create target XML?

zenire commented 3 years ago

I repeatedly started scans with the following results: <exclude_hosts>192.168.1.1</exclude_hosts> succeeds <exclude_hosts>192.168.100.200</exclude_hosts> succeeds <exclude_hosts>192.168.1.1,192.168.2.5</exclude_hosts> fails <exclude_hosts>192.168.1.0/24</exclude_hosts> fails

A fail returns:

Host scan finished.
Scan stopped with errors.
Scan interrupted
Scan stopped with errors.
CipherMonger commented 3 years ago

Yes, I have this set. Here is that section of my target XML: <exclude_hosts>192.168.200.59, 192.168.200.161</exclude_hosts>

In my case, I get Interrupted at 99%.

zenire commented 3 years ago

Yes, I have this set. Here is that section of my target XML: <exclude_hosts>192.168.200.59, 192.168.200.161</exclude_hosts>

In my case, I get Interrupted at 99%.

Could you try a scan where you only exclude 1 host/ip, instead of 2 and see if that works?

jjnicola commented 3 years ago

Hi @ZeNiRe @CipherMonger Ospd calculates the scan progress taking into account the total amount of hosts, dead hosts, excluded hosts, finished hosts. Additionally, if openvas is not able to resolve a host name, it will neither be scanned nor considered dead. If a scan was not stopped (ended), but the scan progress is less than 100%, it is considered "interrupted". It seems there is some issue with this calculation, where the scanner finishes but the progress is still below 100% for With the PRs greenbone/openvas#606 greenbone/ospd-openvas#343 and greenbone/ospd#332 this should be solved (these patches are not included in any stable release), as it is openvas which says now, how many hosts will be finally scanned and how many of them not, simplifying the calculation for ospd.

Despite this ends in the same "interrupted scan", it seems not to be the same issue reported by @tuxmaster5000. IIUC, the original issue is a F&F scan with one host. Therefore, not possible a non scanned host for exclusion, non-resolved, etc. I could reproduce the issue with non-resolved or with excluded hosts (as mentioned, fixed ), but not the original one reported by @tuxmaster5000.

tuxmaster5000 commented 3 years ago

@jjnicola, Yes in my case the scan not stops at 99% or so. It will stops randomly between 39% and 69%.

lyhistory commented 3 years ago

same issue, it would be better if gvm provider more detailed error msg instead of just saying stopped, here is the screenshots image image

bechavod commented 3 years ago

same image

Virsacer commented 3 years ago

I have the same problem: Interrupted at 76 % 20.8.0 Release

==> /gvm/var/log/gvm/ospd-openvas.log <==
OSPD[48] 2021-01-08 00:30:14,317: INFO: (ospd.command.command) Scan 6e9d313f-ab6e-44a7-91c0-0c0e8a166134 added to the queue in position 1.
OSPD[48] 2021-01-08 00:30:15,178: INFO: (ospd.ospd) Currently 1 queued scans.
OSPD[48] 2021-01-08 00:30:15,231: INFO: (ospd.ospd) Starting scan 6e9d313f-ab6e-44a7-91c0-0c0e8a166134.

==> /gvm/var/log/gvm/gvmd.log <==
event task:MESSAGE:2021-01-08 00h30.19 CET:2318: Status of task xxxxx (1efb9532-bda3-4935-9773-319ba47f0417) has changed to Running
WARNING: cipher_setiv: ivlen=15 blklen=16

==> /gvm/var/log/gvm/ospd-openvas.log <==
OSPD[48] 2021-01-08 02:11:29,582: INFO: (ospd.ospd) 6e9d313f-ab6e-44a7-91c0-0c0e8a166134: Host scan finished.
OSPD[48] 2021-01-08 02:11:29,584: INFO: (ospd.ospd) 6e9d313f-ab6e-44a7-91c0-0c0e8a166134: Scan interrupted.

==> /gvm/var/log/gvm/gvmd.log <==
event task:MESSAGE:2021-01-08 02h11.30 CET:2318: Status of task xxxxx (1efb9532-bda3-4935-9773-319ba47f0417) has changed to Interrupted

==> /gvm/var/log/gvm/ospd-openvas.log <==
OSPD[48] 2021-01-08 02:11:30,300: INFO: (ospd.ospd) 6e9d313f-ab6e-44a7-91c0-0c0e8a166134: Scan stopped with errors.
OSPD[48] 2021-01-08 02:11:30,301: INFO: (ospd.ospd) 6e9d313f-ab6e-44a7-91c0-0c0e8a166134: Scan interrupted.
OSPD[48] 2021-01-08 02:11:30,404: INFO: (ospd.ospd) 6e9d313f-ab6e-44a7-91c0-0c0e8a166134: Scan stopped with errors.
OSPD[48] 2021-01-08 02:11:30,404: INFO: (ospd.ospd) 6e9d313f-ab6e-44a7-91c0-0c0e8a166134: Scan interrupted.

In most situations it should be fine to mix the latest releases with using release branches. But personally I would never do that and I would not advise to do that. I don't even advise users to build from git checkouts. Even the release announcement says

GVM is published as regularly updated and tested source code releases.

So my general advise is to use the latest released versions. If you are familiar with fixing issues by yourself especially with debugging error messages like the one you got it is fine to build from git.

And when are the 31+100+91+36+221+167 commits in the 20.08-branches released as stable?

wisukind commented 3 years ago

I also have this problem. Happens, randomly, on various different tasks. gvm@ov-master-eqi:~$ gvmd --versionGreenbone Vulnerability Manager 20.08.0~git-3276b8ff-gvmd-20.08GIT revision 3276b8ff-gvmd-20.08Manager DB revision 233 gvm@ov-master-eqi:~$ uname -aLinux ov-master-eqi 5.0.0-32-generic

34~18.04.2-Ubuntu SMP Thu Oct 10 10:36:02 UTC 2019 x86_64 x86_64

x86_64 GNU/Linux On Fri, 2021-01-08 at 08:38 -0800, Virsacer wrote:

I have the same problem:

Interrupted at 76 %

20.8.0 Release ==> /gvm/var/log/gvm/ospd-openvas.log <==OSPD[48] 2021-01-08 00:30:14,317: INFO: (ospd.command.command) Scan 6e9d313f-ab6e-44a7- 91c0-0c0e8a166134 added to the queue in position 1.OSPD[48] 2021-01- 08 00:30:15,178: INFO: (ospd.ospd) Currently 1 queued scans.OSPD[48] 2021-01-08 00:30:15,231: INFO: (ospd.ospd) Starting scan 6e9d313f- ab6e-44a7-91c0-0c0e8a166134. ==> /gvm/var/log/gvm/gvmd.log <==event task:MESSAGE:2021-01-08 00h30.19 CET:2318: Status of task xxxxx (1efb9532-bda3-4935-9773- 319ba47f0417) has changed to RunningWARNING: cipher_setiv: ivlen=15 blklen=16 ==> /gvm/var/log/gvm/ospd-openvas.log <==OSPD[48] 2021-01-08 02:11:29,582: INFO: (ospd.ospd) 6e9d313f-ab6e-44a7-91c0-0c0e8a166134: Host scan finished.OSPD[48] 2021-01-08 02:11:29,584: INFO: (ospd.ospd) 6e9d313f-ab6e-44a7-91c0-0c0e8a166134: Scan interrupted. ==> /gvm/var/log/gvm/gvmd.log <==event task:MESSAGE:2021-01-08 02h11.30 CET:2318: Status of task xxxxx (1efb9532-bda3-4935-9773- 319ba47f0417) has changed to Interrupted ==> /gvm/var/log/gvm/ospd-openvas.log <==OSPD[48] 2021-01-08 02:11:30,300: INFO: (ospd.ospd) 6e9d313f-ab6e-44a7-91c0-0c0e8a166134: Scan stopped with errors.OSPD[48] 2021-01-08 02:11:30,301: INFO: (ospd.ospd) 6e9d313f-ab6e-44a7-91c0-0c0e8a166134: Scan interrupted.OSPD[48] 2021-01-08 02:11:30,404: INFO: (ospd.ospd) 6e9d313f-ab6e-44a7-91c0-0c0e8a166134: Scan stopped with errors.OSPD[48] 2021-01-08 02:11:30,404: INFO: (ospd.ospd) 6e9d313f- ab6e-44a7-91c0-0c0e8a166134: Scan interrupted.

In most situations it should be fine to mix the latest releases with using release branches. But personally I would never do that and I would not advise to do that. I don't even advise users to build from git checkouts. Even the release announcement says

GVM is published as regularly updated and tested source code releases.

So my general advise is to use the latest released versions. If you are familiar with fixing issues by yourself especially with debugging error messages like the one you got it is fine to build from git.

And when are the 31+100+91+36+221+167 commits in the 20.08-branches released as stable?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

jjnicola commented 3 years ago

Hello, I added more debug level log messages to ospd and ospd-openvas. Still not able to reproduce the issue, despite I tried with many different setups. I hope the new logs messages can help to find the issue. It would be nice if you could set debug log level and share the new logs. Also, if possible, please provide the scan config, target config and task config, as well as the environment you are scanning (master/sensor setup, through a firewall, etc). Does it happen always with the same target? Maybe you find a segmentation fault in some system log file (syslog, kernel.log, debug, messages,etc). Any information is helpful.

greenbone/ospd#352 greenbone/ospd-openvas#375

tuxmaster5000 commented 3 years ago

The look for segfault was my first idea, but in none of the logs of CentOS7 are something of this logged. As soon as an new version of ospd is released, I will try debug it again.

tuxmaster5000 commented 3 years ago

Now I have updated all components to the last version. Now it looks much more stable. On my first tests, no such error is occurs any more. The only problem is the gsa service, but this is another story.

jjnicola commented 3 years ago

Hi @tuxmaster5000, that's good news! thank you very much for the update. Many changes were done to tackle this issue despite I was not able to reproduce it, so we will keep an eye on it.

CipherMonger commented 3 years ago

I'm still running into issues on my setup, but I'm unsure how to enable debug logging. I'd be happy to do so if someone can point me in the right direction. Here's a rundown of what I am doing:

Scan config: Default "Full and fast"

Target config: 10.16.200.0/24, 192.168.0.0/24, 192.168.200.0/24, 192.168.201.0/24, 192.168.206.0/24, 192.168.207.0/24, 192.168.208.0/24, 192.168.210.0/24, 192.168.211.0/24, 192.168.212.0/24 Port list - All IANA assigned TCP

Task config: Add results to assets - yes Apply overrides - yes Min QoD - 70 Alterable task - yes Auto Delete Reports - Do not auto delete Scanner - OpenVAS Default Scan Config - Full and fast Order for target hosts - sequential Maximum concurrent NVTs - 4 Maximum concurrent hosts - 20

I can provide more detail if needed, just let me know.

cfi-gb commented 3 years ago

Could be also related / fixed with https://github.com/greenbone/ospd-openvas/pull/390 ?

jjnicola commented 3 years ago

@CipherMonger,

Thank you very much, I would appreciate very much more details!. I have been solving some issues related to interrupted scans, but there is still one which is quite hard to reproduce.

The first check is to check in the log if all hosts were scanned. At the beginning of the scan you get the total amount of hosts, and at the end, openvas logs how many hosts were scanned. You should be able to check this with the default log level, as this logs come from openvas.

The new debug log messages come from ospd-openvas, and they show the scan progress for each host. You can check (once you set the DEBUG level and if openvas scanned all hosts) if the last host finished or the scan is interrupted and the host never reached the 100% Options for ospd-openvas are set at the daemon startup. I don't know how you start ospd-openvas.

I have just one question now. Do you set the openvas option test_alive_hosts_only = yes ? The default is "no". So, don't worry if you don't know this option.

Thanks a lot again.

wisukind commented 3 years ago

Still happening with 20.08.1, although it's becoming rare. However, now there is no longer the "Could not connect to Scanner" error preceding the interrupted state change. It just happens, without reasons.

Greenbone Vulnerability Manager 20.08.1~git-df9da22ee-gvmd-20.08 GIT revision df9da22ee-gvmd-20.08 Manager DB revision 233 Copyright (C) 2009-2021 Greenbone Networks GmbH License: AGPL-3.0-or-later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

OpenVAS 20.8.2 gvm-libs 20.8.2~git-a34a5785-gvm-libs-20.08 Most new code since 2005: (C) 2021 Greenbone Networks GmbH Nessus origin: (C) 2004 Renaud Deraison deraison@nessus.org License GPLv2: GNU GPL version 2 This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

OSP Server for openvas: 20.8.2.dev1 OSP: 20.8.3.dev1 OSPd OpenVAS: 20.8.3.dev1

Happening on Linux Mint 19.3 Tricia, 64bits.

wisukind commented 3 years ago

I am now able to reproduce the issue systematically on one task. Here are the latest debug logs from ospd-openvas right before it happens:

OSPD[15899] 2021-03-23 15:16:07,838: DEBUG: (ospd_openvas.daemon) Plugin feed version: 202103231106 OSPD[15899] 2021-03-23 15:16:07,838: DEBUG: (ospd.ospd) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Current scan status: RUNNING, OSPD[15899] 2021-03-23 15:16:07,867: DEBUG: (ospd_openvas.daemon) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Inserting 22 results into scan collection table OSPD[15899] 2021-03-23 15:16:07,934: DEBUG: (ospd_openvas.daemon) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Inserting 24 results into scan collection table OSPD[15899] 2021-03-23 15:16:07,999: DEBUG: (ospd_openvas.daemon) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Inserting 23 results into scan collection table OSPD[15899] 2021-03-23 15:16:08,063: DEBUG: (ospd_openvas.daemon) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Inserting 22 results into scan collection table OSPD[15899] 2021-03-23 15:16:08,129: DEBUG: (ospd_openvas.daemon) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Inserting 23 results into scan collection table OSPD[15899] 2021-03-23 15:16:08,195: DEBUG: (ospd_openvas.daemon) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Inserting 24 results into scan collection table OSPD[15899] 2021-03-23 15:16:08,262: DEBUG: (ospd_openvas.daemon) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Inserting 24 results into scan collection table OSPD[15899] 2021-03-23 15:16:08,329: DEBUG: (ospd_openvas.daemon) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Inserting 23 results into scan collection table OSPD[15899] 2021-03-23 15:16:08,397: DEBUG: (ospd_openvas.daemon) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Inserting 24 results into scan collection table OSPD[15899] 2021-03-23 15:16:08,471: DEBUG: (ospd_openvas.daemon) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Inserting 24 results into scan collection table OSPD[15899] 2021-03-23 15:16:08,539: DEBUG: (ospd_openvas.daemon) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Inserting 23 results into scan collection table OSPD[15899] 2021-03-23 15:16:08,608: DEBUG: (ospd_openvas.daemon) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Inserting 21 results into scan collection table OSPD[15899] 2021-03-23 15:16:08,677: DEBUG: (ospd_openvas.daemon) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Inserting 23 results into scan collection table OSPD[15899] 2021-03-23 15:16:08,747: DEBUG: (ospd_openvas.daemon) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Inserting 24 results into scan collection table OSPD[15899] 2021-03-23 15:16:08,814: DEBUG: (ospd_openvas.daemon) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Inserting 23 results into scan collection table OSPD[15899] 2021-03-23 15:16:08,882: DEBUG: (ospd_openvas.daemon) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Inserting 23 results into scan collection table OSPD[15899] 2021-03-23 15:16:08,961: DEBUG: (ospd_openvas.daemon) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Inserting 24 results into scan collection table OSPD[15899] 2021-03-23 15:16:09,031: DEBUG: (ospd_openvas.daemon) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Inserting 25 results into scan collection table OSPD[15899] 2021-03-23 15:16:09,101: DEBUG: (ospd_openvas.daemon) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Inserting 26 results into scan collection table OSPD[15899] 2021-03-23 15:16:09,171: DEBUG: (ospd_openvas.daemon) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Inserting 27 results into scan collection table OSPD[15899] 2021-03-23 15:16:09,241: DEBUG: (ospd_openvas.daemon) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Inserting 24 results into scan collection table OSPD[15899] 2021-03-23 15:16:09,308: DEBUG: (ospd_openvas.daemon) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Inserting 11 results into scan collection table OSPD[15899] 2021-03-23 15:16:09,312: DEBUG: (ospd.scan) Calculating scan progress with the following data: total_hosts: 2538 exc_hosts: 2664 count_alive: 44 count_dead: 1229 host_prgresses: 100

OSPD[15899] 2021-03-23 15:16:09,312: DEBUG: (ospd_openvas.daemon) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Host 10.168.72.248 has progress: 100 OSPD[15899] 2021-03-23 15:16:09,323: DEBUG: (ospd_openvas.daemon) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Host 10.168.72.248 set HOST_END OSPD[15899] 2021-03-23 15:16:09,323: DEBUG: (ospd.scan) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Setting the following hosts as dead: [] OSPD[15899] 2021-03-23 15:16:09,324: DEBUG: (ospd.scan) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Setting the following hosts as finished: ['10.168.72.248'] OSPD[15899] 2021-03-23 15:16:09,324: DEBUG: (ospd.scan) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Remove the following hosts from the target list, as they are already finished or are dead: ['10.168.72.248'] OSPD[15899] 2021-03-23 15:16:10,465: DEBUG: (ospd_openvas.daemon) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Target is finished OSPD[15899] 2021-03-23 15:16:10,465: DEBUG: (ospd_openvas.daemon) 40fbf270-2791-4c90-8d1c-5bda816eb81e: End Target. Release main database OSPD[15899] 2021-03-23 15:16:10,466: INFO: (ospd.ospd) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Host scan finished. OSPD[15899] 2021-03-23 15:16:10,467: DEBUG: (ospd.ospd) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Current scan status: RUNNING, OSPD[15899] 2021-03-23 15:16:10,469: DEBUG: (ospd.scan) Calculating scan progress with the following data: total_hosts: 2538 exc_hosts: 2664 count_alive: 45 count_dead: 1229 host_prgresses: 0

OSPD[15899] 2021-03-23 15:16:10,470: DEBUG: (ospd.ospd) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Current scan progress: 0, OSPD[15899] 2021-03-23 15:16:10,470: INFO: (ospd.ospd) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Host scan finished. Progress: 0, Status: RUNNING OSPD[15899] 2021-03-23 15:16:10,470: DEBUG: (ospd.ospd) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Set scan status INTERRUPTED, OSPD[15899] 2021-03-23 15:16:10,470: INFO: (ospd.ospd) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Scan interrupted. OSPD[15899] 2021-03-23 15:16:10,471: DEBUG: (ospd.ospd) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Current scan progress: 0, OSPD[15899] 2021-03-23 15:16:10,474: DEBUG: (root) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Current progress: {'count_alive': 45, 'count_dead': 1229, 'count_excluded': 2664, 'count_total': 2538, 'current_hosts': {}, 'overall': 0} OSPD[15899] 2021-03-23 15:16:10,921: DEBUG: (ospd.server) New connection from ('10.194.157.7', 55298) OSPD[15899] 2021-03-23 15:16:11,479: DEBUG: (ospd.ospd) Handling get_scans command request. OSPD[15899] 2021-03-23 15:16:11,480: DEBUG: (ospd.ospd) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Current scan status: INTERRUPTED, OSPD[15899] 2021-03-23 15:16:11,480: DEBUG: (ospd.ospd) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Current scan progress: 0, OSPD[15899] 2021-03-23 15:16:11,494: INFO: (ospd.ospd) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Scan process is dead and its progress is 0 OSPD[15899] 2021-03-23 15:16:11,494: DEBUG: (ospd.ospd) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Set scan status INTERRUPTED, OSPD[15899] 2021-03-23 15:16:11,495: INFO: (ospd.ospd) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Scan interrupted. OSPD[15899] 2021-03-23 15:16:11,495: DEBUG: (ospd.ospd) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Check scan process: Progress 0 Status: INTERRUPTED

I'm thinking of upgrading ospd-openvas to the latest master branch and see if I can reproduce it. Thoughts ?

wisukind commented 3 years ago

On openvas side however,it looks like the scan has finished properly. Here are the latest logs from openvas when this happened:

lib misc:MESSAGE:2021-03-23 19h54.17 utc:29037: open_sock_tcp: 10.168.72.185:22 time-out. lib misc:MESSAGE:2021-03-23 19h54.17 utc:29041: open_sock_tcp: 10.168.72.185:80 time-out. lib misc:MESSAGE:2021-03-23 19h54.22 utc:29313: open_sock_tcp: 10.168.72.185:22 time-out. lib misc:MESSAGE:2021-03-23 19h54.25 utc:28918: open_sock_tcp: 10.168.72.185:443 time-out. lib misc:MESSAGE:2021-03-23 19h54.58 utc:29041: open_sock_tcp: 10.168.72.185:80 time-out. lib misc:MESSAGE:2021-03-23 19h55.01 utc:31365: open_sock_tcp: 10.168.72.185:22 time-out. lib misc:MESSAGE:2021-03-23 19h55.39 utc:888: open_sock_tcp: 10.168.72.185:5900 time-out. lib misc:MESSAGE:2021-03-23 19h56.19 utc:888: open_sock_tcp: 10.168.72.185:5900 time-out. lib misc:MESSAGE:2021-03-23 19h57.00 utc:4498: open_sock_tcp: 10.168.72.185:443 time-out. lib misc:MESSAGE:2021-03-23 19h57.40 utc:4498: open_sock_tcp: 10.168.72.185:443 time-out. lib misc:MESSAGE:2021-03-23 20h04.37 utc:22310: open_sock_tcp: 10.168.72.185:80 time-out. lib misc:MESSAGE:2021-03-23 20h05.22 utc:23952: open_sock_tcp: 10.168.72.185:5900 time-out. sd main:MESSAGE:2021-03-23 20h06.22 utc:24482: Vulnerability scan d06f2740-2fc3-4a76-835f-e80bc68ae378 finished for host 10.168.72.168 in 8038.54 seconds sd main:MESSAGE:2021-03-23 20h09.16 utc:5574: Vulnerability scan d06f2740-2fc3-4a76-835f-e80bc68ae378 finished for host 10.168.72.148 in 9579.65 seconds sd main:MESSAGE:2021-03-23 20h11.43 utc:20389: Vulnerability scan d06f2740-2fc3-4a76-835f-e80bc68ae378 finished for host 10.168.72.160 in 8480.50 seconds sd main:MESSAGE:2021-03-23 20h24.19 utc:20244: Vulnerability scan d06f2740-2fc3-4a76-835f-e80bc68ae378 finished for host 10.168.72.181 in 8345.88 seconds sd main:MESSAGE:2021-03-23 20h26.05 utc:30317: Vulnerability scan d06f2740-2fc3-4a76-835f-e80bc68ae378 finished for host 10.168.72.140 in 13812.41 seconds sd main:MESSAGE:2021-03-23 20h33.54 utc:23014: Vulnerability scan d06f2740-2fc3-4a76-835f-e80bc68ae378 finished for host 10.168.72.188 in 7944.55 seconds sd main:MESSAGE:2021-03-23 20h44.34 utc:29681: Vulnerability scan d06f2740-2fc3-4a76-835f-e80bc68ae378 finished for host 10.168.72.247 in 6737.75 seconds sd main:MESSAGE:2021-03-23 20h51.35 utc:30039: Vulnerability scan d06f2740-2fc3-4a76-835f-e80bc68ae378 finished for host 10.168.72.230 in 8744.28 seconds sd main:MESSAGE:2021-03-23 20h57.54 utc:7557: Vulnerability scan d06f2740-2fc3-4a76-835f-e80bc68ae378 finished for host 10.168.72.231 in 8104.23 seconds sd main:MESSAGE:2021-03-23 21h01.17 utc:25220: Vulnerability scan d06f2740-2fc3-4a76-835f-e80bc68ae378 finished for host 10.168.72.184 in 10417.37 seconds sd main:MESSAGE:2021-03-23 21h03.03 utc:24792: Vulnerability scan d06f2740-2fc3-4a76-835f-e80bc68ae378 finished for host 10.168.72.190 in 9615.79 seconds sd main:MESSAGE:2021-03-23 21h09.03 utc:23852: Vulnerability scan d06f2740-2fc3-4a76-835f-e80bc68ae378 finished for host 10.168.72.241 in 8326.90 seconds sd main:MESSAGE:2021-03-23 21h09.38 utc:22195: Vulnerability scan d06f2740-2fc3-4a76-835f-e80bc68ae378 finished for host 10.168.72.185 in 10125.86 seconds sd main:MESSAGE:2021-03-23 21h16.09 utc:23295: Vulnerability scan d06f2740-2fc3-4a76-835f-e80bc68ae378 finished for host 10.168.72.248 in 8041.47 seconds sd main:MESSAGE:2021-03-23 21h16.09 utc:15941: Vulnerability scan d06f2740-2fc3-4a76-835f-e80bc68ae378 finished in 22218 seconds: 1274 hosts

wisukind commented 3 years ago

I'll redo the same task with a different scanner, and see if I have the same problem.

jjnicola commented 3 years ago

Hi @wisukind, Thanks a lot for reporting this. I really appreciate your help in this bug hunting. I was trying to reproduce this. Looking at your logs, my first try was with target with many hosts excluded. But, one more time, I couldn't reproduce the issue.

What I find strange in your logs is that the total_hosts != alive_hosts + exclude_hosts + dead_hosts Also, Openvas says that it scanned 1274 hosts. Could you check in your log if you find something like this: ?

sd   main:MESSAGE:2021-03-24 14h58.24 utc:5629: exclude_hosts: Skipped 200 host(s).
sd   main:MESSAGE:2021-03-24 14h58.24 utc:5629: Vulnerability scan da6fd061-68a4-4f8e-a190-f465630f8fe9 started: Target has 54 hosts: 192.168.0.0/24, with max_hosts = 20 and max_checks = 4
alive scan:MESSAGE:2021-03-24 14h58.24 utc:5629: Alive scan da6fd061-68a4-4f8e-a190-f465630f8fe9 started: Target has 54 hosts
OSPD[4517] 2021-03-24 15:58:26,360: DEBUG: (ospd_openvas.daemon) 2ce9960a-42ce-45ed-984b-2d90371f7e30: **Set total hosts counted by OpenVAS**: 254

I set the target 192.168.0.0/24 and excluded the host range 192.168.0.1-192.168.0.200 If I understand correctly, It seems that your scanner didn't update the total amount of hosts.

My log shows:

OSPD[13125] 2021-03-24 16:42:34,086: DEBUG: (ospd_openvas.daemon) 285168d0-5efa-4d0b-86fc-cbe460ac8680: Target is finished
OSPD[13125] 2021-03-24 16:42:34,086: DEBUG: (ospd_openvas.daemon) 285168d0-5efa-4d0b-86fc-cbe460ac8680: End Target. Release main database
OSPD[13125] 2021-03-24 16:42:34,088: INFO: (ospd.ospd) 285168d0-5efa-4d0b-86fc-cbe460ac8680: Host scan finished.
OSPD[13125] 2021-03-24 16:42:34,089: DEBUG: (ospd.ospd) 285168d0-5efa-4d0b-86fc-cbe460ac8680: Current scan status: RUNNING,
OSPD[13125] 2021-03-24 16:42:34,089: DEBUG: (ospd.ospd) Calculating scan progress with the following data:
OSPD[13125] 2021-03-24 16:42:34,090: DEBUG: (ospd.ospd) 285168d0-5efa-4d0b-86fc-cbe460ac8680: Current scan progress: 100,
OSPD[13125] 2021-03-24 16:42:34,093: DEBUG: (root) 285168d0-5efa-4d0b-86fc-cbe460ac8680: Current progress: 
{'count_alive': 1,
 'count_dead': 53,
 'count_excluded': 200,
 'count_total': 254,
 'current_hosts': {},
 'overall': 100}
OSPD[13125] 2021-03-24 16:42:34,095: DEBUG: (ospd.ospd) 285168d0-5efa-4d0b-86fc-cbe460ac8680: Current scan progress: 100,
OSPD[13125] 2021-03-24 16:42:34,096: DEBUG: (ospd.ospd) 285168d0-5efa-4d0b-86fc-cbe460ac8680: Set scan status FINISHED,
OSPD[13125] 2021-03-24 16:42:34,097: INFO: (ospd.ospd) 285168d0-5efa-4d0b-86fc-cbe460ac8680: Scan finished.

There, you can see that the sum of hosts is the same as count_total.

Did you updated openvas, ospd, ospd-openvas? I have installed the same version, but updated with the last patches/commits:

.venv) jnicola@xxxxxx:~/dev/ospd-openvas$ ospd-openvas --version
OSP Server for openvas: 20.8.2.dev1
OSP: 20.8.3.dev1
OSPd OpenVAS: 20.8.3.dev1

(.venv) jnicola@xxxxx:~/dev/ospd-openvas$ openvas -V
OpenVAS 20.8.2
gvm-libs 20.8.2~git-e177033b-gvm-libs-20.08

Instead of trying with master, I would try with the current state of the mentioned modules, as master can be still unstable and error prone.

Thanks a lot again!

wisukind commented 3 years ago

Hi Juan,

I looked in my logs and I got the following:

openvas logs:

sd main:MESSAGE:2021-03-23 15h05.54 utc:15941: exclude_hosts: Skipped 1264 host(s). sd main:MESSAGE:2021-03-23 15h05.54 utc:15941: Vulnerability scan d06f2740-2fc3-4a76-835f-e80bc68ae378 started: Target has 1274 hosts: 10.168.75.128/27, 10.168.75.192/27, 10.168.79.128/25, 10.168.72.0/24, 10.168.76.0/23, 10.168.78.0/24, 10.168.73.0/24, 10.168.75.32/27, 10.224.0.0/27, 10.168.188.0/23, 10.168.190.0/23, with max_hosts = 20 and max_checks = 4

ospd-openvas logs:

OSPD[15899] 2021-03-23 09:05:54,578: DEBUG: (ospd_openvas.daemon) 40fbf270-2791-4c90-8d1c-5bda816eb81e: Set total hosts counted by OpenVAS: 2538

Regarding the version I'm using:

gvm@ov-slave-monterrey:/opt/gvm/var/log/gvm$ /opt/gvm/bin/ospd-scanner/bin/ospd-openvas --version OSP Server for openvas: 20.8.2.dev1 OSP: 20.8.3.dev1 OSPd OpenVAS: 20.8.3.dev1

and

gvm@ov-slave-monterrey:/opt/gvm/var/log/gvm$ openvas -V OpenVAS 20.8.2 gvm-libs 20.8.2~git-cc5089ae-gvm-libs-20.08

I havn't tried updating the scanner yet, but if you want I can update it and retry with the latest stable version. I am also currently scanning the same target with a different slave scanner. So far it's working fine, but it's still at 0% progress.

jjnicola commented 3 years ago

Hi @wisukind , I was able to reproduce the issue. I am pretty sure that you added some other network into the "excluded hosts" list, which is not present in the target list, or you just added a network/range twice. E.g.

My target: 192.168.0.0/24
Excluded hosts: 192.168.0.1-192.168.0.200, 192.168.1.0/24

Openvas only exclude the 200 hosts present in the target

sd   main:MESSAGE:2021-03-25 07h38.28 utc:27553: exclude_hosts: Skipped 200 host(s).
sd   main:MESSAGE:2021-03-25 07h38.28 utc:27553: Vulnerability scan f8a58ede-52fb-43d3-930c-c95033f2213c started: Target has 54 hosts: 192.168.0.0/24, with max_hosts = 20 and max_checks = 4

But ospd still count all excluded hosts, even those not present in the target list

OSPD[13125] 2021-03-25 08:38:37,354: INFO: (ospd.ospd) 6a5e2d95-469e-49ff-b328-a17e0659907e: Host scan finished. Progress: 0, Status: RUNNING
OSPD[13125] 2021-03-25 08:38:37,354: DEBUG: (ospd.ospd) 6a5e2d95-469e-49ff-b328-a17e0659907e: Set scan status INTERRUPTED,
OSPD[13125] 2021-03-25 08:38:37,355: INFO: (ospd.ospd) 6a5e2d95-469e-49ff-b328-a17e0659907e: Scan interrupted.
OSPD[13125] 2021-03-25 08:38:37,355: DEBUG: (ospd.ospd) 6a5e2d95-469e-49ff-b328-a17e0659907e: Current scan progress: 0,
OSPD[13125] 2021-03-25 08:38:37,360: DEBUG: (root) 6a5e2d95-469e-49ff-b328-a17e0659907e: Current progress: 
{'count_alive': 2,
 'count_dead': 52,
 'count_excluded': 454,
 'count_total': 254,
 'current_hosts': {},
 'overall': 0}
OSPD[13125] 2021-03-25 08:38:38,892: DEBUG: (ospd_openvas.daemon) Current feed version: 202103240547
OSPD[13125] 2021-03-25 08:38:38,893: DEBUG: (ospd_openvas.daemon) Plugin feed version: 202103240547
OSPD[13125] 2021-03-25 08:38:38,894: DEBUG: (ospd.ospd) 6a5e2d95-469e-49ff-b328-a17e0659907e: Current scan status: INTERRUPTED,

Therefore the excluded hosts is bigger than the count_total to be scanned.

Could you check this? If you confirm this, I think it is not necessary to check with other version. You will find the same issue.

Unfortunately, I think this is a different issue than the original one, in which the progress calculation ends with a value != 0 I am already working on this issue. Thanks a lot again!

Regards, Juan

wisukind commented 3 years ago

Indeed; problems started to happens after I've updated the exclude lists of those targets. I have cleaned it; and I am restarting the scan. Let's see if the problem persist.

Thanks

wisukind commented 3 years ago

Hi Juan,

I confirm cleaning the excluded list fix the problem. It seems ospd-openvas is confused when networks in excluded list are actually not part of the target ranges. This is a different problem, as you noticed, but still is a bug. This shouldn't happen. In case a network in the exclude list isn't part of the target list, it should just be ignored.

Thanks for catching up !

jjnicola commented 3 years ago

Hi ! thanks you for reporting. I am already working on a fix greenbone/ospd#377

CipherMonger commented 3 years ago

After running another scan yesterday, I checked the openvas.log output. The start of the scan has:

sd   main:MESSAGE:2021-03-25 13h34.55 utc:7966: openvas 20.8.0 started
sd   main:MESSAGE:2021-03-25 13h34.58 utc:7966: exclude_hosts: Skipped 2 host(s).
sd   main:MESSAGE:2021-03-25 13h35.16 utc:7966: Vulnerability scan 1322fbfe-b1f5-4d31-8a85-c69e49068e57 started: Target has 2528 hosts

The end of the scan shows:

sd main:MESSAGE:2021-03-25 23h06.13 utc:7966: Vulnerability scan 1322fbfe-b1f5-4d31-8a85-c69e49068e57 finished in 34278 seconds: 2528 hosts

So in my case, it looks like the numbers match. As far as the alive test, mine is set to "Use Scanner Default" (I'm using the web interface).

wisukind commented 3 years ago

Problem still exist with 21.04.1. I am able to reproduce the issue systematically on one task:

gvm@ov-slave-monterrey:~$ /opt/gvm/bin/ospd-scanner/bin/ospd-openvas --version OSP Server for openvas: 21.4.1 OSP: 21.4.1 OSPd OpenVAS: 21.4.1 gvm@ov-slave-monterrey:~$ gvmd --version Greenbone Vulnerability Manager 20.08.0~git-d47d159a-gvmd-20.08 GIT revision d47d159a-gvmd-20.08 Manager DB revision 233

Ospd-openvas logs read the following right before interruption:

OSPD[15586] 2021-06-07 16:00:35,069: DEBUG: (root) 26b33555-8984-45d8-b463-d87c5b1fcaad: Current progress: 
OSPD[15586] 2021-06-07 16:00:35,070: DEBUG: (ospd.scan) 26b33555-8984-45d8-b463-d87c5b1fcaad: Setting the following hosts as dead: []
OSPD[15586] 2021-06-07 16:00:35,070: DEBUG: (ospd.scan) 26b33555-8984-45d8-b463-d87c5b1fcaad: Setting the following hosts as finished: []
OSPD[15586] 2021-06-07 16:00:35,083: DEBUG: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Current scan status: RUNNING,
OSPD[15586] 2021-06-07 16:00:35,083: DEBUG: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Current scan progress: 67,
OSPD[15586] 2021-06-07 16:00:35,083: DEBUG: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Check scan process: 
OSPD[15586] 2021-06-07 16:00:35,084: DEBUG: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Current scan status: RUNNING,
OSPD[15586] 2021-06-07 16:00:35,084: DEBUG: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Current scan progress: 67,
OSPD[15586] 2021-06-07 16:00:35,084: DEBUG: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Current scan status: RUNNING,
OSPD[15586] 2021-06-07 16:00:35,123: DEBUG: (ospd_openvas.daemon) 26b33555-8984-45d8-b463-d87c5b1fcaad: Inserting 6 results into scan collection table
OSPD[15586] 2021-06-07 16:00:35,123: DEBUG: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Current scan progress: 67,
OSPD[15586] 2021-06-07 16:00:35,124: DEBUG: (root) 26b33555-8984-45d8-b463-d87c5b1fcaad: Current progress: 
OSPD[15586] 2021-06-07 16:00:35,125: DEBUG: (ospd.scan) 26b33555-8984-45d8-b463-d87c5b1fcaad: Setting the following hosts as dead: []
OSPD[15586] 2021-06-07 16:00:35,128: DEBUG: (ospd.scan) 26b33555-8984-45d8-b463-d87c5b1fcaad: Setting the following hosts as finished: []
OSPD[15586] 2021-06-07 16:00:35,179: DEBUG: (ospd_openvas.daemon) 26b33555-8984-45d8-b463-d87c5b1fcaad: Inserting 2 results into scan collection table
OSPD[15586] 2021-06-07 16:00:35,180: DEBUG: (ospd_openvas.daemon) 26b33555-8984-45d8-b463-d87c5b1fcaad: Host 10.168.68.19 has progress: 100
OSPD[15586] 2021-06-07 16:00:35,180: DEBUG: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Current scan progress: 67,
OSPD[15586] 2021-06-07 16:00:35,181: DEBUG: (root) 26b33555-8984-45d8-b463-d87c5b1fcaad: Current progress: 
OSPD[15586] 2021-06-07 16:00:35,181: DEBUG: (ospd.scan) 26b33555-8984-45d8-b463-d87c5b1fcaad: Setting the following hosts as dead: []
OSPD[15586] 2021-06-07 16:00:35,181: DEBUG: (ospd.scan) 26b33555-8984-45d8-b463-d87c5b1fcaad: Setting the following hosts as finished: ['10.168.68.19']
OSPD[15586] 2021-06-07 16:00:35,182: DEBUG: (ospd.scan) 26b33555-8984-45d8-b463-d87c5b1fcaad: Remove the following hosts from the target list, as they are already finished or are dead: ['10.168.68.19']
OSPD[15586] 2021-06-07 16:00:35,233: DEBUG: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Current scan progress: 67,
OSPD[15586] 2021-06-07 16:00:35,234: DEBUG: (root) 26b33555-8984-45d8-b463-d87c5b1fcaad: Current progress: 
OSPD[15586] 2021-06-07 16:00:35,234: DEBUG: (ospd.scan) 26b33555-8984-45d8-b463-d87c5b1fcaad: Setting the following hosts as dead: []
OSPD[15586] 2021-06-07 16:00:35,235: DEBUG: (ospd.scan) 26b33555-8984-45d8-b463-d87c5b1fcaad: Setting the following hosts as finished: []
OSPD[15586] 2021-06-07 16:00:35,696: DEBUG: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Results sent successfully to the client. Cleaning temporary result list.
OSPD[15586] 2021-06-07 16:00:36,238: DEBUG: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Current scan progress: 67,
OSPD[15586] 2021-06-07 16:00:36,238: DEBUG: (root) 26b33555-8984-45d8-b463-d87c5b1fcaad: Current progress: 
OSPD[15586] 2021-06-07 16:00:36,239: DEBUG: (ospd.scan) 26b33555-8984-45d8-b463-d87c5b1fcaad: Setting the following hosts as dead: []
OSPD[15586] 2021-06-07 16:00:36,239: DEBUG: (ospd.scan) 26b33555-8984-45d8-b463-d87c5b1fcaad: Setting the following hosts as finished: []
OSPD[15586] 2021-06-07 16:00:36,240: DEBUG: (ospd_openvas.daemon) 26b33555-8984-45d8-b463-d87c5b1fcaad: Target is finished
OSPD[15586] 2021-06-07 16:00:36,240: DEBUG: (ospd_openvas.daemon) 26b33555-8984-45d8-b463-d87c5b1fcaad: End Target. Release main database
OSPD[15586] 2021-06-07 16:00:36,241: INFO: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Host scan finished.
OSPD[15586] 2021-06-07 16:00:36,241: DEBUG: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Current scan status: RUNNING,
OSPD[15586] 2021-06-07 16:00:36,241: DEBUG: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Current scan progress: 67,
OSPD[15586] 2021-06-07 16:00:36,242: DEBUG: (root) 26b33555-8984-45d8-b463-d87c5b1fcaad: Current progress: 
OSPD[15586] 2021-06-07 16:00:36,242: DEBUG: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Current scan progress: 67,
OSPD[15586] 2021-06-07 16:00:36,242: INFO: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Host scan finished. Progress: 67, Status: RUNNING
OSPD[15586] 2021-06-07 16:00:36,242: DEBUG: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Set scan status INTERRUPTED,
OSPD[15586] 2021-06-07 16:00:36,243: INFO: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Scan interrupted.
OSPD[15586] 2021-06-07 16:00:36,243: DEBUG: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Current scan progress: 67,
OSPD[15586] 2021-06-07 16:00:36,244: DEBUG: (root) 26b33555-8984-45d8-b463-d87c5b1fcaad: Current progress: 
OSPD[15586] 2021-06-07 16:00:38,263: DEBUG: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Current scan status: INTERRUPTED,
OSPD[15586] 2021-06-07 16:00:38,263: DEBUG: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Current scan progress: 67,
OSPD[15586] 2021-06-07 16:00:38,264: INFO: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Scan process is dead and its progress is 67
OSPD[15586] 2021-06-07 16:00:38,264: DEBUG: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Set scan status INTERRUPTED,
OSPD[15586] 2021-06-07 16:00:38,264: INFO: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Scan interrupted.
OSPD[15586] 2021-06-07 16:00:38,264: DEBUG: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Check scan process: 
OSPD[15586] 2021-06-07 16:00:38,265: DEBUG: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Current scan status: INTERRUPTED,
OSPD[15586] 2021-06-07 16:00:38,265: DEBUG: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Current scan progress: 67,
OSPD[15586] 2021-06-07 16:00:38,265: DEBUG: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Current scan status: INTERRUPTED,
OSPD[15586] 2021-06-07 16:00:38,266: DEBUG: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Results sent successfully to the client. Cleaning temporary result list.
OSPD[15586] 2021-06-07 16:00:39,357: DEBUG: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Current scan status: INTERRUPTED,
OSPD[15586] 2021-06-07 16:00:39,357: DEBUG: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Current scan progress: 67,
OSPD[15586] 2021-06-07 16:00:39,357: INFO: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Scan process is dead and its progress is 67
OSPD[15586] 2021-06-07 16:00:39,358: DEBUG: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Set scan status INTERRUPTED,
OSPD[15586] 2021-06-07 16:00:39,358: INFO: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Scan interrupted.
OSPD[15586] 2021-06-07 16:00:39,358: DEBUG: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Check scan process: 
OSPD[15586] 2021-06-07 16:00:39,358: DEBUG: (ospd.ospd) 26b33555-8984-45d8-b463-d87c5b1fcaad: Current scan status: INTERRUPTED,

While openvas logs reads:

sd   main:MESSAGE:2021-06-07 17h12.36 utc:26209: Vulnerability scan 26b33555-8984-45d8-b463-d87c5b1fcaad started for host: 10.168.68.62
sd   main:MESSAGE:2021-06-07 17h12.56 utc:17406: Vulnerability scan 26b33555-8984-45d8-b463-d87c5b1fcaad finished for host 10.168.64.80 in 3021.22 seconds
sd   main:MESSAGE:2021-06-07 17h14.59 utc:20715: Vulnerability scan 26b33555-8984-45d8-b463-d87c5b1fcaad finished for host 10.168.64.82 in 3013.48 seconds
sd   main:MESSAGE:2021-06-07 17h16.32 utc:14830: Vulnerability scan 26b33555-8984-45d8-b463-d87c5b1fcaad finished for host 10.168.69.178 in 28058.06 seconds
sd   main:MESSAGE:2021-06-07 17h17.37 utc:27318: Vulnerability scan 26b33555-8984-45d8-b463-d87c5b1fcaad finished for host 10.168.64.89 in 2919.64 seconds
sd   main:MESSAGE:2021-06-07 17h18.27 utc:16664: Vulnerability scan 26b33555-8984-45d8-b463-d87c5b1fcaad finished for host 10.168.67.139 in 6660.24 seconds
sd   main:MESSAGE:2021-06-07 17h19.26 utc:19026: Vulnerability scan 26b33555-8984-45d8-b463-d87c5b1fcaad finished for host 10.168.64.75 in 6589.39 seconds
sd   main:MESSAGE:2021-06-07 17h25.13 utc:22916: Vulnerability scan 26b33555-8984-45d8-b463-d87c5b1fcaad finished for host 10.168.64.227 in 3535.93 seconds
sd   main:MESSAGE:2021-06-07 17h25.38 utc:17881: Vulnerability scan 26b33555-8984-45d8-b463-d87c5b1fcaad finished for host 10.168.67.107 in 7031.41 seconds
sd   main:MESSAGE:2021-06-07 17h25.42 utc:16810: Vulnerability scan 26b33555-8984-45d8-b463-d87c5b1fcaad finished for host 10.168.64.223 in 2598.90 seconds
sd   main:MESSAGE:2021-06-07 17h30.05 utc:16237: Vulnerability scan 26b33555-8984-45d8-b463-d87c5b1fcaad finished for host 10.168.64.250 in 4093.58 seconds
sd   main:MESSAGE:2021-06-07 17h31.25 utc:620: Vulnerability scan 26b33555-8984-45d8-b463-d87c5b1fcaad finished for host 10.168.64.77 in 6616.97 seconds
sd   main:MESSAGE:2021-06-07 17h39.07 utc:2648: Vulnerability scan 26b33555-8984-45d8-b463-d87c5b1fcaad finished for host 10.168.68.50 in 2561.54 seconds
sd   main:MESSAGE:2021-06-07 17h42.50 utc:20381: Vulnerability scan 26b33555-8984-45d8-b463-d87c5b1fcaad finished for host 10.168.64.229 in 3457.16 seconds
sd   main:MESSAGE:2021-06-07 17h45.05 utc:7141: Vulnerability scan 26b33555-8984-45d8-b463-d87c5b1fcaad finished for host 10.168.68.49 in 2767.27 seconds
sd   main:MESSAGE:2021-06-07 17h50.14 utc:26209: Vulnerability scan 26b33555-8984-45d8-b463-d87c5b1fcaad finished for host 10.168.68.62 in 2258.95 seconds
sd   main:MESSAGE:2021-06-07 17h52.46 utc:435: Vulnerability scan 26b33555-8984-45d8-b463-d87c5b1fcaad finished for host 10.168.68.18 in 3459.56 seconds
sd   main:MESSAGE:2021-06-07 18h11.29 utc:10152: Vulnerability scan 26b33555-8984-45d8-b463-d87c5b1fcaad finished for host 10.168.64.228 in 5624.51 seconds
sd   main:MESSAGE:2021-06-07 18h25.55 utc:23007: Vulnerability scan 26b33555-8984-45d8-b463-d87c5b1fcaad finished for host 10.168.68.55 in 5926.74 seconds
sd   main:MESSAGE:2021-06-07 18h59.47 utc:26455: Vulnerability scan 26b33555-8984-45d8-b463-d87c5b1fcaad finished for host 10.168.68.20 in 7764.62 seconds
sd   main:MESSAGE:2021-06-07 21h00.35 utc:22287: Vulnerability scan 26b33555-8984-45d8-b463-d87c5b1fcaad finished for host 10.168.68.19 in 15247.11 seconds
sd   main:MESSAGE:2021-06-07 21h00.35 utc:5831: Vulnerability scan 26b33555-8984-45d8-b463-d87c5b1fcaad finished in 51698 seconds: 129 alive hosts of 2046

So indeed it seems the scan is finished but for some reason ospd-openvas put it as Interrupted.

Target details is as below:

image There is no IP exclusion list.

Task detail is as below:

image

wisukind commented 3 years ago

Note also that in my case; although the task is set as finished on openvas side; If I resume the task after Interruption it will resume where it left before crashing (67%), and after some time will be set to Interrupted state again. Restarting the task from scratch doesn't seems to help.

Kraemii commented 3 years ago

Hi! Thank you for your report on this topic. I was working on this for the past week and as before no one in my team was able to reproduce it. What bothers me is that the times in the logs that you posted does not match. You said you are able to reproduce the issue systematically, does that mean on the one specific task you always run into the same error with an interrupted scan? I am not quite sure: which version of ospd-openvas and openvas did you use? I added a log message to narrow down a specific case in which the scanner could possibly get set as interrupted even so openvas is not finished. This is in the newest versions of 20.08, 21.04 and master branch.

Again thank you for your participation in hunting this bug!

wisukind commented 3 years ago

Hello, Yes, I confirm I can reproduce the problem on one particular task, from one particular scanner. I have different slave scanners running, and one in particular is having this problem. Others never (or very barely) suffer this issue. So, If I run a particular task from a particular slave scanner, I reproduce the issue systematically (at least; my 3 latest tentatives in that scenario always ended in interrupted state at the same percentage). I am using the following version:

@.***:~$ /opt/gvm/bin/ospd-scanner/bin/ospd-openvas --version OSP Server for openvas: 21.4.1OSP: 21.4.1OSPd OpenVAS: 21.4.1

@.***:~$ openvas --version OpenVAS 21.4.1~git-51dcbd2f- openvas-21.04GIT revision ~git-51dcbd2f-openvas-21.04gvm-libs 21.4.1~git-714a5ea2-gvm-libs-21.04

I'll install the latest 21.04 stable version and try again on the same target. Hopefully the new error message will give us more informations. Thanks ! On Tue, 2021-06-15 at 06:51 -0700, Christoph Krämer wrote:

Hi!

Thank you for your report on this topic. I was working on this for the past week and as before no one in my team was able to reproduce it. What bothers me is that the times in the logs that you posted does not match.

You said you are able to reproduce the issue systematically, does that mean on the one specific task you always run into the same error with an interrupted scan?

I am not quite sure: which version of ospd-openvas and openvas did you use?

I added a log message to narrow down a specific case in which the scanner could possibly get set as interrupted even so openvas is not finished. This is in the newest versions of 20.08, 21.04 and master branch. Again thank you for your participation in hunting this bug!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

wisukind commented 3 years ago

I've installed latest 21.04 and currently re-running the failing task with it. I'll let you know.*

gvm@ov-slave-monterrey:~$ openvas --version
OpenVAS 21.4.1~git-7d6eff81-openvas-21.04
GIT revision ~git-7d6eff81-openvas-21.04
gvm-libs 21.4.1~git-714a5ea2-gvm-libs-21.04
Most new code since 2005: (C) 2021 Greenbone Networks GmbH
Nessus origin: (C) 2004 Renaud Deraison <deraison@nessus.org>
License GPLv2: GNU GPL version 2
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

gvm@ov-slave-monterrey:~$ /opt/gvm/bin/ospd-scanner/bin/ospd-openvas --version
OSP Server for openvas: 21.4.1
OSP: 21.4.1
OSPd OpenVAS: 21.4.1
wisukind commented 3 years ago

I've installed latest 21.04 and currently re-running the failing task with it. I'll let you know.*

gvm@ov-slave-monterrey:~$ openvas --version OpenVAS 21.4.1~git-7d6eff81-openvas-21.04 GIT revision ~git-7d6eff81-openvas-21.04 gvm-libs 21.4.1~git-714a5ea2-gvm-libs-21.04 Most new code since 2005: (C) 2021 Greenbone Networks GmbH Nessus origin: (C) 2004 Renaud Deraison deraison@nessus.org License GPLv2: GNU GPL version 2 This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

gvm@ov-slave-monterrey:~$ /opt/gvm/bin/ospd-scanner/bin/ospd-openvas --version OSP Server for openvas: 21.4.1 OSP: 21.4.1 OSPd OpenVAS: 21.4.1