search

secureCodeBox / scanner-webapplication-zap

Part of the deprecated secureCodeBox v1, see secureCodeBox/secureCodeBox Repo for v2
Apache License 2.0
6 stars 4 forks source link

Develop: Zap stays in Scanner #26

Open wurstbrot opened 5 years ago

wurstbrot commented 5 years ago

I am using docker-compose.yml with a self build engine and zap. Both are current develop-branch without modifications. I scan the juice shop via swagger: [ { "name": "ZAP Scan 10.10.11.104", "location": "http://10.10.11.104:3000/", "attributes": { "ZAP_BASE_URL": "http://10.10.11.104:3000/", "ZAP_SPIDER_MAX_DEPTH": 1 } } ]

I am not sure which warning/error in the following log is the cause for zap to stay Scanner after finishing "Run OWASP Zap Spider" in the engine.

I see the following in the zap-container-logs:

214907 [ZAP-ProxyThread-70] INFO org.zaproxy.zap.extension.httpsessions.HttpSessionsSite  - Setting new active session for site '10.10.11.104:3000': HttpSession [name=secureCodeBoxSession, active=false, tokenValues='']
214911 [ZAP-ProxyThread-71] INFO org.zaproxy.zap.extension.httpsessions.HttpSessionsSite  - Setting new active session for site '10.10.11.104:3000': HttpSession [name=secureCodeBoxSession, active=true, tokenValues='']
2018-11-16 12:50:00.561  INFO 118 --- [pool-1-thread-4] i.s.zap.service.zap.ZapService           : Recalling 0 requests to zap.
2018-11-16 12:50:00.561  INFO 118 --- [pool-1-thread-4] i.s.zap.service.zap.ZapService           : Starting scanner for targetUrl 'http://10.10.11.104:3000/main.js' and userId -1.
2018-11-16 12:50:00.594  INFO 118 --- [pool-1-thread-4] i.s.zap.service.zap.ZapService           : No custom ZAP replacer rule defined yet.
214949 [ZAP-ProxyThread-78] WARN org.zaproxy.zap.extension.api.API  - Bad request to API endpoint [/xml/ascan/action/scan/] from [127.0.0.1]:
URL Not Found in the Scan Tree (url_not_found)
    at org.zaproxy.zap.extension.ascan.ActiveScanAPI.scanURL(ActiveScanAPI.java:779)
    at org.zaproxy.zap.extension.ascan.ActiveScanAPI.handleApiAction(ActiveScanAPI.java:293)
    at org.zaproxy.zap.extension.api.API.handleApiRequest(API.java:431)
    at org.parosproxy.paros.core.proxy.ProxyThread.processHttp(ProxyThread.java:456)
    at org.parosproxy.paros.core.proxy.ProxyThread.run(ProxyThread.java:317)
    at java.lang.Thread.run(Thread.java:748)
2018-11-16 12:50:00.604 ERROR 118 --- [pool-1-thread-4] i.s.zap.jobs.definition.EngineWorkerJob  : Job execution error!

org.zaproxy.clientapi.core.ClientApiException: URL Not Found in the Scan Tree
    at org.zaproxy.clientapi.core.ApiResponseFactory.getResponse(ApiResponseFactory.java:50) ~[zap-clientapi-1.6.0.jar!/:1.6.0]
    at org.zaproxy.clientapi.core.ClientApi.callApi(ClientApi.java:332) ~[zap-clientapi-1.6.0.jar!/:1.6.0]
    at org.zaproxy.clientapi.gen.Ascan.scan(Ascan.java:278) ~[zap-clientapi-1.6.0.jar!/:1.6.0]
    at org.zaproxy.clientapi.gen.Ascan.scan(Ascan.java:236) ~[zap-clientapi-1.6.0.jar!/:1.6.0]
    at io.securecodebox.zap.service.zap.ZapService.startScannerAsUser(ZapService.java:260) ~[classes!/:na]
    at io.securecodebox.zap.jobs.definition.EngineWorkerJob.executeScanner(EngineWorkerJob.java:239) ~[classes!/:na]
    at io.securecodebox.zap.jobs.definition.EngineWorkerJob.performScannerTask(EngineWorkerJob.java:179) ~[classes!/:na]
    at io.securecodebox.zap.jobs.definition.EngineWorkerJob.execute(EngineWorkerJob.java:108) ~[classes!/:na]
    at de.otto.edison.jobs.service.JobService$1.execute(JobService.java:252) [edison-jobs-0.82.2.jar!/:na]
    at de.otto.edison.jobs.service.JobRunner.executeAndRetry(JobRunner.java:61) [edison-jobs-0.82.2.jar!/:na]
    at de.otto.edison.jobs.service.JobRunner.start(JobRunner.java:50) [edison-jobs-0.82.2.jar!/:na]
    at de.otto.edison.jobs.service.JobService.lambda$startAsync$7(JobService.java:216) [edison-jobs-0.82.2.jar!/:na]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_131]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_131]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_131]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_131]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_131]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_131]
    at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]

2018-11-16 12:50:00.728 ERROR 118 --- [pool-1-thread-4] d.o.e.jobs.eventbus.LogJobEventListener  : 'Fatal error in job engine/worker/owasp/zap (64287835-c2f1-42bd-aee7-e602b53f5034)
org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://engine:8080/box/jobs/ff1825e9-e99d-11e8-a098-0242ac120006/failure": http://engine:8080/box/jobs/ff1825e9-e99d-11e8-a098-0242ac120006/failure; nested exception is java.io.FileNotFoundException: http://engine:8080/box/jobs/ff1825e9-e99d-11e8-a098-0242ac120006/failure
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:674)
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:621)
    at org.springframework.web.client.RestTemplate.postForEntity(RestTemplate.java:415)
    at io.securecodebox.zap.service.engine.EngineTaskApiClient.reportFailure(EngineTaskApiClient.java:145)
    at io.securecodebox.zap.service.engine.ZapTaskService.reportFailure(ZapTaskService.java:66)
    at io.securecodebox.zap.jobs.definition.EngineWorkerJob.execute(EngineWorkerJob.java:111)
    at de.otto.edison.jobs.service.JobService$1.execute(JobService.java:252)
    at de.otto.edison.jobs.service.JobRunner.executeAndRetry(JobRunner.java:61)
    at de.otto.edison.jobs.service.JobRunner.start(JobRunner.java:50)
    at de.otto.edison.jobs.service.JobService.lambda$startAsync$7(JobService.java:216)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: http://engine:8080/box/jobs/ff1825e9-e99d-11e8-a098-0242ac120006/failure
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1872)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
    at org.springframework.http.client.SimpleClientHttpResponse.getBody(SimpleClientHttpResponse.java:85)
    at org.springframework.http.client.BufferingClientHttpResponseWrapper.getBody(BufferingClientHttpResponseWrapper.java:69)
    at io.securecodebox.zap.service.engine.LoggingRequestInterceptor.traceResponse(LoggingRequestInterceptor.java:58)
    at io.securecodebox.zap.service.engine.LoggingRequestInterceptor.intercept(LoggingRequestInterceptor.java:44)
    at org.springframework.http.client.InterceptingClientHttpRequest$InterceptingRequestExecution.execute(InterceptingClientHttpRequest.java:88)
    at org.springframework.http.client.InterceptingClientHttpRequest.executeInternal(InterceptingClientHttpRequest.java:72)
    at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
    at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:660)
    ... 16 more
': '64287835-c2f1-42bd-aee7-e602b53f5034'
2018-11-16 12:50:00.730 ERROR 118 --- [pool-1-thread-4] de.otto.edison.jobs.service.JobRunner    : Fatal error in job engine/worker/owasp/zap (64287835-c2f1-42bd-aee7-e602b53f5034)

org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://engine:8080/box/jobs/ff1825e9-e99d-11e8-a098-0242ac120006/failure": http://engine:8080/box/jobs/ff1825e9-e99d-11e8-a098-0242ac120006/failure; nested exception is java.io.FileNotFoundException: http://engine:8080/box/jobs/ff1825e9-e99d-11e8-a098-0242ac120006/failure
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:674) ~[spring-web-4.3.15.RELEASE.jar!/:4.3.15.RELEASE]
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:621) ~[spring-web-4.3.15.RELEASE.jar!/:4.3.15.RELEASE]
    at org.springframework.web.client.RestTemplate.postForEntity(RestTemplate.java:415) ~[spring-web-4.3.15.RELEASE.jar!/:4.3.15.RELEASE]
    at io.securecodebox.zap.service.engine.EngineTaskApiClient.reportFailure(EngineTaskApiClient.java:145) ~[classes!/:na]
    at io.securecodebox.zap.service.engine.ZapTaskService.reportFailure(ZapTaskService.java:66) ~[classes!/:na]
    at io.securecodebox.zap.jobs.definition.EngineWorkerJob.execute(EngineWorkerJob.java:111) ~[classes!/:na]
    at de.otto.edison.jobs.service.JobService$1.execute(JobService.java:252) ~[edison-jobs-0.82.2.jar!/:na]
    at de.otto.edison.jobs.service.JobRunner.executeAndRetry(JobRunner.java:61) [edison-jobs-0.82.2.jar!/:na]
    at de.otto.edison.jobs.service.JobRunner.start(JobRunner.java:50) [edison-jobs-0.82.2.jar!/:na]
    at de.otto.edison.jobs.service.JobService.lambda$startAsync$7(JobService.java:216) [edison-jobs-0.82.2.jar!/:na]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_131]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_131]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_131]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_131]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_131]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_131]
    at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.io.FileNotFoundException: http://engine:8080/box/jobs/ff1825e9-e99d-11e8-a098-0242ac120006/failure
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1872) ~[na:1.8.0_131]
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474) ~[na:1.8.0_131]
    at org.springframework.http.client.SimpleClientHttpResponse.getBody(SimpleClientHttpResponse.java:85) ~[spring-web-4.3.15.RELEASE.jar!/:4.3.15.RELEASE]
    at org.springframework.http.client.BufferingClientHttpResponseWrapper.getBody(BufferingClientHttpResponseWrapper.java:69) ~[spring-web-4.3.15.RELEASE.jar!/:4.3.15.RELEASE]
    at io.securecodebox.zap.service.engine.LoggingRequestInterceptor.traceResponse(LoggingRequestInterceptor.java:58) ~[classes!/:na]
    at io.securecodebox.zap.service.engine.LoggingRequestInterceptor.intercept(LoggingRequestInterceptor.java:44) ~[classes!/:na]
    at org.springframework.http.client.InterceptingClientHttpRequest$InterceptingRequestExecution.execute(InterceptingClientHttpRequest.java:88) ~[spring-web-4.3.15.RELEASE.jar!/:4.3.15.RELEASE]
    at org.springframework.http.client.InterceptingClientHttpRequest.executeInternal(InterceptingClientHttpRequest.java:72) ~[spring-web-4.3.15.RELEASE.jar!/:4.3.15.RELEASE]
    at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) ~[spring-web-4.3.15.RELEASE.jar!/:4.3.15.RELEASE]
    at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53) ~[spring-web-4.3.15.RELEASE.jar!/:4.3.15.RELEASE]
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:660) ~[spring-web-4.3.15.RELEASE.jar!/:4.3.15.RELEASE]
rseedorff commented 5 years ago

Hi Timo,

you LOG file indicates that your ZAP Spider returns no URLs, in this case the ZAP Scanner complains that its not able to scan...

2018-11-16 12:50:00.561 INFO 118 --- [pool-1-thread-4] i.s.zap.service.zap.ZapService : Recalling 0 requests to zap.

214949 [ZAP-ProxyThread-78] WARN org.zaproxy.zap.extension.api.API - Bad request to API endpoint [/xml/ascan/action/scan/] from [127.0.0.1]: URL Not Found in the Scan Tree (url_not_found)

Have you tried to scan your service (http://10.10.11.104:3000/) local with ZAP first, to test your configuration you would like to automate? Maybe the ZAP_SPIDER_MAX_DEPTH must be >= 1?

KR Robert

wurstbrot commented 5 years ago

Hi Robert,

I tried it now with:

In all cases, I get the same error. In engine I get: engine_1 | 2018-11-16 14:19:02.640 INFO 1 --- [nio-8080-exec-6] s.s.l.TransformFindingsToTargetsListener : Created Targets out of Findings: [Target{name='null', location='http://bodgeit:8080/bodgeit/js/util.js', attributes={request={method=GET, url=http://bodgeit:8080/bodgeit/js/util.js, httpVersion=HTTP/1.1, cookies=[{name=JSESSIONID, value=6FD5717DB5D22B76C9A17F55BCBBD698}, {name=b_id, value=2}], headers=[{name=User-Agent, value=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0}, {name=Pragma, value=no-cache}, {name=Cache-Control, value=no-cache}, {name=Content-Length, value=0}, {name=Referer, value=http://bodgeit:8080/bodgeit/}, {name=Host, value=bodgeit:8080}, {name=Cookie, value=b_id=2; JSESSIONID=6FD5717DB5D22B76C9A17F55BCBBD698}], queryString=[], postData={mimeType=, params=[], text=}, headersSize=323, bodySize=0}, ZAP_BASE_URL=http://bodgeit:8080/bodgeit/}}, Target{name='null', location='http://bodgeit:8080/bodgeit/search.jsp', attributes={request={method=GET, url=http://bodgeit:8080/bodgeit/search.jsp, httpVersion=HTTP/1.1, cookies=[{name=JSESSIONID, value=6FD5717DB5D22B76C9A17F55BCBBD698}], headers=[{name=User-Agent, value=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0}, {name=Pragma, value=no-cache}, {name=Cache-Control, value=no-cache}, {name=Content-Length, value=0}, {name=Referer, value=http://bodgeit:8080/bodgeit/}, {name=Host, value=bodgeit:8080}, {name=Cookie, value=JSESSIONID=6FD5717DB5D22B76C9A17F55BCBBD698}], queryString=[], postData={mimeType=, params=[], text=}, headersSize=315, bodySize=0}, ZAP_BASE_URL=http://bodgeit:8080/bodgeit/}}, Target{name='null', location='http://bodgeit:8080/bodgeit/basket.jsp', attributes={request={method=GET, url=http://bodgeit:8080/bodgeit/basket.jsp, httpVersion=HTTP/1.1, cookies=[{name=JSESSIONID, value=6FD5717DB5D22B76C9A17F55BCBBD698}], headers=[{name=User-Agent, value=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0}, {name=Pragma, value=no-cache}, {name=Cache-Control, value=no-cache}, {name=Content-Length, value=0}, {name=Referer, value=http://bodgeit:8080/bodgeit/}, {name=Host, value=bodgeit:8080}, {name=Cookie, value=JSESSIONID=6FD5717DB5D22B76C9A17F55BCBBD698}], queryString=[], postData={mimeType=, params=[], text=}, headersSize=315, bodySize=0}, ZAP_BASE_URL=http://bodgeit:8080/bodgeit/}}, Target{name='null', location='http://bodgeit:8080/bodgeit/product.jsp?typeid=5', attributes={request={method=GET, url=http://bodgeit:8080/bodgeit/product.jsp?typeid=5, httpVersion=HTTP/1.1, cookies=[{name=JSESSIONID, value=6FD5717DB5D22B76C9A17F55BCBBD698}, {name=b_id, value=2}], headers=[{name=User-Agent, value=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0}, {name=Pragma, value=no-cache}, {name=Cache-Control, value=no-cache}, {name=Content-Length, value=0}, {name=Referer, value=http://bodgeit:8080/bodgeit/}, {name=Host, value=bodgeit:8080}, {name=Cookie, value=b_id=2; JSESSIONID=6FD5717DB5D22B76C9A17F55BCBBD698}], queryString=[{name=typeid, value=5}], postData={mimeType=, params=[], text=}, headersSize=333, bodySize=0}, ZAP_BASE_URL=http://bodgeit:8080/bodgeit/}}, Target{name='null', location='http://bodgeit:8080/bodgeit/home.jsp', attributes={request={method=GET, url=http://bodgeit:8080/bodgeit/home.jsp, httpVersion=HTTP/1.1, cookies=[{name=JSESSIONID, value=6FD5717DB5D22B76C9A17F55BCBBD698}], headers=[{name=User-Agent, value=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0}, {name=Pragma, value=no-cache}, {name=Cache-Control, value=no-cache}, {name=Content-Length, value=0}, {name=Referer, value=http://bodgeit:8080/bodgeit/}, {name=Host, value=bodgeit:8080}, {name=Cookie, value=JSESSIONID=6FD5717DB5D22B76C9A17F55BCBBD698}], queryString=[], postData={mimeType=, params=[], text=}, headersSize=313, bodySize=0}, ZAP_BASE_URL=http://bodgeit:8080/bodgeit/}}, Target{name='null', location='http://bodgeit:8080/bodgeit/.svn/entries', attributes={request={method=GET, url=http://bodgeit:8080/bodgeit/.svn/entries, httpVersion=HTTP/1.1, cookies=[], headers=[{name=User-Agent, value=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0}, {name=Pragma, value=no-cache}, {name=Cache-Control, value=no-cache}, {name=Content-Length, value=0}, {name=Host, value=bodgeit:8080}], queryString=[], postData={mimeType=, params=[], text=}, headersSize=225, bodySize=0}, ZAP_BASE_URL=http://bodgeit:8080/bodgeit/}}, Target{name='null', location='http://bodgeit:8080/bodgeit/style.css', attributes={request={method=GET, url=http://bodgeit:8080/bodgeit/style.css, httpVersion=HTTP/1.1, cookies=[{name=JSESSIONID, value=6FD5717DB5D22B76C9A17F55BCBBD698}, {name=b_id, value=2}], headers=[{name=User-Agent, value=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0}, {name=Pragma, value=no-cache}, {name=Cache-Control, value=no-cache}, {name=Content-Length, value=0}, {name=Referer, value=http://bodgeit:8080/bodgeit/}, {name=Host, value=bodgeit:8080}, {name=Cookie, value=b_id=2; JSESSIONID=6FD5717DB5D22B76C9A17F55BCBBD698}], queryString=[], postData={mimeType=, params=[], text=}, headersSize=322, bodySize=0}, ZAP_BASE_URL=http://bodgeit:8080/bodgeit/}}, Target{name='null', location='http://bodgeit:8080/bodgeit/login.jsp', attributes={request={method=GET, url=http://bodgeit:8080/bodgeit/login.jsp, httpVersion=HTTP/1.1, cookies=[{name=JSESSIONID, value=6FD5717DB5D22B76C9A17F55BCBBD698}], headers=[{name=User-Agent, value=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0}, {name=Pragma, value=no-cache}, {name=Cache-Control, value=no-cache}, {name=Content-Length, value=0}, {name=Referer, value=http://bodgeit:8080/bodgeit/}, {name=Host, value=bodgeit:8080}, {name=Cookie, value=JSESSIONID=6FD5717DB5D22B76C9A17F55BCBBD698}], queryString=[], postData={mimeType=, params=[], text=}, headersSize=314, bodySize=0}, ZAP_BASE_URL=http://bodgeit:8080/bodgeit/}}, Target{name='null', location='http://bodgeit:8080/bodgeit/admin.jsp', attributes={request={method=GET, url=http://bodgeit:8080/bodgeit/admin.jsp, httpVersion=HTTP/1.1, cookies=[{name=JSESSIONID, value=6FD5717DB5D22B76C9A17F55BCBBD698}, {name=b_id, value=2}], headers=[{name=User-Agent, value=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0}, {name=Pragma, value=no-cache}, {name=Cache-Control, value=no-cache}, {name=Content-Length, value=0}, {name=Referer, value=http://bodgeit:8080/bodgeit/}, {name=Host, value=bodgeit:8080}, {name=Cookie, value=b_id=2; JSESSIONID=6FD5717DB5D22B76C9A17F55BCBBD698}], queryString=[], postData={mimeType=, params=[], text=}, headersSize=322, bodySize=0}, ZAP_BASE_URL=http://bodgeit:8080/bodgeit/}}, Target{name='null', location='http://bodgeit:8080/bodgeit/contact.jsp', attributes={request={method=GET, url=http://bodgeit:8080/bodgeit/contact.jsp, httpVersion=HTTP/1.1, cookies=[{name=JSESSIONID, value=6FD5717DB5D22B76C9A17F55BCBBD698}], headers=[{name=User-Agent, value=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0}, {name=Pragma, value=no-cache}, {name=Cache-Control, value=no-cache}, {name=Content-Length, value=0}, {name=Referer, value=http://bodgeit:8080/bodgeit/}, {name=Host, value=bodgeit:8080}, {name=Cookie, value=JSESSIONID=6FD5717DB5D22B76C9A17F55BCBBD698}], queryString=[], postData={mimeType=, params=[], text=}, headersSize=316, bodySize=0}, ZAP_BASE_URL=http://bodgeit:8080/bodgeit/}}, Target{name='null', location='http://bodgeit:8080/bodgeit/.svn/wc.db', attributes={request={method=GET, url=http://bodgeit:8080/bodgeit/.svn/wc.db, httpVersion=HTTP/1.1, cookies=[], headers=[{name=User-Agent, value=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0}, {name=Pragma, value=no-cache}, {name=Cache-Control, value=no-cache}, {name=Content-Length, value=0}, {name=Host, value=bodgeit:8080}], queryString=[], postData={mimeType=, params=[], text=}, headersSize=223, bodySize=0}, ZAP_BASE_URL=http://bodgeit:8080/bodgeit/}}, Target{name='null', location='http://bodgeit:8080/bodgeit/product.jsp?prodid=2', attributes={request={method=GET, url=http://bodgeit:8080/bodgeit/product.jsp?prodid=2, httpVersion=HTTP/1.1, cookies=[{name=JSESSIONID, value=6FD5717DB5D22B76C9A17F55BCBBD698}, {name=b_id, value=2}], headers=[{name=User-Agent, value=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0}, {name=Pragma, value=no-cache}, {name=Cache-Control, value=no-cache}, {name=Content-Length, value=0}, {name=Referer, value=http://bodgeit:8080/bodgeit/}, {name=Host, value=bodgeit:8080}, {name=Cookie, value=b_id=2; JSESSIONID=6FD5717DB5D22B76C9A17F55BCBBD698}], queryString=[{name=prodid, value=2}], postData={mimeType=, params=[], text=}, headersSize=333, bodySize=0}, ZAP_BASE_URL=http://bodgeit:8080/bodgeit/}}, Target{name='null', location='http://bodgeit:8080/bodgeit/.git/index', attributes={request={method=GET, url=http://bodgeit:8080/bodgeit/.git/index, httpVersion=HTTP/1.1, cookies=[], headers=[{name=User-Agent, value=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0}, {name=Pragma, value=no-cache}, {name=Cache-Control, value=no-cache}, {name=Content-Length, value=0}, {name=Host, value=bodgeit:8080}], queryString=[], postData={mimeType=, params=[], text=}, headersSize=223, bodySize=0}, ZAP_BASE_URL=http://bodgeit:8080/bodgeit/}}, Target{name='null', location='http://bodgeit:8080/bodgeit/', attributes={request={method=GET, url=http://bodgeit:8080/bodgeit/, httpVersion=HTTP/1.1, cookies=[], headers=[{name=User-Agent, value=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0}, {name=Pragma, value=no-cache}, {name=Cache-Control, value=no-cache}, {name=Content-Length, value=0}, {name=Host, value=bodgeit:8080}], queryString=[], postData={mimeType=, params=[], text=}, headersSize=213, bodySize=0}, ZAP_BASE_URL=http://bodgeit:8080/bodgeit/}}, Target{name='null', location='http://bodgeit:8080/bodgeit/about.jsp', attributes={request={method=GET, url=http://bodgeit:8080/bodgeit/about.jsp, httpVersion=HTTP/1.1, cookies=[{name=JSESSIONID, value=6FD5717DB5D22B76C9A17F55BCBBD698}], headers=[{name=User-Agent, value=Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0}, {name=Pragma, value=no-cache}, {name=Cache-Control, value=no-cache}, {name=Content-Length, value=0}, {name=Referer, value=http://bodgeit:8080/bodgeit/}, {name=Host, value=bodgeit:8080}, {name=Cookie, value=JSESSIONID=6FD5717DB5D22B76C9A17F55BCBBD698}], queryString=[], postData={mimeType=, params=[], text=}, headersSize=314, bodySize=0}, ZAP_BASE_URL=http://bodgeit:8080/bodgeit/}}]

I updated the title, as it stays in scanner, saying the task spider is complete.

Cheers, Timo

wurstbrot commented 5 years ago

I guess due to the error message, that the targetUrl is not in the siteTree and the source of the error is here: https://github.com/igorhvr/zaproxy/blob/master/src/org/zaproxy/zap/extension/ascan/ActiveScanAPI.java#L98

J12934 commented 5 years ago

Hi Timo Can you please double check if your engine version really is from the current develop branch. The log from the engine looks to me as it is not. We have recently changed the format zap spider findings get transformed into and this appears to still be the old format.

wurstbrot commented 5 years ago

First test: With the current development version and a local build it works. More tests tomorrow.

wurstbrot commented 5 years ago

Using latest securecodebox/engine:develop and securecodebox/zap:develop (using docker-compose pull) results in an error on the engine.

Used versions: securecodebox/engine develop 0af6e826d7c8 2 days ago 172MB securecodebox/zap develop 019df2720f2d 9 days ago 559MB

Logs:

2018-11-22 07:49:31.948 ERROR 9 --- [nio-8080-exec-8] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception

org.springframework.security.web.firewall.RequestRejectedException: The request was rejected because the URL was not normalized.
    at org.springframework.security.web.firewall.StrictHttpFirewall.getFirewalledRequest(StrictHttpFirewall.java:248) ~[spring-security-web-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:193) ~[spring-security-web-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) ~[spring-security-web-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
    at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
    at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106) ~[spring-boot-actuator-1.5.13.RELEASE.jar!/:1.5.13.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1468) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]

With the self build of the current master of github it works. By using develop branch of engine, it is not working. Maybe I am forced to use the new api (which is hard without documentation)?

wurstbrot commented 5 years ago

The last problem doesn't has to be a problem with zap. The engine shows that error continuously and zap does it's job. The test happend with the images from this morning (engine+zap from develop): $ docker images | grep securecodebox securecodebox/engine develop 5191b1e2f556 20 hours ago 172MB securecodebox/arachni develop c2c858d4fe64 20 hours ago 1.88GB securecodebox/engine 0af6e826d7c8 3 days ago 172MB securecodebox/engine f4547a285c98 7 days ago 165MB securecodebox/zap develop 019df2720f2d 10 days ago 559MB securecodebox/arachni eed1c249881c 10 days ago 1.88GB securecodebox/amass oss c42e42ec9c17 2 months ago 31.5MB securecodebox/nikto v0.9.0 22f0967523fa 2 months ago 285MB securecodebox/arachni v0.9.0 80649130b684 2 months ago 1.88GB securecodebox/sslyze v0.9.0 0794f118d43c 2 months ago 1.05GB securecodebox/nmap v0.9.0 6799a25e1cca 2 months ago 91.5MB securecodebox/bodgeit latest d7eb6d6890bd 9 months ago 127MB

J12934 commented 5 years ago

Using latest securecodebox/engine:develop and securecodebox/zap:develop (using docker-compose pull) results in an error on the engine.

Used versions: securecodebox/engine develop 0af6e826d7c8 2 days ago 172MB securecodebox/zap develop 019df2720f2d 9 days ago 559MB

Logs:

2018-11-22 07:49:31.948 ERROR 9 --- [nio-8080-exec-8] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception

org.springframework.security.web.firewall.RequestRejectedException: The request was rejected because the URL was not normalized.
  at org.springframework.security.web.firewall.StrictHttpFirewall.getFirewalledRequest(StrictHttpFirewall.java:248) ~[spring-security-web-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
  at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:193) ~[spring-security-web-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
  at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) ~[spring-security-web-4.2.6.RELEASE.jar!/:4.2.6.RELEASE]
  at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
  at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
  at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
  at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
  at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
  at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
  at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106) ~[spring-boot-actuator-1.5.13.RELEASE.jar!/:1.5.13.RELEASE]
  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.3.17.RELEASE.jar!/:4.3.17.RELEASE]
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) ~[tomcat-embed-core-8.5.31.jar!/:8.5.31]
  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
  at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
  at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
  at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
  at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
  at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1468) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
  at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181]
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]
  at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.31.jar!/:8.5.31]
  at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]

With the self build of the current master of github it works. By using develop branch of engine, it is not working. Maybe I am forced to use the new api (which is hard without documentation)?

Does this error crash the container?

Also the current develop state is already relatively well documented, the only thing missing is the generated markdown rest api documentation. But you can always access the dynamic swagger page of the engine. (located at /swagger-ui.html).

wurstbrot commented 5 years ago

Does this error crash the container?

no