search

jfrog / frogbot

🐸 Scans your Git repository with JFrog Xray for security vulnerabilities. 🤖
https://docs.jfrog-applications.jfrog.io/
Apache License 2.0
290 stars 61 forks source link

Problems executing frogbot in Enterprise env without internet connection #687

Closed Mathias83 closed 2 months ago

Mathias83 commented 2 months ago

Hi, I can't run frogbot. I suspect there are problems because I can't access the internet from our network. Can someone help me

Output of „frogbot scan-repository“ 11:38:32 [🔵Info] Frogbot version: 2.20.1 11:38:32 [Debug] Reading config from file system. Looking for .frogbot/frogbot-config.yml 11:38:32 [Debug] frogbot-config.yml wasn't found in /.frogbot/frogbot-config.yml. Searching for it in upstream directories 11:38:32 [Debug] Attempting to download frogbot-config.yml from ART/testRepo 11:38:32 [Debug] The frogbot-config.yml will be downloaded from master branch 11:38:32 [Debug] The .frogbot/frogbot-config.yml file wasn't recognized in <ART/testRepo> 11:38:32 [Debug] Sending HTTP HEAD request to: 'https://github.com/jfrog/frogbot' 11:38:32 [Debug] Locking config file to run config AddOrEdit command. 11:38:32 [Debug] Creating lock in: /tmp/jfrog.cli.temp.-1713699512-182712745/locks/config 11:38:32 [Debug] Releasing lock: /tmp/jfrog.cli.temp.-1713699512-182712745/locks/config/jfrog-cli.conf.lck.216.1713699512633572200 11:38:32 [Debug] Config AddOrEdit command completed successfully. config file is released. 11:38:32 [Debug] Usage Report: Sending info... 11:38:32 [🔵Info] Running Frogbot "scan-repository" command 11:38:32 [Debug] Sending HTTP POST request to: https://usage-ecosystem.jfrog.io/api/usage/report 11:38:32 [Debug] Sending HTTP GET request to: https://artifactory/artifactory/api/system/version 11:38:32 [Debug] Sending HTTP GET request to: https://artifactory/xray/api/v1/system/version 11:38:32 [Debug] Artifactory response: 200 11:38:32 [Debug] JFrog Artifactory version is: 7.71.11 11:38:32 [Debug] Sending HTTP POST request to: https://artifactory/artifactory/api/system/usage 11:38:33 [Debug] JFrog Xray version is: 3.90.1 11:38:33 [Debug] Sending HTTP POST request to: https://artifactory/xray/api/v1/usage/events/send 11:39:02 [Debug] Can't check access to 'https://github.com/jfrog/frogbot', error while sending request: Head "https://github.com/jfrog/frogbot": dial tcp 140.82.121.3:443: i/o timeout 11:39:02 [Debug] Sending an error report to JFrog analytics... 11:39:02 [Debug] Sending HTTP GET request to: https://artifactory/xsc/api/v1/system/version 11:39:02 [Debug] failed to check availability of Xsc service:server response: 404 Not Found 404 page not found

eranturgeman commented 2 months ago

Hello @Mathias83, thank you for utilizing Frogbot and bringing this issue to our attention! While encountering the 404 error, it is probably not the cause for the failure. This error pertains to an optional reporting service we recently introduced, which relies on the availability of the Xsc service. However, its absence should not impact the overall action. Additionally, I noticed a timeout occurred during your execution. To address this, I'd like to investigate further to determine the root cause of the failure. Could you kindly provide the complete log from your execution? Additionally, I'm curious whether you attempted to rerun the action or if it was executed only once.

With this additional information, I'll be able to conduct a thorough examination of the issue and either provide you with an update or request further details if needed.

Mathias83 commented 2 months ago

Hi @eranturgeman,

Frogbot runs in a Docker container with the corresponding properties as shown below.

` JF_RELEASES_REPO=jfrog-release-ext JF_PASSWORD=password JF_USER=username JF_XRAY_URL= JF_GIT_OWNER=ART JF_GIT_REPO=testRepo JF_GIT_PROVIDER=bitbucketServer JF_URL=https://artifactory JF_GIT_TOKEN=token JFROG_CLI_LOG_LEVEL=DEBUG JF_GIT_BASE_BRANCH=master JF_GIT_API_ENDPOINT=https://coderepo

`

I have already shared the complete edition of Frogbot except for this passage that comes at the end

Reporting to JFrog analytics is skipped... 11:39:02 [🚨Error] Status: 404 , Body: <!doctype html>HTTP Status 404 – Not Found

HTTP Status 404 – Not Found

Type Status Report

Description The origin server did not find a current representation for the target resource or is not willing to disclose that one exists.

ParseError: invalid character '<' looking for beginning of value

eranturgeman commented 2 months ago

I need the complete log where it indicates the exit status The info you provided about the container cannot help me since it doesn't contain the actual values I assume. Also I cannot understand where is this body in your message came from: <!doctype html>HTTP Status 404 – Not Found

Can you please provide the full log in DEBUG mode, as Frogbot prints it to you?

Mathias83 commented 2 months ago

Hallo @eranturgeman,

that's all there is. I run Frogbot and at the end of the HTML output is finished


bash-5.1$ frogbot scan-repository
13:41:33 [🔵Info] Frogbot version: 2.20.1
13:41:33 [Debug] Reading config from file system. Looking for .frogbot/frogbot-config.yml
13:41:33 [Debug] frogbot-config.yml wasn't found in /.frogbot/frogbot-config.yml. Searching for it in upstream directories
13:41:33 [Debug] Attempting to download frogbot-config.yml from ART/testrepo
13:41:33 [Debug] The frogbot-config.yml will be downloaded from master branch
13:41:33 [Debug] The .frogbot/frogbot-config.yml file wasn't recognized in <ART/testrepo>
13:41:33 [Debug] Sending HTTP HEAD request to: 'https://github.com/jfrog/frogbot'
13:41:33 [Debug] Locking config file to run config AddOrEdit command.
13:41:33 [Debug] Creating lock in: /tmp/jfrog.cli.temp.-1713706893-4068491470/locks/config
13:41:33 [Debug] Releasing lock: /tmp/jfrog.cli.temp.-1713706893-4068491470/locks/config/jfrog-cli.conf.lck.225.1713706893644275100       
13:41:33 [Debug] Config AddOrEdit command completed successfully. config file is released.
13:41:33 [Debug] Usage Report: Sending info...
13:41:33 [🔵Info] Running Frogbot "scan-repository" command
13:41:33 [Debug] Sending HTTP GET request to: https://artifactory/artifactory/api/system/version
13:41:33 [Debug] Sending HTTP GET request to: https://artifactory/xray/api/v1/system/version
13:41:33 [Debug] Sending HTTP POST request to: https://usage-ecosystem.jfrog.io/api/usage/report
13:41:34 [Debug] JFrog Xray version is: 3.90.1
13:41:34 [Debug] Sending HTTP POST request to: https://artifactory/xray/api/v1/usage/events/send
13:41:34 [Debug] Artifactory response: 200 
13:41:34 [Debug] JFrog Artifactory version is: 7.71.11
13:41:34 [Debug] Sending HTTP POST request to: https://artifactory/artifactory/api/system/usage
13:42:03 [Debug] Can't check access to 'https://github.com/jfrog/frogbot', error while sending request:
Head "https://github.com/jfrog/frogbot": dial tcp 140.82.121.3:443: i/o timeout
13:42:03 [Debug] Sending an error report to JFrog analytics...
13:42:03 [Debug] Sending HTTP GET request to: https://artifactory/xsc/api/v1/system/version
13:42:03 [Debug] failed to check availability of Xsc service:server response: 404 Not Found
404 page not found

Reporting to JFrog analytics is skipped...
13:42:03 [🚨Error] Status: 404 , Body: <!doctype html><html lang="en"><head><title>HTTP Status 404 – Not Found</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 404 – Not Found</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Description</b> The origin server did not find a current representation for the target resource or is not willing to disclose that one exists.</p><hr class="line" /></body></html> ParseError: invalid character '<' looking for beginning of value
bash-5.1$
Mathias83 commented 2 months ago

I played around a bit and adjusted the URL of the Git repository. Now the message is different although I specify the user and the token. The audit log of the server also shows the user but apparently without a token? Apparently also with basic Auth instead of bearer token. Is that how it should be?

19:47:09 [Debug] Sending an error report to JFrog analytics...
19:47:09 [Debug] Sending HTTP GET request to: https://artifactory/xsc/api/v1/system/version
19:47:09 [Debug] failed to check availability of Xsc service:server response: 404 Not Found
404 page not found

Reporting to JFrog analytics is skipped...
19:47:09 [🚨Error] git clone frogbot from https://***.git failed with error: authentication required
eranturgeman commented 2 months ago

@Mathias83,

Could you kindly share the complete CI file you used? Feel free to redact any sensitive data. I also have a few more questions: 1) From where is Frogbot being downloaded? Did you utilize the JF_RELEASES_REPO environment variable? 2) Could you please clarify your last question? I didn't fully grasp what you were asking and what specific changes you made. 3) Can you provide the original and updated values of the URL you modified? Additionally, what was the last known value with which Frogbot ran successfully? 4) What value did you assign to JF_GIT_API_ENDPOINT?

Please disregard any error logs you encounter in Debug mode. If they were critical, they wouldn't appear in Debug mode. You may continue to see 404 error messages from Xsc, as I assume it's either not deployed or inaccessible due to lack of internet access.

Regarding your authentication question, you have two options: Using JF_USER and JF_PASSWORD Using JF_ACCESS_TOKEN (in which case, a username is not required)

Mathias83 commented 2 months ago

Hi thanks i have found a solution by myself.