Velocidex / velociraptor

Digging Deeper....
https://docs.velociraptor.app/
Other
2.98k stars 491 forks source link

Cannot download artifact exchange to server #2389

Closed sgtrivas closed 1 year ago

sgtrivas commented 1 year ago

HELP! I am trying my best to learn how to better use this tool but I keep running into errors that I find hard to trouble shoot. I tried to run the Server.Import.ArtifactExchange and I received this error in the log.

2023-01-20T15:40:03Z DEFAULT Running query on behalf of user vagrant
2023-01-20T15:40:03Z DEFAULT Starting</> query execution.

2023-01-20T15:40:03Z | ERROR | Plugin http_client not found. Current Scope is: [NULL], [$cache, $device_manager, config, $acl, $uploader, Artifact, $root], [ExchangeURL, Prefix], [X], [Server_Import_ArtifactExchange_0_1], [$Query]

2023-01-20T15:40:03Z | ERROR | Plugin http_client not found. Did you mean client_delete clients?

2023-01-20T15:40:03Z | DEFAULT | Query Server.Import.ArtifactExchange: Emitted 0 rows

2023-01-20T15:40:03Z | DEBUG | Query Stats: {"RowsScanned":0,"PluginsCalled":1,"FunctionsCalled":0,"ProtocolSearch":0,"ScopeCopy":3}

scudette commented 1 year ago

It is likely that you selected the option Do you want to restrict VQL functionality on the server? during the config generation step. That options creates an allow list with only safe plugins allowed to run on the server.

The http client plugin is used to fetch the bundle from the exchange but it is not allowed in the default allow list. This is why the plugin is not found because it is disabled.

you can add it to the allow list temporarily or while learning the tool disable the allow list entirely

sgtrivas commented 1 year ago

that solved that problem! DEFAULT | http_client: Downloading https://github.com/Velocidex/velociraptor-docs/raw/gh-pages/exchange/artifact_exchange.zip into /tmp/tmp1426181718.zip

DEFAULT | Query Server.Import.ArtifactExchange: Emitted 174 rows THAT PROBLEM IS SOLVED..... HOWEVER Next question: now when I run an artifact like Windows.sysinternal.autoruns on my target system I get this error: http_client: Error Get "https://localhost:8000/public/30990d6c7411ef568f05ddecd1a7fe39e1ac3ec0209b32b1e5ba987e120355ca": dial tcp 127.0.0.1:8000: connectex: No connection could be made because the target machine actively refused it. while fetching https://localhost:8000/public/30990d6c7411ef568f05ddecd1a7fe39e1ac3ec0209b32b1e5ba987e120355ca

Is there something in the client config that is causing this? or is it with the windows host that the client is installed?

scudette commented 1 year ago

The Autoruns artifact needs to download the autoruns tool from the server. When you built your configuration in the wizard there is a question like What is the public DNS name of the Master Frontend. This is the publicly accessible host name where clients download the tools from

The default is localhost but this obviously only works in the client and server are on the same machine. You need to have a publicly routable DNS name here instead of localhost.

If you use self signed mode it is possible to put an ip address here.

sgtrivas commented 1 year ago

Server config: version: name: velociraptor version: 0.6.7-4 commit: c6f11a7 build_time: "2022-12-06T13:31:56Z" ci_build_url: https://github.com/Velocidex/velociraptor/actions/runs/3629952817 compiler: go1.19.3 Client: server_urls:

=============================================== version: name: velociraptor version: 0.6.7-4 commit: c6f11a7 build_time: "2022-12-06T13:31:56Z" ci_build_url: https://github.com/Velocidex/velociraptor/actions/runs/3629952817 compiler: go1.19.3 Client: server_urls:

what do i need to change?

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     # End
sgtrivas commented 1 year ago

P.S. if i change the "server_urls: https://192.168.38.105:8000/" to anything but the IP to the server, the client will not connect. If i add for example 8.8.8.8:" it changes nothing. i still get the "https://localhost..." error from before....

scudette commented 1 year ago

If you updated the server urls it might be that the old value is stored in the inventory. simply pop up the tool setup screen and redownload the tool and watch the serve_url parameter. It should take the value from the Client.server_urls in the server's config value and it should be an externally accessible URL

sgtrivas commented 1 year ago

I'm sorry for not getting back to you sooner, but I found something interesting. I ran the "config generate" and went thru the config process. I noticed a few things:

  1. when I added the DNS IP, i.e. 8.8.8.8, it used that IP as the client-server URL, which caused the client to not connect to the server.
  2. when I re-ran the config generator, I used the Server IP, and client instantly connected to the server but I was still getting the earlier error.
  3. Some artifacts can upload the binary, i.e., autoruns, and override the upstream URL, but I was still getting the "http_client" error as before.
  4. This changed, I changed the serve locally URL to the Servers IP, and I was able to run the artifacts that required communication with the server exchange, autoruns.exe (tho autoruns seems to timeout, probably need to extend the time) and winPMEM (which didn't return anything?)
  5. Is it an issue that the server & client are deployed in a self-contained Virtualbox lab? the client.server.url is a private IP, and it is an Ubuntu server on the same Virtbox virtual NIC. I do have another NIC that is nat to the external internet that I enable when I need to download something. if the client.server.url isn't my velociraptor's server; how does it connect?

For context:

I am an intern with a university and was tasked with building a malware analysis and incident response lab for a team here. I used an amalgamation of Detection Lab and Detection Lab ELK as the lab since it uses Vagrant to build and destroy the images quickly. Velociraptor is installed as the server (Ubuntu) and client (Win10-Victim) for the team to use to perform a hunt on malware that they want to test and write reports on. I wanted to ensure that they entirely use this tool because I think it is one of the best I have come across, and it's open source, so it can remain out of the budget.

scudette commented 1 year ago

Using external tools with Velociraptor is documented here https://docs.velociraptor.app/docs/extending_vql/#using-external-tools

The way it works is basically - the server sends the client a URL and a hash. The client downloads from that URL and checks the hash to ensure the right tool is downloaded, then the client just runs it.

Normally it is easiest to just serve the tool from Velociraptor itself but this is not mandatory - sometimes it is better to serve it from another URL (e.g. better bandwidth etc). This can be configured in the tool setup screen.

It doesnt really matter that client and server are running on VMs - You will need to ensure there is connectivity between the client and whatever is hosting the tool.

image