elasticsearch: port 9200 connectivity issue

[aesophor]

Dear TAs,

Here are what work for me so far

• Ubuntu - ELK is up and running (I can log into kibana with admin/admin)
• Windows 10 - Installed winlogbeat and successfully started the service.

The problem

No logs from winlogbeat were sent to elasticsearch (?)

What I've tried

I've read the official documentation of elastic and, if I'm not mistaken, winlogbeat should send the logs via port 9200.

The official doc suggests that the user should run the following command to verify if the nodes are running:

$ curl -X GET "localhost:9200/_cat/nodes?v&pretty"

However, I received this output:

$ curl -X GET "localhost:9200/_cat/nodes?v&pretty"
curl: (52) Empty reply from server

Visiting this URL via a browser gives:

My winlogbeat.yml

Using the following config, I can start the winlogbeat service successfully.

# =================================== Kibana ===================================
setup.kibana:
  # Kibana Host
  host: "192.168.131.137:5601"

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["192.168.131.137:9200"]

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "admin"
  password: "admin"

The problem is that there isn't any log from winlogbeat showing up in elasticsearch. Did I miss something? Thanks in advance! :pleading_face: :pleading_face:

[dsnslab]

The output should be logstash

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["<IP>:<port>"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

[joydec1118]

but still i do not see any logs in my kibana. is something wrong in my network setting?

[aesophor]

@joydec1118

Same here. From the Ubuntu terminal, I can see a bunch of logs coming in, but they still won't show up in kibana.

[dsnslab]

You can google setup ELK and you will find lots of documents

Hint: Just follow the instruction as @joydec1118 show in the picture.

[nelson870708]

@aesophor

In my opinion, although, Logstash collects the logs, according to the logs, it does not transform it to Elasticsearch. That's why the data does not appear on Kibana. However, I am also stuck in the same problem. Do you have any idea?

[HC-TW]

@aesophor Same problem here.

[aesophor]

@nelson870708 @HC-TW @joydec1118

I've just figured this out. If this shouldn't be posted, please tell me and I'll delete the comment immediately.

[Spoiler Alert]

Running .\winlogbeat setup --dashboards will create a index pattern "winlogbeat-*" for you. However, since your logs are stored in the logstash, you need to manually create another index pattern "logstash*". This way you'll be able to see the logs via kibana.

Note that some events such as "running an .exe" are not logged by default. You'll need to enable them manually by editing group policies via gpedit.msc. See: https://superuser.com/questions/603226/does-windows-log-programs-that-have-been-run-called

For the remaining parts I'm still exploring... QQ Edit1: Still looking for a way to change "fields.hostname" to my student ID... 😕 (it shows "unknown" by default) Edit2: Found it. You need to edit winlogbeat.yml. (See the general section) If for some unknown reasons it doesn't take effect, just reboot this entire stuff (both ELK and Windows 10)

Microsoft Windows... What a nightmare

[dsnslab]

@aesophor It's OK. You are encouraged to share some information about setting ELK stack. But don't show the answer directly.

[nelson870708]

@aesophor

Thanks for your help. Now, I see the logs.

[aesophor]

@nelson870708 That's great! :smile:

[chiha8888]

@nelson870708 @HC-TW @joydec1118

I've just figured this out. If this shouldn't be posted, please tell me and I'll delete the comment immediately.

[Spoiler Alert]

Running .\winlogbeat setup --dashboards will create a index pattern "winlogbeat-" for you. However, since your logs are stored in the logstash, you need to manually create another index pattern "logstash". This way you'll be able to see the logs via kibana.

Note that some events such as "running an .exe" are not logged by default. You'll need to enable them manually by editing group policies via gpedit.msc. See: https://superuser.com/questions/603226/does-windows-log-programs-that-have-been-run-called

For the remaining parts I'm still exploring... QQ Edit1: Still looking for a way to change "fields.hostname" to my student ID... (it shows "unknown" by default) Edit2: Found it. You need to edit winlogbeat.yml. (See the general section) If for some unknown reasons it doesn't take effect, just reboot this entire stuff (both ELK and Windows 10)

Microsoft Windows... What a nightmare

How to \" manually create another index pattern "logstash*". \" ?

I can now get the winlogbeat data in ubuntu by checking http://localhost:9200/_cat/indices?v, but does not show in Kibana. (elasticsearch(port:9200) and kibana(port:5601) are all FINE)

This is my Kibana Dashboard apperence:

[aesophor]

@chiha8888 Maybe this will help: https://www.elastic.co/guide/en/kibana/current/index-patterns.html#settings-create-pattern

[nelson870708]

@chiha8888

In addition Which account do you login the Kibana? You should login with username: admin password: admin not the one that username is "kibanaserver"