deviantony / docker-elk

The Elastic stack (ELK) powered by Docker and Compose.
MIT License
17.08k stars 6.74k forks source link

I can't view my Windows logs,abouts CSP security,NODE_ENV === "production" #940

Closed HaSaKiYasuooo closed 9 months ago

HaSaKiYasuooo commented 10 months ago

kibana.yml

server.name: kibana
server.host: 0.0.0.0
elasticsearch.hosts: [ http://elasticsearch:9200 ]

monitoring.ui.container.elasticsearch.enabled: true
monitoring.ui.container.logstash.enabled: true
csp.strict: false
server.securityResponseHeaders.disableEmbedding: true
xpack.security.encryptionKey: "fgjh2D8Gjk3dsFgjHs2fJGH5Dkfyr4Hj"
xpack.encryptedSavedObjects.encryptionKey: "zxcPASdf31fdszMPo9lkjOPqw12FsweQ"
xpack.reporting.encryptionKey: "kLmnoPQrst3UVWXnpqRsTUvWxy123456"
server.publicBaseUrl: "https://kibana.just4you.eu.org/"
elasticsearch.username: kibana_system
elasticsearch.password: ${KIBANA_SYSTEM_PASSWORD}

elastic-agenmt.yml

revision: 2
outputs:
  default:
    type: elasticsearch
    hosts:
      - 'https://elk.just4you.eu.org/'
    username: 'elastic'
    password: 'changeme'

docker-compose logs which contains error and waring infos

# kibana logs
[2023-11-22T15:26:33.583+00:00][WARN ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, but is not supported for Linux Ubuntu 20.04 OS. Automatically setting 'xpack.screenshotting.browser.chromium.disableSandbox: true'.
[2023-11-22T15:26:33.735+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 172.22.0.2:9200
[2023-11-22T15:26:46.192+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception
        Root causes:
                security_exception: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]
# logstash logs
[2023-11-22T15:26:31,627][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2023-11-22T15:26:32,238][INFO ][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>"Connect to elasticsearch:9200 [elasticsearch/172.22.0.2] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to elasticsearch:9200 [elasticsearch/172.22.0.2] failed: Connection refused>}
[2023-11-22T15:26:32,242][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://logstash_internal:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/172.22.0.2] failed: Connection refused"}

[2023-11-22T15:26:48,181][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}

[2023-11-22T15:26:42,282][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://logstash_internal:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/172.22.0.2] failed: Connection refused"}
# elasticserach logs
WARNING: COMPAT locale provider will be removed in a future release
{"@timestamp":"2023-11-22T15:25:59.333Z", "log.level": "INFO", "message":"Java vector incubator API enabled; uses preferredBitSize=256", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.apache.lucene.internal.vectorization.PanamaVectorizationProvider","elasticsearch.node.name":"elasticsearch","elasticsearch.cluster.name":"docker-cluster"}

error page image image nginx-proxy-manager config image windows dashboard image docker and compose version image

antoineco commented 10 months ago

Could you please share the output of docker compose up setup?

HaSaKiYasuooo commented 9 months ago

Could you please share the output of docker compose up setup?

the "docker compose up setup" is by the default,and elasticsearch,logstash
ubuntu@ip-172-31-95-197:~/docker-elk$ docker compose up setup
[+] Building 0.8s (5/5) FINISHED                                                                                                               docker:default
 => [setup internal] load build definition from Dockerfile                                                                                               0.0s
 => => transferring dockerfile: 194B                                                                                                                     0.0s
 => [setup internal] load .dockerignore                                                                                                                  0.0s
 => => transferring context: 160B                                                                                                                        0.0s
 => [setup internal] load metadata for docker.elastic.co/elasticsearch/elasticsearch:8.11.1                                                              0.6s
 => CACHED [setup 1/1] FROM docker.elastic.co/elasticsearch/elasticsearch:8.11.1@sha256:cf3edd6518b0159d50c0f932f6cacd63930db01e1fb740499eca477543d42b3  0.0s
 => [setup] exporting to image                                                                                                                           0.0s
 => => exporting layers                                                                                                                                  0.0s
 => => writing image sha256:230b56ba34cbfeae8232eceb13df796d537157c438aaf24e12fbdc473f06ba69                                                             0.0s
 => => naming to docker.io/library/docker-elk-setup                                                                                                      0.0s
[+] Running 2/2
 ✔ Container docker-elk-elasticsearch-1  Running                                                                                                         0.0s
 ✔ Container docker-elk-setup-1          Created                                                                                                         0.1s
Attaching to docker-elk-setup-1
docker-elk-setup-1  | [+] Waiting for availability of Elasticsearch. This can take several minutes.
docker-elk-setup-1  |    ⠿ Elasticsearch is running
docker-elk-setup-1  | [+] Waiting for initialization of built-in users
docker-elk-setup-1  |    ⠿ Built-in users were initialized
docker-elk-setup-1  | [+] Role 'heartbeat_writer'
docker-elk-setup-1  |    ⠿ Creating/updating
docker-elk-setup-1  | [+] Role 'metricbeat_writer'
docker-elk-setup-1  |    ⠿ Creating/updating
docker-elk-setup-1  | [+] Role 'filebeat_writer'
docker-elk-setup-1  |    ⠿ Creating/updating
docker-elk-setup-1  | [+] Role 'logstash_writer'
docker-elk-setup-1  |    ⠿ Creating/updating
docker-elk-setup-1  | [+] User 'filebeat_internal'
docker-elk-setup-1  |    ⠿ No password defined, skipping
docker-elk-setup-1  | [+] User 'kibana_system'
docker-elk-setup-1  |    ⠿ User exists, setting password
docker-elk-setup-1  | [+] User 'logstash_internal'
docker-elk-setup-1  |    ⠿ User exists, setting password
docker-elk-setup-1  | [+] User 'heartbeat_internal'
docker-elk-setup-1  |    ⠿ No password defined, skipping
docker-elk-setup-1  | [+] User 'metricbeat_internal'
docker-elk-setup-1  |    ⠿ No password defined, skipping
docker-elk-setup-1  | [+] User 'monitoring_internal'
docker-elk-setup-1  |    ⠿ No password defined, skipping
docker-elk-setup-1  | [+] User 'beats_system'
docker-elk-setup-1  |    ⠿ No password defined, skipping
docker-elk-setup-1 exited with code 0
-------------
CONTAINER ID   IMAGE                             COMMAND                  CREATED          STATUS                      PORTS                                                                                                                                                                                NAMES
3f37ab5e35ec   docker-elk-setup                  "/entrypoint.sh"         50 seconds ago   Exited (0) 48 seconds ago                                                                                                                                                                                        docker-elk-setup-1
793cf69d4f4f   docker-elk-kibana                 "/bin/tini -- /usr/l…"   10 hours ago     Up 10 hours                 0.0.0.0:5601->5601/tcp, :::5601->5601/tcp                                                                                                                                            docker-elk-kibana-1
44762bac28e8   docker-elk-logstash               "/usr/local/bin/dock…"   10 hours ago     Up 10 hours                 0.0.0.0:5044->5044/tcp, :::5044->5044/tcp, 0.0.0.0:9600->9600/tcp, :::9600->9600/tcp, 0.0.0.0:50000->50000/tcp, :::50000->50000/tcp, 0.0.0.0:50000->50000/udp, :::50000->50000/udp   docker-elk-logstash-1
cdf06845e4ad   docker-elk-elasticsearch          "/bin/tini -- /usr/l…"   10 hours ago     Up 10 hours                 0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 0.0.0.0:9300->9300/tcp, :::9300->9300/tcp                                                                                                 docker-elk-elasticsearch-1
00a4232429ac   jc21/nginx-proxy-manager:latest   "/init"                  12 hours ago     Up 12 hours                 0.0.0.0:80-81->80-81/tcp, :::80-81->80-81/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp 
antoineco commented 9 months ago

Thank you.

I see that you don't have Metricbeat running. You need to run agents like Metricbeat and Filebeat to collect data on your host(s), and forward it to Elasticsearch.

Check the extensions/ directory inside the repository.

HaSaKiYasuooo commented 9 months ago

谢谢。

我发现您没有运行 Metricbeat。您需要运行 Metricbeat 和 Filebeat 等代理来收集主机上的数据,并将其转发到 Elasticsearch。

检查extensions/存储库内的目录。

When I install Windows agent, I chose Run standalone, so can these issues just be solved with Fleet and Metricbeat image

antoineco commented 9 months ago

Right, the Elastic Agent is fine too, it runs Metricbeat and Filebeat for you.

Can you share the logs of the agent? If it's not sending data, it means that it probably wasn't able to connect to Elasticsearch.

antoineco commented 9 months ago

Closing due to inactivity. Feel free to reopen.

HaSaKiYasuooo commented 9 months ago

是的,Elastic Agent 也很好,它为您运行 Metricbeat 和 Filebeat。

可以分享一下agent的日志吗?如果它没有发送数据,则意味着它可能无法连接到 Elasticsearch。

Windows has used the agent to transfer logs ,so where can I view agent logs? image

antoineco commented 9 months ago

In the Windows Event Viewer: https://www.freecodecamp.org/news/event-viewer-how-to-access-the-windows-10-activity-log/