search

linuxserver / docker-diskover

A Docker container for the Diskover space mapping application
GNU General Public License v3.0
76 stars 16 forks source link

About - xpack.security.enabled=true #49

Closed martadinata666 closed 2 years ago

martadinata666 commented 2 years ago

linuxserver.io

Expected Behavior

When execute docker exec -u abc -it diskover python3 /app/diskover/diskover.py -i diskover-test /data, diskover should executed and store data to elastic.

Current Behavior

dedyms@homelab:~/docker/apptest$ docker exec -u abc -it diskover python3 /app/diskover/diskover.py -i diskover-test /data

            _ _     _                       
           | (_)   | |                      
         __| |_ ___| | _______   _____ _ __ 
        / _` | / __| |/ / _ \ \ / / _ \ '__| /)___(\
       | (_| | \__ \   < (_) \ V /  __/ |    (='.'=)
        \__,_|_|___/_|\_\___/ \_/ \___|_|   (\")_(\")

            "Crawling all your stuff."
            v2.0-rc.3 community edition (ce)
            https://github.com/diskoverdata/

2022-02-06 13:23:13,532 - elasticsearch - WARNING - HEAD http://elasticsearch:9200/diskover-test [status:401 request:0.018s]
Traceback (most recent call last):
  File "/app/diskover/diskover.py", line 938, in <module>
    indexexits = check_index_exists(options.index, es)
  File "/app/diskover/diskover_elasticsearch.py", line 123, in check_index_exists
    if es.indices.exists(index=indexname):
  File "/usr/lib/python3.9/site-packages/elasticsearch/client/utils.py", line 168, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/usr/lib/python3.9/site-packages/elasticsearch/client/indices.py", line 332, in exists
    return self.transport.perform_request(
  File "/usr/lib/python3.9/site-packages/elasticsearch/transport.py", line 415, in perform_request
    raise e
  File "/usr/lib/python3.9/site-packages/elasticsearch/transport.py", line 381, in perform_request
    status, headers_response, data = connection.perform_request(
  File "/usr/lib/python3.9/site-packages/elasticsearch/connection/http_urllib3.py", line 277, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/lib/python3.9/site-packages/elasticsearch/connection/base.py", line 330, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
elasticsearch.exceptions.AuthenticationException: AuthenticationException(401, '')

Steps to Reproduce

  1. Deploy diskover like sampled compose with - xpack.security.enabled=true

Suggestion/Help for test

  1. Change - xpack.security.enabled=false as set it to true, making elastic looking for SSL connection or enforced password login behavior. Not sure which one it is.

Result after

- xpack.security.enabled=false

dedyms@homelab:~/docker/apptest$ docker exec -u abc -it diskover python3 /app/diskover/diskover.py -i diskover-test /data

            _ _     _                       
           | (_)   | |                      
         __| |_ ___| | _______   _____ _ __ 
        / _` | / __| |/ / _ \ \ / / _ \ '__| /)___(\
       | (_| | \__ \   < (_) \ V /  __/ |    (='.'=)
        \__,_|_|___/_|\_\___/ \_/ \___|_|   (\")_(\")

            "Holy s*i# there are so many temp files."
            v2.0-rc.3 community edition (ce)
            https://github.com/diskoverdata/

2022-02-06 13:24:31,147 - diskover - INFO - Config file: /etc/diskover/config.yaml
2022-02-06 13:24:31,147 - diskover - INFO - Config env var DISKOVERDIR: None
2022-02-06 13:24:31,147 - diskover - INFO - No plugins loaded
2022-02-06 13:24:31,147 - diskover - INFO - Creating index diskover-test...
2022-02-06 13:24:34,075 - diskover_elasticsearch - INFO - Tuning index settings for crawl
2022-02-06 13:24:35,048 - diskover - INFO - maxthreads set to 12
2022-02-06 13:24:35,113 - diskover - INFO - Crawling dir tree /data...
2022-02-06 13:24:35,113 - diskover - INFO - found 7 subdirs at level 1, starting threads...
2022-02-06 13:24:35,115 - diskover - INFO - [ThreadPoolExecutor-0_0] finished crawling /data (7 dirs, 0 files, 0 B) in 0d:0h:00m:00s
2022-02-06 13:24:36,048 - diskover - INFO - [ThreadPoolExecutor-0_0] finished crawling /data/zoneminderdata (-1 dirs, 0 files, 0 B) in 0d:0h:00m:00s
2022-02-06 13:24:36,609 - diskover - INFO - [ThreadPoolExecutor-0_6] finished crawling /data/Porn (2 dirs, 183 files, 931.52 MB) in 0d:0h:00m:01s
2022-02-06 13:24:38,117 - diskover - INFO - CRAWL STATS (path /data, files 1844, dirs 90, elapsed 0:00:03.004205, perf 651.753 inodes/s, 5 paths still scanning ['/data/RandomFiles', '/data/Pictures', '/data/PDF', '/data/Downloads', '/data/InstaSave'], memory usage 37.37 MB)
2022-02-06 13:24:38,156 - diskover - INFO - ES UPLOAD STATS (path /data, uploaded 1186 docs, elapsed 0:00:03.004205, perf 394.780 docs/s)
2022-02-06 13:24:39,080 - diskover - INFO - [ThreadPoolExecutor-0_4] finished crawling /data/Downloads (75 dirs, 1005 files, 91.18 GB) in 0d:0h:00m:03s
2022-02-06 13:24:39,515 - diskover - INFO - [ThreadPoolExecutor-0_2] finished crawling /data/Pictures (6 dirs, 4463 files, 17.85 GB) in 0d:0h:00m:04s
2022-02-06 13:24:40,141 - diskover - INFO - [ThreadPoolExecutor-0_3] finished crawling /data/PDF (640 dirs, 2384 files, 10.86 GB) in 0d:0h:00m:05s
2022-02-06 13:24:40,980 - diskover - WARNING - [ThreadPoolExecutor-0_1] OS ERROR: [Errno 13] Permission denied: '/data/RandomFiles/testfolderfromlinux'
2022-02-06 13:24:40,980 - diskover - WARNING - [ThreadPoolExecutor-0_1] OS ERROR: [Errno 13] Permission denied: '/data/RandomFiles/testfolderfromwindows'
2022-02-06 13:24:41,189 - diskover - INFO - CRAWL STATS (path /data, files 11231, dirs 1269, elapsed 0:00:06.075956, perf 2321.610 inodes/s, 2 paths still scanning ['/data/RandomFiles', '/data/InstaSave'], memory usage 40.66 MB)
2022-02-06 13:24:41,190 - diskover - INFO - ES UPLOAD STATS (path /data, uploaded 11762 docs, elapsed 0:00:06.075956, perf 1935.827 docs/s)
2022-02-06 13:24:41,291 - diskover - INFO - [ThreadPoolExecutor-0_1] finished crawling /data/RandomFiles (539 dirs, 3196 files, 36.63 GB) in 0d:0h:00m:06s
2022-02-06 13:24:41,486 - diskover - INFO - [ThreadPoolExecutor-0_5] finished crawling /data/InstaSave (0 dirs, 8288 files, 1.68 GB) in 0d:0h:00m:06s
2022-02-06 13:24:41,496 - diskover - INFO - *** finished walking /data ***
2022-02-06 13:24:41,496 - diskover - INFO - *** walk files 19519, skipped 1532 ***
2022-02-06 13:24:41,496 - diskover - INFO - *** walk size 159.1 GB ***
2022-02-06 13:24:41,496 - diskover - INFO - *** walk du size 158.46 GB ***
2022-02-06 13:24:41,496 - diskover - INFO - *** walk dirs 1269, skipped 87 ***
2022-02-06 13:24:41,496 - diskover - INFO - *** walk took 0d:0h:00m:06s ***
2022-02-06 13:24:41,496 - diskover - INFO - *** walk perf 3515.307 inodes/s ***
2022-02-06 13:24:41,496 - diskover - INFO - *** docs indexed 20786 ***
2022-02-06 13:24:41,496 - diskover - INFO - *** indexing perf 3261.143 docs/s ***
2022-02-06 13:24:41,496 - diskover - INFO - *** bulk uploads took 0d:0h:00m:05s ***
2022-02-06 13:24:41,496 - diskover - INFO - *** warnings/errors 2 ***
2022-02-06 13:24:41,496 - diskover - INFO - Crawling dir tree /data completed in 0d:0h:00m:06s
2022-02-06 13:24:41,497 - diskover_elasticsearch - INFO - Setting index settings back to defaults

Compose file

version: '2'
services:
  diskover:
    image: lscr.io/linuxserver/diskover
    container_name: diskover
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Asia/Jakarta
      - ES_HOST=elasticsearch
      - ES_PORT=9200
    volumes:
      - ./diskover/config:/config
      - /media/Clouds:/data
    ports:
      - 28000:80
    mem_limit: 4096m
    restart: unless-stopped
    depends_on:
      - elasticsearch
  elasticsearch:
    container_name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:7.16.2
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./diskover/esdata:/usr/share/elasticsearch/data
    #ports:
    #  - 9200:9200
    depends_on:
      - elasticsearch-helper
    restart: unless-stopped
  elasticsearch-helper:
    image: alpine
    command: sh -c "sysctl -w vm.max_map_count=262144"
    privileged: true

Environment

OS: Ubuntu 20.04 CPU architecture: x86_64

github-actions[bot] commented 2 years ago

Thanks for opening your first issue here! Be sure to follow the bug or feature issue templates!

github-actions[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.