piler docker cannot start

[GUHU0433]

Hy, I want to use piler docker (community edition). I started with vers. 1.3.10 which was called the "community edition". After starting the container, I got the message: "cp: cannot stat '/etc/piler/piler-nginx.conf.dist': No such file or directory" I assume, sth got wrong.??

[jsuto]

I've just updated the docker image, sutoj/piler:1.4.4, try using it.

[jsuto]

Also, checkout the docker compose stuff in the piler source tree docker directory: https://bitbucket.org/jsuto/piler/src/master/docker/

[sushshring]

Still seeing this error on the image 1.4.4

[jsuto]

Something is wrong on your side. Try removing your volumes and start over.

docker run --rm -ti --name aaa sutoj/piler:1.4.4 bash
root@fd4adef256ad:/# ls -la /etc/piler/piler-nginx.conf.dist 
-rw-r--r-- 1 root root 2552 Jul 23 12:35 /etc/piler/piler-nginx.conf.dist

[zyradyl]

Hello, Sorry to reopen this, but the issue is occurring when you set a docker volume to the /etc/piler mount point. The docker volume that is mounted is blank, it has no files in it, however the build script stores files there and the startup script expects them to be there as well.

docker volumes, as well as most container volumes (i am deploying to a home kubernetes cluster), do not act as an overlay FS, they simply mount over the directory, which erases the contents as far as piler is concerned.

the only solution I have found to this so far is to just manually create the images in the directory that piler is expecting, but it is a bit hostile for a user experience.

edit: also needed to manually copy manticore.conf.dist to manticore.conf for sed edit 2: the mounts also seem to have overwritten the webserver root, i had to pull that in from the deb file as well. edit 3: perhaps instructions and volume mounts should be updated to only point at /var/piler/store? as of now this unfortunately means my deployment's webserver won't take advantage of my SSD based zfs pvc volume that the rest of the image is hosted on.

final edit: it seems like everything is running decently well now, except for the fact that no files in the store directory are being created during the run of the import. I'm hoping the script just first goes through and processes before actually downloading? idk

final edit2: nope, looks like having to work with the manual stuff caused me to mess something up. this is a nightmare lmao

final edit3: is tika even used anymore? and it does look like you're using a volume just to /store now which is nice. i guess i can rebuild the helm chart again.

((thank you for the wonderful software btw))

[jsuto]

How can I reproduce the issue you have?

[dieechtenilente]

I got kind of the same issue until I used 1.4.4 However, this results in the following error:

 docker logs piler
DEBUG: Generating piler.key
DEBUG: Making an ssl certificate
DEBUG: Writing /etc/piler/piler-nginx.conf
DEBUG: Writing /etc/piler/piler.conf
DEBUG: Writing /etc/piler/config-site.php
Database
information_schema
piler
DEBUG: mysql is ready
DEBUG: no metadata table, creating tables
 * Starting periodic command scheduler cron
   ...done.
 * Starting nginx nginx
   ...done.
DEBUG: main1.spp does not exist, creating index files
Manticore 6.2.12 dc5144d35@230822
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2023, Manticore Software LTD (https://manticoresearch.com)

WARNING: Error initializing columnar storage: MCL requires a CPU that supports SSE 4.2
WARNING: Error initializing secondary index: MCL requires a CPU that supports SSE 4.2
using config file '/etc/piler/manticore.conf'...
WARNING: key 'preopen_indexes' is deprecated in /etc/piler/manticore.conf line 16; use 'preopen_tables' instead.
FATAL: no tables found in config file '/etc/piler/manticore.conf'

docker-compose.yaml

version: "3"
services:
  traefik:
    image: traefik
    container_name: traefik
    restart: unless-stopped
    ports:
      - "443:443"
    volumes:
      - ./traefik.yaml:/etc/traefik/traefik.yaml:ro
      - ./acme.json:/etc/traefik/acme.json

  mysql:
    image: mariadb:10.9.7
    container_name: mysql
    restart: unless-stopped
    cap_drop:
      - ALL
    cap_add:
      - dac_override
      - setuid
      - setgid
    environment:
      - MYSQL_DATABASE=piler
      - MYSQL_USER=piler
      - MYSQL_PASSWORD=piler123
      - MYSQL_RANDOM_ROOT_PASSWORD=yes
    command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
    volumes:
      - db_data:/var/lib/mysql

  tika:
    image: sutoj/tika
    container_name: tika
    restart: unless-stopped
    cap_drop:
      - ALL
    deploy:
      resources:
        reservations:
          memory: 512M
        limits:
          memory: 512M

  memcached:
    image: memcached:latest
    container_name: memcached
    restart: unless-stopped
    cap_drop:
      - ALL
    command: -m 64

  piler:
    image: sutoj/piler:1.4.4
    container_name: piler
    init: true
    environment:
      - MYSQL_HOSTNAME=mysql
      - MYSQL_DATABASE=piler
      - MYSQL_USER=piler
      - MYSQL_PASSWORD=piler123
      - TIKA_HOSTNAME=tika
      - PILER_HOSTNAME=archive.yourdomain.com
      - MULTITENANCY=0
      - MEMCACHED_HOSTNAME=memcached
      - NODE_TYPE=WORKER
    ports:
      - "2525:25"
    volumes:
      - piler_etc:/etc/piler
      - piler_var:/var/piler
    healthcheck:
      test: curl -s smtp://localhost/
      interval: "60s"
      timeout: "3s"
      start_period: "15s"
      retries: 3
    deploy:
      resources:
        reservations:
          memory: 512M
        limits:
          memory: 512M

    depends_on:
      - "memcached"
      - "mysql"
      - "tika"

volumes:
  db_data: {}
  piler_etc: {}
  piler_var: {}

[jsuto]

I've just created a simpler docker file in the bitbucket repo for the opensource edition, see https://bitbucket.org/jsuto/piler/src/master/docker/docker-compose.yaml

Let me know if it works better for you.

[dieechtenilente]

@jsuto No luck with this docker-compose. I think, the manticore.conf is either missing some variables or some other files containing those variables

piler  | Database
piler  | information_schema
piler  | piler
piler  | DEBUG: mysql is ready
piler  | DEBUG: metadata table exists
piler  |  * Starting periodic command scheduler cron
piler  |    ...done.
piler  |  * Starting nginx nginx
piler  |    ...done.
piler  | DEBUG: main1.spp does not exist, creating index files
piler  | PHP Fatal error:  Uncaught Error: Undefined constant "NODE_TYPE" in /etc/piler/manticore.conf:27
piler  | Stack trace:
piler  | #0 {main}
piler  |   thrown in /etc/piler/manticore.conf on line 27
piler  | Manticore 6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)
piler  | Copyright (c) 2001-2016, Andrew Aksyonoff
piler  | Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
piler  | Copyright (c) 2017-2023, Manticore Software LTD (https://manticoresearch.com)
piler  |
piler  | using config file '/etc/piler/manticore.conf'...
piler  | FATAL: no tables found in config file '/etc/piler/manticore.conf'
piler exited with code 1```

[jsuto]

I don't think you are using the proper compose file or image. NODE_TYPE is a variable for the enterprise product. Again try the compose file I linked above.

[dieechtenilente]

I don't think you are using the proper compose file or image. NODE_TYPE is a variable for the enterprise product. Again try the compose file I linked above.

I used the linked docker-compose file with sutoj/piler:1.4.4

[cat24max]

This issue still exists.

[jsuto]

OK, then try the following. Clone the piler repo from https://bitbucket.org/jsuto/piler, then in the docker directory execute docker-compose start, and show me the output.

[dieechtenilente]

Okay somehow cloning the repo worked. I think the container needs volumes and cannot work with mounts.