search

bitsensor / elastalert

ElastAlert that exposes REST API's for manipulating rules and alerts
https://bitsensor.io/
Other
367 stars 216 forks source link

Failing to start #153

Open mrwhoo opened 4 years ago

mrwhoo commented 4 years ago

Hello,

CentOS 7 Docker: Docker version 19.03.5, build 633a0ea ES, Kibana - 7.5.1

Any tips ... google is not helpful!

Thank you in advance.

@bitsensor/elastalert@2.0.1 start /opt/elastalert-server sh ./scripts/start.sh

00:26:05.086Z INFO elastalert-server: Config: No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json. 00:26:05.088Z INFO elastalert-server: Config: Proceeding to look for normal config file. 00:26:05.089Z INFO elastalert-server: Config: A config file was found in /opt/elastalert-server/config/config.json. Using that config. 00:26:05.109Z INFO elastalert-server: Router: Listening for GET request on /. 00:26:05.110Z INFO elastalert-server: Router: Listening for GET request on /status. 00:26:05.110Z INFO elastalert-server: Router: Listening for GET request on /status/control/:action. 00:26:05.110Z INFO elastalert-server: Router: Listening for GET request on /status/errors. 00:26:05.110Z INFO elastalert-server: Router: Listening for GET request on /rules. 00:26:05.111Z INFO elastalert-server: Router: Listening for GET request on /rules/:id. 00:26:05.112Z INFO elastalert-server: Router: Listening for POST request on /rules/:id. 00:26:05.112Z INFO elastalert-server: Router: Listening for DELETE request on /rules/:id. 00:26:05.112Z INFO elastalert-server: Router: Listening for GET request on /templates. 00:26:05.112Z INFO elastalert-server: Router: Listening for GET request on /templates/:id. 00:26:05.112Z INFO elastalert-server: Router: Listening for POST request on /templates/:id. 00:26:05.112Z INFO elastalert-server: Router: Listening for DELETE request on /templates/:id. 00:26:05.112Z INFO elastalert-server: Router: Listening for POST request on /test. 00:26:05.112Z INFO elastalert-server: Router: Listening for GET request on /config. 00:26:05.112Z INFO elastalert-server: Router: Listening for POST request on /config. 00:26:05.113Z INFO elastalert-server: Router: Listening for POST request on /download. 00:26:05.113Z INFO elastalert-server: Router: Listening for GET request on /metadata/:type. 00:26:05.113Z INFO elastalert-server: Router: Listening for GET request on /mapping/:index. 00:26:05.113Z INFO elastalert-server: Router: Listening for POST request on /search/:index. 00:26:05.116Z INFO elastalert-server: ProcessController: Starting ElastAlert 00:26:05.116Z INFO elastalert-server: ProcessController: Creating index 00:26:05.513Z INFO elastalert-server: ProcessController: Elastic Version:7 Mapping used for string:{'type': 'keyword'} Index elastalert_status already exists. Skipping index creation.

00:26:05.513Z INFO elastalert-server: ProcessController: Index create exited with code 0 00:26:05.513Z INFO elastalert-server: ProcessController: Starting elastalert with arguments [none] 00:26:05.520Z INFO elastalert-server: ProcessController: Started Elastalert (PID: 36) 00:26:05.521Z INFO elastalert-server: Server: Server listening on port 3030 00:26:05.522Z INFO elastalert-server: Server: Websocket listening on port 3333 00:26:05.522Z INFO elastalert-server: Server: Server started 00:26:06.142Z ERROR elastalert-server: ProcessController: Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main

00:26:06.142Z ERROR elastalert-server: ProcessController: "main", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/opt/elastalert/elastalert/elastalert.py", line 1929, in sys.exit(main(sys.argv[1:])) File "/opt/elastalert/elastalert/elastalert.py", line 1925, in main

00:26:06.142Z ERROR elastalert-server: ProcessController: client.start() File "/opt/elastalert/elastalert/elastalert.py", line 1106, in start

00:26:06.143Z ERROR elastalert-server: ProcessController: self.run_all_rules() File "/opt/elastalert/elastalert/elastalert.py", line 1158, in run_all_rules

00:26:06.143Z ERROR elastalert-server: ProcessController: self.send_pending_alerts() File "/opt/elastalert/elastalert/elastalert.py", line 1534, in send_pending_alerts

00:26:06.143Z ERROR elastalert-server: ProcessController: pending_alerts = self.find_recent_pending_alerts(self.alert_time_limit) File "/opt/elastalert/elastalert/elastalert.py", line 1526, in find_recent_pending_alerts

00:26:06.144Z ERROR elastalert-server: ProcessController: size=1000) File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/client/utils.py", line 84, in _wrapped

00:26:06.144Z ERROR elastalert-server: ProcessController: return func(*args, params=params, **kwargs) TypeError: search() got an unexpected keyword argument 'doc_type'

00:26:06.174Z ERROR elastalert-server: ProcessController: ElastAlert exited with code 1 00:26:06.174Z INFO elastalert-server: Server: Stopping server 00:26:06.175Z INFO elastalert-server: ProcessController: ElastAlert is not running 00:26:06.175Z INFO elastalert-server: Server: Server stopped. Bye!`

spacecabbie commented 4 years ago

Same here has to do with new version that was launched this week seeing last update in the repo might be a while.

alexshurik commented 4 years ago

Same. Any updates?

MisderGAO commented 4 years ago

same problem. container can not be started

deric commented 4 years ago

See Yelp/elastalert, Python 3 is required for Yelp/elastalert > 0.2 If you want to use Python 2, you should build this image with e.g.:

make build v=v0.1.39

The version was bumped here. Yes, the naming and versioning of these two projects is confusing...

rbkumar88 commented 4 years ago

getting the following error if i try to build using the above command,

Installed /usr/lib/python2.7/site-packages/configparser-0.0.0-py2.7.egg
error: The 'configparser>=3.5.0' distribution was not found and is required by elastalert

Also tried to run with the already hosted image "bitsensor/elastalert:2.0.0" which uses v0.1.39 build of elastalert I believe and getting the following error

npm info it worked if it ends with ok
npm verb cli [ '/usr/local/bin/node', '/usr/local/bin/npm', 'start', '--verbose' ]
npm info using npm@6.9.0
npm info using node@v12.2.0
npm verb run-script [ 'prestart', 'start', 'poststart' ]
npm info lifecycle @bitsensor/elastalert@2.0.1~prestart: @bitsensor/elastalert@2.0.1
npm info lifecycle @bitsensor/elastalert@2.0.1~start: @bitsensor/elastalert@2.0.1

> @bitsensor/elastalert@2.0.1 start /opt/elastalert-server
> sh ./scripts/start.sh

19:12:31.345Z  INFO elastalert-server: Config:  No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json.
19:12:31.346Z  INFO elastalert-server: Config:  Proceeding to look for normal config file.
19:12:31.347Z  INFO elastalert-server: Config:  A config file was found in /opt/elastalert-server/config/config.json. Using that config.
19:12:31.360Z  INFO elastalert-server: Router:  Listening for GET request on /.
19:12:31.360Z  INFO elastalert-server: Router:  Listening for GET request on /status.
19:12:31.361Z  INFO elastalert-server: Router:  Listening for GET request on /status/control/:action.
19:12:31.361Z  INFO elastalert-server: Router:  Listening for GET request on /status/errors.
19:12:31.361Z  INFO elastalert-server: Router:  Listening for GET request on /rules.
19:12:31.362Z  INFO elastalert-server: Router:  Listening for GET request on /rules/:id.
19:12:31.362Z  INFO elastalert-server: Router:  Listening for POST request on /rules/:id.
19:12:31.362Z  INFO elastalert-server: Router:  Listening for DELETE request on /rules/:id.
19:12:31.362Z  INFO elastalert-server: Router:  Listening for GET request on /templates.
19:12:31.362Z  INFO elastalert-server: Router:  Listening for GET request on /templates/:id.
19:12:31.362Z  INFO elastalert-server: Router:  Listening for POST request on /templates/:id.
19:12:31.363Z  INFO elastalert-server: Router:  Listening for DELETE request on /templates/:id.
19:12:31.366Z  INFO elastalert-server: Router:  Listening for POST request on /test.
19:12:31.366Z  INFO elastalert-server: Router:  Listening for GET request on /config.
19:12:31.366Z  INFO elastalert-server: Router:  Listening for POST request on /config.
19:12:31.366Z  INFO elastalert-server: Router:  Listening for POST request on /download.
19:12:31.366Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/:type.
19:12:31.366Z  INFO elastalert-server: Router:  Listening for GET request on /mapping/:index.
19:12:31.366Z  INFO elastalert-server: Router:  Listening for POST request on /search/:index.
19:12:31.368Z  INFO elastalert-server: ProcessController:  Starting ElastAlert
19:12:31.368Z  INFO elastalert-server: ProcessController:  Creating index
19:12:32.150Z  INFO elastalert-server:
    ProcessController:  Elastic Version:7
    Mapping used for string:{'type': 'keyword'}
    Index elastalert already exists. Skipping index creation.

19:12:32.150Z  INFO elastalert-server: ProcessController:  Index create exited with code 0
19:12:32.151Z  INFO elastalert-server: ProcessController:  Starting elastalert with arguments [none]
19:12:32.160Z  INFO elastalert-server: ProcessController:  Started Elastalert (PID: 51)
19:12:32.161Z  INFO elastalert-server: Server:  Server listening on port 3030
19:12:32.162Z  INFO elastalert-server: Server:  Websocket listening on port 3333
19:12:32.162Z  INFO elastalert-server: Server:  Server started
19:12:32.163Z ERROR elastalert-server:
    Server:  Error creating data folder with error: [Error: EROFS: read-only file system, mkdir '/opt/elastalert/server_data'] {
      errno: -30,
      code: 'EROFS',
      syscall: 'mkdir',
      path: '/opt/elastalert/server_data'
    }
(node:28) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'testFolder' of undefined
    at /opt/elastalert-server/src/controllers/test/index.js:18:64
(node:28) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:28) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
19:12:32.813Z ERROR elastalert-server:
    ProcessController:  Traceback (most recent call last):
      File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
        "__main__", fname, loader, pkg_name)
      File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
        exec code in run_globals
      File "/usr/lib/python2.7/site-packages/elastalert-0.1.39-py2.7.egg/elastalert/elastalert.py", line 1929, in <module>

19:12:32.814Z ERROR elastalert-server:
    ProcessController:      sys.exit(main(sys.argv[1:]))
      File "/usr/lib/python2.7/site-packages/elastalert-0.1.39-py2.7.egg/elastalert/elastalert.py", line 1925, in main
        client.start()
      File "/usr/lib/python2.7/site-packages/elastalert-0.1.39-py2.7.egg/elastalert/elastalert.py", line 1106, in start
        self.run_all_rules()
      File "/usr/lib/python2.7/site-packages/elastalert-0.1.39-py2.7.egg/elastalert/elastalert.py", line 1158, in run_all_rules

19:12:32.815Z ERROR elastalert-server:
    ProcessController:      self.send_pending_alerts()
      File "/usr/lib/python2.7/site-packages/elastalert-0.1.39-py2.7.egg/elastalert/elastalert.py", line 1534, in send_pending_alerts

19:12:32.815Z ERROR elastalert-server:
    ProcessController:      pending_alerts = self.find_recent_pending_alerts(self.alert_time_limit)
      File "/usr/lib/python2.7/site-packages/elastalert-0.1.39-py2.7.egg/elastalert/elastalert.py", line 1526, in find_recent_pending_alerts

19:12:32.816Z ERROR elastalert-server:
    ProcessController:      size=1000)
      File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/client/utils.py", line 84, in _wrapped
        return func(*args, params=params, **kwargs)
    TypeError: search() got an unexpected keyword argument 'doc_type'

19:12:32.857Z ERROR elastalert-server: ProcessController:  ElastAlert exited with code 1
19:12:32.857Z  INFO elastalert-server: Server:  Stopping server
19:12:32.857Z  INFO elastalert-server: ProcessController:  ElastAlert is not running
19:12:32.857Z  INFO elastalert-server: Server:  Server stopped. Bye!
npm verb lifecycle @bitsensor/elastalert@2.0.1~start: unsafe-perm in lifecycle true
npm verb lifecycle @bitsensor/elastalert@2.0.1~start: PATH: /usr/local/lib/node_modules/npm/node_modules/npm-lifecycle/node-gyp-bin:/opt/elastalert-server/node_modules/.bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
npm verb lifecycle @bitsensor/elastalert@2.0.1~start: CWD: /opt/elastalert-server
npm info lifecycle @bitsensor/elastalert@2.0.1~poststart: @bitsensor/elastalert@2.0.1
npm verb exit [ 0, true ]
npm timing npm Completed in 3023ms
npm info ok
smallersoup commented 3 years ago

same problem. container can not be started

Following is my pod.yaml 

kind: Pod
apiVersion: v1
metadata:
  name: es-alert-server-0
  generateName: es-alert-server-
  namespace: cdc
  labels:
    app: es-alert-server
    controller-revision-hash: es-alert-server-6d89cdf5d4
    statefulset.kubernetes.io/pod-name: es-alert-server-0
  annotations:
    kubesphere.io/containerSecrets: '{"es-alert-server":"demo-registry-secret"}'
    kubesphere.io/restartedAt: '2021-03-26T10:33:46.549Z'
    logging.kubesphere.io/logsidecar-config: '{}'
spec:
  volumes:
    - name: host-time
      hostPath:
        path: /etc/localtime
        type: ''
    - name: volume-3y9242
      persistentVolumeClaim:
        claimName: es-alert-data
    - name: volume-y3r2wv
      configMap:
        name: elastalert-test-yaml
        defaultMode: 420
    - name: volume-3c4pip
      configMap:
        name: elastalert-yaml
        defaultMode: 420
    - name: volume-qvzigq
      configMap:
        name: es-alert-config
        defaultMode: 420
    - name: volume-013ds1
      secret:
        secretName: es-cdc-a-cert
        defaultMode: 420
    - name: default-token-hg8h5
      secret:
        secretName: default-token-hg8h5
        defaultMode: 420
  containers:
    - name: es-alert-server
      image: 'harbor-xadd.staff.xdf.cn/library/karql-elastalert:4.0.0'
      command:
        - /bin/sh
        - '-c'
        - >-
          sed -i 's|10000|60000|' /opt/elastalert-server/src/common/websocket.js
          && npm start
      ports:
        - name: tcp-3030
          containerPort: 3030
          protocol: TCP
        - name: tcp-3333
          containerPort: 3333
          protocol: TCP
      resources:
        limits:
          cpu: '1'
          memory: 1000Mi
        requests:
          cpu: 100m
          memory: 100Mi
      volumeMounts:
        - name: host-time
          readOnly: true
          mountPath: /etc/localtime
        - name: volume-3y9242
          mountPath: /opt/elastalert
        - name: volume-y3r2wv
          readOnly: true
          mountPath: /opt/elastalert/config-test.yaml
          subPath: config-test.yaml
        - name: volume-3c4pip
          readOnly: true
          mountPath: /opt/elastalert/config.yaml
          subPath: config.yaml
        - name: volume-qvzigq
          readOnly: true
          mountPath: /opt/elastalert-server/config/config.json
          subPath: config.json
        - name: volume-013ds1
          readOnly: true
          mountPath: /certs
        - name: default-token-hg8h5
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: IfNotPresent
  restartPolicy: Always
  terminationGracePeriodSeconds: 30
  dnsPolicy: ClusterFirst
  serviceAccountName: default
  serviceAccount: default
  nodeName: master03
  securityContext: {}
  imagePullSecrets:
    - name: demo-registry-secret
  hostname: es-alert-server-0
  subdomain: es-alert-server-z93a
  affinity: {}
  schedulerName: default-scheduler
  tolerations:
    - key: node.kubernetes.io/not-ready
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - key: node.kubernetes.io/unreachable
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
  priority: 0
  enableServiceLinks: true


08:44:48.601Z INFO elastalert-server: Server: Server started

08:44:48.603Z ERROR elastalert-server:

Server: Error creating data folder with error: [Error: EACCES: permission denied, mkdir '/opt/elastalert/server_data'] {

errno: -13,

code: 'EACCES',

syscall: 'mkdir',

path: '/opt/elastalert/server_data'

}

rewrite docker image command to : sleep 10000, then exec container ，see following :

/opt/elastalert $ pwd
/opt/elastalert
/opt/elastalert $ mkdir aa
mkdir: can't create directory 'aa': Permission denied
/opt/elastalert $ ls -l
total 16
drwxrwxrwx    2 root     root             6 Mar 26 09:20 config-test.yaml
drwxrwxrwx    2 root     root             6 Mar 26 09:20 config.yaml
drwx------    2 root     root         16384 Mar 21 16:27 lost+found
/opt/elastalert $ whoami
node
/opt/elastalert $ cd ..
/opt $ ls -l
total 4
drwxr-xr-x    5 root     root          4096 Mar 21 16:27 elastalert
drwxr-xr-x    1 node     node            16 Mar 26 09:20 elastalert-server
drwxr-xr-x    1 node     node            80 Mar 12 12:00 yarn-v1.22.5
/opt $

user: node creates a directory in root's directory: elastalert