No output from curator

[OmisNomis]

I have followed the steps listed in the installation instructions (I downloaded the Centos 7 RPM package as the server has no internet access) but I cannot run any commands.

Expected Behavior

When running a simple command such as curator --help I expect to see some output

Actual Behavior

There is nothing outputted.

Steps to Reproduce the Problem

1. Download rpm package from the link provided above
2. Perform a local YUM install
3. Confirm it has installed in /opt/elasticsearch-curator/ and the sym-link has been created in /usr/bin/curator
4. Try to run any command using the curator or curator_cli

Specifications

• Version: elasticsearch-curator-5.5.4-1.x86_64.rpm
• Platform: Centos 7

[OmisNomis]

Some examples:

server ~$ curator --help
server ~$ curator doesntexit
server ~$ curator --pleeeease
server ~$ curator --

[untergeek]

Does it run if you run it from /opt/elasticsearch-curator/curator or /opt/elasticsearch-curator/curator_cli?

I have the strangest issues with RHEL/CentOS 7 and this is not the first time this particular behavior has manifested.

[OmisNomis]

Hi @untergeek - neither of those work, no. I tried both of them (sorry, I didn't put it in the description)

[OmisNomis]

I'm assuming it doesn't need Python to run, as it has been installed via the RPM package? I have python installed, but it's V2...

[untergeek]

And you're 100% certain you downloaded the v7 RPM and not the v6 one?

[OmisNomis]

@untergeek 100%... I even removed everything and then got a colleague to sit along side me to make sure I wasn't doing anything stupid

[untergeek]

I can tell you that I cannot replicate this with my CentOS 7 test bed:

[buh@centos7-pkg-test ~]$ cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
[buh@centos7-pkg-test ~]$ uname -a
Linux centos7-pkg-test.untergeek.net 4.15.17-3-pve #1 SMP PVE 4.15.17-12 (Fri, 08 Jun 2018 11:18:32 +0200) x86_64 x86_64 x86_64 GNU/Linux
[buh@centos7-pkg-test ~]$ wget https://packages.elastic.co/curator/5/centos/7/Packages/elasticsearch-curator-5.5.4-1.x86_64.rpm
--2018-07-10 15:54:18--  https://packages.elastic.co/curator/5/centos/7/Packages/elasticsearch-curator-5.5.4-1.x86_64.rpm
Resolving packages.elastic.co (packages.elastic.co)... 107.21.237.188, 54.235.82.130, 184.72.242.47, ...
Connecting to packages.elastic.co (packages.elastic.co)|107.21.237.188|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14959417 (14M) [application/x-redhat-package-manager]
Saving to: ‘elasticsearch-curator-5.5.4-1.x86_64.rpm’

100%[=====================================================================================================================================================================================================================>] 14,959,417  11.6MB/s   in 1.2s

2018-07-10 15:54:19 (11.6 MB/s) - ‘elasticsearch-curator-5.5.4-1.x86_64.rpm’ saved [14959417/14959417]

[buh@centos7-pkg-test ~]$ sudo yum install elasticsearch-curator-5.5.4-1.x86_64.rpm
Loaded plugins: fastestmirror
Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
Examining elasticsearch-curator-5.5.4-1.x86_64.rpm: elasticsearch-curator-5.5.4-1.x86_64
Marking elasticsearch-curator-5.5.4-1.x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package elasticsearch-curator.x86_64 0:5.5.4-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===============================================================================================================================================================================================================================================================
 Package                                                          Arch                                              Version                                             Repository                                                                        Size
===============================================================================================================================================================================================================================================================
Installing:
 elasticsearch-curator                                            x86_64                                            5.5.4-1                                             /elasticsearch-curator-5.5.4-1.x86_64                                             54 M

Transaction Summary
===============================================================================================================================================================================================================================================================
Install  1 Package

Total size: 54 M
Installed size: 54 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : elasticsearch-curator-5.5.4-1.x86_64                                                                                                                                                                                                        1/1
  Verifying  : elasticsearch-curator-5.5.4-1.x86_64                                                                                                                                                                                                        1/1

Installed:
  elasticsearch-curator.x86_64 0:5.5.4-1

Complete!
[buh@centos7-pkg-test ~]$ curator --version
curator, version 5.5.4

[OmisNomis]

How bizarre! Is there anything else I can check? I've tried installing it three times now without success.

[untergeek]

Do you have anything else on the box? Is it in any way a non-standard installation? (security add-ons, selinux, etc?)

[OmisNomis]

There is the standard ELK software, that I installed using the RPM files, and node/Go but nothing that isn't standard.

[untergeek]

Version and patch level information? i.e. cat /etc/redhat-release and uname -a output, please.

[OmisNomis]

I'll have to do that tomorrow, if that's ok.

[OmisNomis]

cat /etc/redhat-release -> CentOS Linux release 7.4.1708 (Core)

uname -a -> Linux 3.10.0-693.21.1.el7.x86_64 #1 SMP Wed Mar 7 19:03:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux (server name removed)

[ceeeekay]

I have exactly the same problem, although I have one box that works (prod), and one that doesn't (dev). Ubuntu 16.04.4 LTS this time.

The only notable difference between the two is that dev has been hacked around a bit for different purposes, and used to have a pip-installed version of elasticsearch-curator on it, but this was removed prior to apt-installing 5.5.4 from the Elastic repo. I'm starting to suspect that left something behind which is affecting Curator.

I've spent about half a day on this so far and have no clue where to look, because the dev box refuses to give me any output at any time.

The only notable difference between the boxes is the affected box has the following extra packages on it, although I have no idea if they're related or not. Attempting to remove these threatens to uninstall ubuntu-server, so I'm not touching them for now:

diff of dpkg -l | grep python

-python3-apport
-python3-chardet
-python3-debian
-python3-newt
-python3-pkg-resources
-python3-problem-report
-python3-pycurl
-python3-six
-python3-software-properties

Happy to try anything you like on the dev box. It's got to the point where I'm considering rebuilding it for this issue.

[ceeeekay]

@untergeek any suggestions?

[heroInCommunity]

I have the same issue with Docker debian:stable-slim Dockerfile is:

FROM debian:stable-slim
RUN apt-get update && apt-get install apt-transport-https curl gnupg gnupg1 gnupg2 cron -y &&\
    curl -L https://packages.elastic.co/GPG-KEY-elasticsearch | apt-key add - &&\
    echo "deb [arch=amd64] https://packages.elastic.co/curator/5/debian9 stable main" > /etc/apt/sources.list.d/curator.list &&\
    apt-get update &&\
    apt-get install elasticsearch-curator -y &&\
    chmod a+x /etc/cron.daily/* &&\
    rm -rf /var/lib/apt/lists/*
CMD cron -f

Debian version is 9.3 (output of cat /etc/debian_version), curator - 5.5.4 (it was displayed during installation). Curator does not show any output on any parameters (including --help, --config) running from both locations /usr/bin/curator and /opt/elasticsearch-curator.

Previous version (it was like 5.4.*) worked fine on Debian 9.3 (with manual installation of libssl1.0.0 and setting unicode locales LC_ALL and LANG). However, I decided to try without that workarounds, as version 5.5.4 was promised to fix that issues, and failed now.

[anthonyloukinas]

Also having this issue with Docker using the latest Elasticsearch image from elastic.co running CentOS 7.

[root@4a06effd36e8 ~]# yum install elasticsearch-curator
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile
 * base: mirrors.liquidweb.com
 * extras: ftp.osuosl.org
 * updates: distro.ibiblio.org
Resolving Dependencies
--> Running transaction check
---> Package elasticsearch-curator.x86_64 0:5.5.4-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================================================================================================================================================================================================
 Package                                                                 Arch                                                     Version                                                     Repository                                                   Size
================================================================================================================================================================================================================================================================
Installing:
 elasticsearch-curator                                                   x86_64                                                   5.5.4-1                                                     curator-5                                                    14 M

Transaction Summary
================================================================================================================================================================================================================================================================
Install  1 Package

Total download size: 14 M
Installed size: 54 M
Is this ok [y/d/N]: y
Downloading packages:
elasticsearch-curator-5.5.4-1.x86_64.rpm                                                                                                                                                                                                 |  14 MB  00:00:01
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : elasticsearch-curator-5.5.4-1.x86_64                                                                                                                                                                                                         1/1
  Verifying  : elasticsearch-curator-5.5.4-1.x86_64                                                                                                                                                                                                         1/1

Installed:
  elasticsearch-curator.x86_64 0:5.5.4-1

Complete!
[root@4a06effd36e8 ~]#
[root@4a06effd36e8 ~]# curator --help
[root@4a06effd36e8 ~]# curator_cli --help

[root@4a06effd36e8 ~]# ls /usr/bin/ | grep curator
curator
curator_cli

[root@4a06effd36e8 opt]# cd /opt/elasticsearch-curator/
[root@4a06effd36e8 elasticsearch-curator]# ls
cacert.pem  curator  curator_cli  es_repo_mgr  lib

[root@4a06effd36e8 ~]# cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)

[root@4a06effd36e8 yum.repos.d]# cat curator.repo
[curator-5]
name=CentOS/RHEL 7 repository for Elasticsearch Curator 5.x packages
baseurl=https://packages.elastic.co/curator/5/centos/7
gpgcheck=1
gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch

Using this specific image docker.elastic.co/elasticsearch/elasticsearch:6.3.0

[OmisNomis]

I actually gave up on using Curator and just made a simple Node process that runs on a cron job every night.

[anthonyloukinas]

Looks like I'll do the same thing with Python. I actually like that solution better.. gives us some more customizability.

The bug still stands however, and I'd like to see it debugged some more. I was able to reproduce the same issue I pasted above on another host system running the same 6.3.0 latest container image.

[untergeek]

If you're going that route, @anthonyloukinas, you might as well use the curator API to achieve the same.

[heroInCommunity]

@untergeek could you please tell if you've managed to reproduce the issue on either CentOS or Debian? Please tell if you need any additional info.

[untergeek]

I have not yet been able to reproduce this on either CentOS or Debian (or Ubuntu).

[anthonyloukinas]

@untergeek Have you tried to reproduce this using Docker? Try to use the latest elastic.co elasticsearch image on any base operating system and see if you can make it work.

docker pull docker.elastic.co/elasticsearch/elasticsearch:6.3.0
docker run -it docker.elastic.co/elasticsearch/elasticsearch:6.3.0 bash

Copy the repo into /etc/yum.repos.d/ and then yum install elasticsearch-curator.

That's all I'm doing and I get no output. Elasticsearch itself is working completely fine. I've now tried this on CentOS + Ubuntu + OSX, as a docker image.

[anthonyloukinas]

@untergeek The Curator API looks nice, but it doesn't state what the actual PIP package is in the Documentation, and it also states it only support 5.x versions of Elasticsearch which doesn't help me as I'm running 6.3. I imagine nothing major has changed and it will most likely work.. Might take some tinkering.

Let me know if you have any working examples + which packages to install if you know off top. I installed pip install curator but i'm not sure this is the correct package.

EDIT: I located the actual package, elasticsearch-curator. They don't mention that in the documentation anywhere...

[untergeek]

And it also apparently hasn't been updated to reflect that it runs with any 5.x or 6.x Elasticsearch. It's the underlying API that Curator uses, so if Curator does it, that's how.

[anthonyloukinas]

Yup I can confirm that ES 6.x works fine with elasticsearch-curator. I'm pasting a working example in case anyone else here is having trouble getting the native package to work. I'll be running this in cron nightly at 12:30am.

Side Note: My indices look like this in ES netflow-index-2018.08.02. Logstash every night at midnight automatically creates new indices so that I can close them out easier.

from elasticsearch import Elasticsearch
import curator

es = Elasticsearch([{'host': 'localhost'}])
def delete_indices(index, days):
    """ Delete indices older than X days """
    ilo = curator.IndexList(es)
    ilo.filter_by_regex(kind="prefix", value=index)
    ilo.filter_by_age(source="creation_date", direction="older", unit="days", unit_count=days)
    delete = curator.DeleteIndices(ilo)
    try:
        delete.do_action()
        print "Removed "+index+" indices older than "+days+" days old."
    except curator.NoIndices:
        print "No "+index+" available for removal."

delete_indices("netflow-index*", 7)
delete_indices("syslog-index*", 60)

[untergeek]

@anthonyloukinas if you've installed elasticsearch-curator via pip, you have everything. You actually don't need the RPM.

Try it: curator --version or curator_cli --version. The entry_points were installed somewhere when you installed via pip.

[anthonyloukinas]

Good call! That is working on my localhost OSX curator --help.

When I install python + elasticsearch-curator on my Elasticsearch container I will try that and report back.

[untergeek]

I highly recommend using the latest 3.6 branch of Python. I haven't tested it with 3.7 yet

[ceeeekay]

@OmisNomis @untergeek Why has this been closed? I still don't see any way to get any output from Curator. None of the suggestions here work in my environment, and it's completely unusable.

[OmisNomis]

@ceeeekay I'd suggest you have a look at using the Elasticube API and run a small application on a cron job to get rid of the indices you don't want.

[heroInCommunity]

@OmisNomis you are proposing a workaround, while curator seems still have the problem with no output. Should the ticket be closed?

[OmisNomis]

@heroInCommunity I don't need this open, but perhaps I closed it in haste from a 'community' point of view, I just didn't see much movement on the underlying issue actually being fixed.

@untergeek - do you want this re-opened?

[ceeeekay]

@OmisNomis we use curator for a whole bunch of things, not just deleting indices. We close, delete, forcemerge and create snapshots in cron and it's all integrated with our automation tools. Changing to another application isn't a small task for us.

[OmisNomis]

@ceeeekay - happy to re-open it. I'll leave it in the hands of @untergeek on how he wants to progress it.

[untergeek]

I'm happy to re-open it. The easiest workaround right now is to install via pip, rather than RPM or DEB. I will have to do some docker-level debugging to find out why it doesn't work, and I'm a bit busy to delve deep super soon.

[untergeek]

I'm reasonably certain that a pip install will work without problems. It will function just like an RPM installation, though the installed paths may be a bit different. The entry_points that get installed will make it function just the same. If this approach is taken, I do recommend using Python 3.6, rather than 2.7. 3.5 is also acceptable.

[heroInCommunity]

Indeed, with pip it works, although again I needed to export LC_ALL=C.UTF-8 and LANG=C.UTF-8, like it was with previous version of curator.

[untergeek]

I needed to export LC_ALL=C.UTF-8 and LANG=C.UTF-8

That is true, regardless. The libraries expect UTC, as does Elasticsearch.

[ceeeekay]

In my case, missing LC_ALL was the cause of the problem, and the difference between the working and failing machines. Adding it to /etc/environment has solved the problem. Note that this is an apt install.

# curator
#

# export LC_ALL=en_NZ.UTF-8
# curator
Usage: curator [OPTIONS] ACTION_FILE

Error: Missing argument "action_file".

I guess an error at this point would have saved a lot of time?

[untergeek]

Thank you, @ceeeekay. That may actually be the solution for several others. I am unsure why no error is generated by docker images. I normally see an error when the wrong LC is set.

I will do some digging and see what else I can find.

[sferry]

Hi, I ve install curator via rpm, and though $ curator --help does give me the output there is nothing if I try $ curator_cli --config config.yml show_indices

I turned loggig on to DEBUG and left blacklist to [] to get the maximum info.

some lines puzzle me : << Traceback (most recent call last): File "/home/buh/.local/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn File "/home/buh/.local/lib/python3.6/site-packages/urllib3/util/connection.py", line 83, in create_connection File "/home/buh/.local/lib/python3.6/site-packages/urllib3/util/connection.py", line 73, in create_connection ConnectionRefusedError: [Errno 111] Connection refused

and << File "/usr/local/lib/python3.6/http/client.py", line 1239, in request File "/usr/local/lib/python3.6/http/client.py", line 1285, in _send_request File "/usr/local/lib/python3.6/http/client.py", line 1234, in endheaders File "/usr/local/lib/python3.6/http/client.py", line 1026, in _send_output File "/usr/local/lib/python3.6/http/client.py", line 964, in send File "/home/buh/.local/lib/python3.6/site-packages/urllib3/connection.py", line 166, in connect

These directories does not exist. And I don't see where the setting of theses directories is comming from. For some obscure reason the installed python is quite old: python-2.7.5-69.el7_5.x86_64

the server is $cat /etc/redhat-release CentOS Linux release 7.5.1804 (Core) $uname -a Linux xxxx.xxxx.xxxx.x 3.10.0-862.2.3.el7.x86_64 #1 SMP Wed May 9 18:05:47 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Is there somewhere a PATH definition to set ? Could it be related to to the current issue?

[untergeek]

@sferry Your issue seems to be unrelated to the parent issue. Yes, you are not seeing output, but if you turn on debug logging, you are seeing a ConnectionRefusedError, which indicates that Curator can't connect to the target machine.

Your system python version is irrelevant here, as the RPM version bundles its own (File "/home/buh/.local/lib/python3.6/site-packages/urllib3/connection.py").

So, while I appreciate that you're having a bad experience, I do not think it's related to this issue. Since it's a usage question at this point, please ask for some help in the discussion forums at https://discuss.elastic.co and I'll be happy to answer there.

[ambis]

Can confirm. CentOS Linux release 7.5.1804 (Core) (SELinux as enforcing)

Installed via yum repo. No output unless I run LC_ALL=fi_FI.UTF-8 LANG=fi_FI.UTF-8 curator_cli

Is this a Python problem, our could maybe the tool recongnize this situation and maybe tell the user what to do? This is pretty bad DX.. :)

[marvin-w]

Setting LC_ALL to any available locale solved it for me as well, thanks!

[iget-esoares]

I can confirm that it doesn't run if there's any problem with LC_ALL being unset.

[konstantin-kornienko-epam]

Same thing (Centos 7), thanks for this topic!

[zstyblik]

I can confirm that it doesn't run if there's any problem with LC_ALL being unset.

Well, my LC_ALL was set and it wouldn't run anyway.

[root@somewhere opt]# cat /etc/centos-release
CentOS Linux release 7.6.1810 (Core)
[root@somewhere opt]# printenv | grep LC
LC_ALL=C
LC_CTYPE=cs_CZ.utf-8
[root@somewhere opt]# curator
[root@somewhere opt]# curator --help
[root@somewhere opt]# LC_ALL=cs_CZ.utf-8 curator --help
Usage: curator [OPTIONS] ACTION_FILE

  Curator for Elasticsearch indices.

  See http://elastic.co/guide/en/elasticsearch/client/curator/current

Options:
  --config PATH  Path to configuration file. Default: ~/.curator/curator.yml
  --dry-run      Do not perform any changes.
  --version      Show the version and exit.
  --help         Show this message and exit.
[root@somewhere opt]# LC_ALL=cs_CZ.utf-8 curator --version
curator, version 5.7.6

Actually, it seems it doesn't work at any Centos 7 boxes for me. python 2.7.5 + LC_ALL=C. curator has been installed from package.

[untergeek]

LC_ALL has to be set to a UTF-8 value for Curator to work.

[ghost]

notice: from python 3.7 onwards it is not necessary to explicitly set LC_ALL or LC because this was fixed in python itself (PEP-538 and PEP-540)

for more details see this documentation from the used "click" library: https://click.palletsprojects.com/en/latest/unicode-support/?highlight=LANG or this downstream bugreport:

https://bugs.launchpad.net/kolla-ansible/+bug/1919328/comments/8