6mile
commented
6 years ago How big is your root disk? Do you have a separate disk for the Elasticsearch indexes?
If Metricbeat is running you can temporarily or permanently disable it. Metricbeat makes some VERY large indexes in Elasticsearch every day. Run this command to see all your indexes:
curl -X GET "localhost:9200/_cat/indices?v"
And you can see all your metricbeat indices with:
curl 'localhost:9200/_cat/indices'|grep metricbeat
If you don't need to keep the Metricbeat data you can delete it with:
curl -XDELETE http://localhost:9200/metricbeat"
Then do a df -h and confirm that your root volume has space.
You will probably have to restart Elasticsearch:
systemctl restart elasticsearch
When I log in to the SecureStack SIPServer I get an error that there is no space left on the device: creating lib/ansible.egg-info writing requirements to lib/ansible.egg-info/requires.txt writing lib/ansible.egg-info/PKG-INFO writing top-level names to lib/ansible.egg-info/top_level.txt writing dependency_links to lib/ansible.egg-info/dependency_links.txt writing manifest file 'lib/ansible.egg-info/SOURCES.txt' reading manifest file 'lib/ansible.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' no previously-included directories found matching 'ticket_stubs' no previously-included directories found matching 'hacking' warning: no files found matching 'SYMLINK_CACHE.json' writing manifest file 'lib/ansible.egg-info/SOURCES.txt' error: No space left on device -bash: cannot create temp file for here-document: No space left on device
Sure enough if I do a
df -hI see the root disk is full: [securestack@securestack-server-gce01 ~]$ df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 10G 10G 32K 100% / devtmpfs 2.3G 0 2.3G 0% /dev tmpfs 2.3G 0 2.3G 0% /dev/shm tmpfs 2.3G 241M 2.0G 11% /run tmpfs 2.3G 0 2.3G 0% /sys/fs/cgroup tmpfs 455M 0 455M 0% /run/user/0 tmpfs 455M 0 455M 0% /run/user/6969The SecureStack SIPServer is still working but I don't see any current data.