search

canonical / prometheus-openstack-exporter

OpenStack exporter for the prometheus monitoring system
GNU General Public License v3.0
128 stars 113 forks source link

pickle EOF Error #71

Closed kguille closed 1 year ago

kguille commented 5 years ago

When I run the prometheus-openstack-exporter I've this message:

Traceback (most recent call last): File "/home/ubuntu/prometheus-openstack-exporter-canonical/prometheus-openstack-exporter", line 727, in do_GET collectors = [COLLECTORS[collector]() for collector in config['enabled_collectors']] File "/home/ubuntu/prometheus-openstack-exporter-canonical/prometheus-openstack-exporter", line 326, in init self.prodstack = pickle.load(f)[0] File "/usr/lib/python2.7/pickle.py", line 1384, in load return Unpickler(file).load() File "/usr/lib/python2.7/pickle.py", line 864, in load dispatchkey File "/usr/lib/python2.7/pickle.py", line 886, in load_eof raise EOFError EOFError

I don't know how to correct it. Is there someone who can help me ?

jeevadotnet commented 5 years ago

I get exactly the same, I also closed my post about 5 minutes ago because there is a duplicated thread on the issues list.

kguille commented 5 years ago

I know, I saw your post but I don't find any solution, and I really need to correct the problem...

jeevadotnet commented 5 years ago

I'm in the same boat. Got >100 openstack nodes I want to monitor. where want = need

kguille commented 5 years ago

I understand but the other post is dated from December 2017, that's quite old...

jeevadotnet commented 5 years ago

I've also tried changing the pickle to cPickle as per the other thread but got a new error : Traceback (most recent call last): File "/opt/prometheus-openstack-exporter/prometheus-openstack-exporter", line 728, in do_GET collectors = [COLLECTORS[collector]() for collector in config['enabled_collectors']] File "/opt/prometheus-openstack-exporter/prometheus-openstack-exporter", line 243, in __init__ self.prodstack = pickle.load(f)[0] EOFError

jeevadotnet commented 5 years ago

Okay tested older versions now: If you go back to: https://github.com/CanonicalLtd/prometheus-openstack-exporter/tree/281fa84751f4b55cb3ab0416729e6f6fe560162f

and change prometheus-openstack-exporter import pickle to import cPickle as pickle

and change prometheus-openstack-exporter uncomment use nova volumes

it will work

kguille commented 5 years ago

I'm gonna have a look to this ! Thanks you !

But where do you find the line "use nova volumes" to uncomment ?

kguille commented 5 years ago

I've tested the old version and I've an another error when I refresh the web page :

Traceback (most recent call last): File "/opt/prometheus-openstack-exporter/prometheus-openstack-exporter", line 631, in do_GET neutron = Neutron() File "/opt/prometheus-openstack-exporter/prometheus-openstack-exporter", line 236, in init self.prodstack = pickle.load(f)[0] EOFError

What should I do to correct it ?

kguille commented 5 years ago

When I try to start my service, the systemctl status command show me that :

Feb 14 16:02:32 osdemo sh[11579]: keystone = keystone_client.Client(*ks_creds) Feb 14 16:02:32 osdemo sh[11579]: File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py", line 176, in init Feb 14 16:02:32 osdemo sh[11579]: self.authenticate() Feb 14 16:02:32 osdemo sh[11579]: File "/usr/lib/python2.7/dist-packages/positional/init.py", line 101, in inner Feb 14 16:02:32 osdemo sh[11579]: return wrapped(args, kwargs) Feb 14 16:02:32 osdemo sh[11579]: File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 581, in authenticate Feb 14 16:02:32 osdemo sh[11579]: resp = self.get_raw_token_from_identity_service(kwargs) Feb 14 16:02:32 osdemo sh[11579]: File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py", line 220, in get_raw_token_from_identityservice Feb 14 16:02:32 osdemo sh[11579]: ("Authorization Failed: %s") % e) Feb 14 16:02:32 osdemo sh[11579]: AuthorizationFailure: Authorization Failed: global name 'v3_auth' is not defined

jeevadotnet commented 5 years ago

AuthorizationFailure: Authorization Failed: global name 'v3_auth' is not defined

You need to use the RC for v3 in your admin.novarc file

jeevadotnet commented 5 years ago

So I tried my fix in my production environment. (It works in my test env).

Now I also get the issue of on the node metrics website:

  File "/opt/prometheus-openstack-exporter/prometheus-openstack-exporter", line 630, in do_GET
    neutron = Neutron()
  File "/opt/prometheus-openstack-exporter/prometheus-openstack-exporter", line 235, in __init__
    self.prodstack = pickle.load(f)[0]
EOFError

and from the process:


   Loaded: loaded (/etc/systemd/system/prometheus-openstack-exporter.service; disabled; vendor preset: enabled)
   Active: active (running) since Thu 2019-02-21 11:17:44 UTC; 3s ago
 Main PID: 39583 (python)
    Tasks: 2
   Memory: 64.2M
      CPU: 468ms
   CGroup: /system.slice/prometheus-openstack-exporter.service
           └─39583 python /opt/prometheus-openstack-exporter/prometheus-openstack-exporter /etc/prometheus-openstack-exporter/prometheus-openstack-exporter.yaml

Feb 21 11:17:44 B-05-37-openstack-ctl systemd[1]: Started prometheus-openstack-exporter.
Feb 21 11:17:45 B-05-37-openstack-ctl python[39583]: Starting data gather thread
Feb 21 11:17:45 B-05-37-openstack-ctl sh[39583]: Starting data gather thread
Feb 21 11:17:45 B-05-37-openstack-ctl python[39583]: Error getting stats: Traceback (most recent call last):
                                                       File "/opt/prometheus-openstack-exporter/prometheus-openstack-exporter", line 195, in run
                                                         keystone, nova, neutron, cinder = get_clients()
                                                       File "/opt/prometheus-openstack-exporter/prometheus-openstack-exporter", line 106, in get_clients
                                                         auth_domain = v3.Password(**ks_creds_domain)
                                                     TypeError: __init__() takes at least 2 arguments (1 given)
Feb 21 11:17:45 B-05-37-openstack-ctl sh[39583]: Error getting stats: Traceback (most recent call last):
Feb 21 11:17:45 B-05-37-openstack-ctl sh[39583]:   File "/opt/prometheus-openstack-exporter/prometheus-openstack-exporter", line 195, in run
Feb 21 11:17:45 B-05-37-openstack-ctl sh[39583]:     keystone, nova, neutron, cinder = get_clients()
Feb 21 11:17:45 B-05-37-openstack-ctl sh[39583]:   File "/opt/prometheus-openstack-exporter/prometheus-openstack-exporter", line 106, in get_clients
Feb 21 11:17:45 B-05-37-openstack-ctl sh[39583]:     auth_domain = v3.Password(**ks_creds_domain)
Feb 21 11:17:45 B-05-37-openstack-ctl sh[39583]: TypeError: __init__() takes at least 2 arguments (1 given)```
kguille commented 5 years ago

Ok but I don't remember where I can change this error. I didn't try the service since last week. Do you have an example for the admin.novarc file ?

Actually, my admin.novarc file looks like this :

export OS_USERNAME=admin export OS_TENANT_NAME=service export OS_PASSWORD=stack

export OS_REGION_NAME=RegionOne

export OS_AUTH_URL=http://192.168.1.9:35357/v3

Is there any mistake ?

songshanshi commented 5 years ago

use nova volumes self.use_nova_volumes = config.get('use_nova_volumes', True) is this? @kguille this mistake you need get v2 admin.novarc in your dashbroad of openstack

devrandom0 commented 5 years ago

I think you have many project. You should wait

jeevadotnet commented 5 years ago

Anyone got to an answer for this issue?

I'm using @sinamoghaddas docker from his fork and getting this error when I go to http://server:9183/metrics

Traceback (most recent call last):
  File "/prometheus-openstack-exporter", line 728, in do_GET
    collectors = [COLLECTORS[collector]() for collector in config['enabled_collectors']]
  File "/prometheus-openstack-exporter", line 327, in __init__
    self.prodstack = pickle.load(f)[0]
  File "/usr/local/lib/python2.7/pickle.py", line 1384, in load
    return Unpickler(file).load()
  File "/usr/local/lib/python2.7/pickle.py", line 864, in load
    dispatch[key](self)
  File "/usr/local/lib/python2.7/pickle.py", line 886, in load_eof
    raise EOFError
EOFError
devrandom0 commented 5 years ago

@jeevadotnet I think you have many project. You should wait

paulbrowne commented 5 years ago

I see exactly the same issue with the most recent commit, using a Ubuntu 18.04 base container image and a freshly deployed OpenStack with only 2 non-services projects in it;

#### Run a base Ubuntu 18.04 container
docker run -it -p 9183:9183 --name prometheus-openstack-exporter-18.04 ubuntu:18.04

# Install pre-requisites into container
apt-get update
apt-get install python-neutronclient python-novaclient python-keystoneclient \
python-netaddr unzip wget python-pip python-dev python-yaml -y
apt-get install python-openstackclient -y
apt-get install vim wget -y
apt-get install python-swift -y
apt-get install python-prometheus-client -y

# Create cache location
mkdir /var/cache/prometheus-openstack-exporter/
touch /var/cache/prometheus-openstack-exporter/staging2

# Clone, edit exporter config and OpenStack API credentials
cd /root/
apt-get install git -y
git clone https://github.com/CanonicalLtd/prometheus-openstack-exporter.git
cd prometheus-openstack-exporter/
vim prometheus-openstack-exporter.yaml 
vim novarc

# Source OpenStack credentials, test API access works with an image list (successful)
source novarc 
openstack image list

# Run the exporter
./prometheus-openstack-exporter prometheus-openstack-exporter.yaml
Starting data gather thread
Client setup done, keystone ver 3
Error getting tenants.list, continue with projects.list
172.17.0.1 - - [23/Apr/2019 14:04:41] "GET /metrics HTTP/1.1" 500 -

#### Test metrics collection from the container host;
[root@prometheus prometheus-openstack-exporter]# curl localhost:9183/metrics
Traceback (most recent call last):
  File "./prometheus-openstack-exporter", line 728, in do_GET
    collectors = [COLLECTORS[collector]() for collector in config['enabled_collectors']]
  File "./prometheus-openstack-exporter", line 327, in __init__
    self.prodstack = pickle.load(f)[0]
  File "/usr/lib/python2.7/pickle.py", line 1384, in load
    return Unpickler(file).load()
  File "/usr/lib/python2.7/pickle.py", line 864, in load
    dispatch[key](self)
  File "/usr/lib/python2.7/pickle.py", line 886, in load_eof
    raise EOFError
EOFError
onstring commented 5 years ago

I've met with the same issue(getting 500 errors when run it), the solution (actually it is not an issue) is just waiting for long enough time for the script to generate the cache file. And README file mentioned this as well:

https://github.com/CanonicalLtd/prometheus-openstack-exporter/commit/88016eff5393f9ca6e4fbffe9c576bfffb878070

digitalrane commented 4 years ago

Hi - from reading the comments - has anyone identified a bug here, or is the issue that folks with larger environments are not waiting long enough for the initial scrape?

If there is a bug here, or some enhancement - PRs addressing this based on your fixes you've found would be very much appreciated. Thanks!

rezabojnordi commented 3 years ago

I had this problem and after checking the code I realized the problem is from my network please check the network

Pjack commented 1 year ago

Close the issue that stale over years.

davidbradlycurtis commented 1 year ago

In my case my issue was I wasn't running the exporter with permissions to write to the cache_file, meaning the file never was written, causing the pickle EOF error.

Running the exporter as sudo (or root) resolved this error.

davidbradlycurtis commented 1 year ago

Yes, running the exporter with elevated permissions (sudo or as root user) fixed the issue for me.

On Sun, Sep 10, 2023 at 4:45 AM m4xlmum @.***> wrote:

Have you solved it yet.

— Reply to this email directly, view it on GitHub https://github.com/canonical/prometheus-openstack-exporter/issues/71#issuecomment-1712768387, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALEPTET5YXS2BHXDPCC3S73XZWD35ANCNFSM4GXM7EQQ . You are receiving this because you commented.Message ID: @.***>