line / promgen

Promgen is a configuration file generator for Prometheus
https://line.github.io/promgen/
MIT License
1.04k stars 150 forks source link

Issues updating promgen version to 0.60.0 on Centos7 #483

Closed mysterious-maniac closed 4 months ago

mysterious-maniac commented 4 months ago

Hi all,

Looking to update our promgen from 0.46 -> latest 0.60.

A major difference I found is that recent versions are using python3.9, so it has been installed onto the Centos7 box.

However, when following the readme, docker-compose-bootstrap outputs the following message continuously:

# docker run --rm -it -v ~/.config/promgen:/etc/promgen/ line/promgen docker-compose-bootstrap
Waiting for database to startup
Waiting for database to startup
Waiting for database to startup

I've pulled the latest docker image and ~/.config/promgen/ has the same configuration files like we have on our 0.46. Is there something else I should be doing differently when upgrading between versions?

Thanks.

mysterious-maniac commented 4 months ago

Also note that this upgrade would be using the same redis/mysql instance.

I was able to get the bootstrap to work on versions 0.57 and below, however the migrate function would not work with errors regarding MySQLdb:

File "/usr/local/lib/python3.6/site-packages/MySQLdb/__init__.py", line 84, in Connect
    return Connection(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/connections.py", line 179, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
MySQLdb._exceptions.OperationalError: (2002, "Can't connect to MySQL server on '127.0.0.1' (115)")

The 0.57 will run with some issues and the "service" page will give an internal error: image

I don't really understand the bootstrap and migrate component of this. Are they absolutely necessary?

In my mind, this should be plug and play after pulling the docker image since I'm mounting a working configuration from ~/.config/promgen.

kfdm commented 4 months ago

I don't have a lot of time to debug this right now, but here are a few quick thoughts.

docker-compose-bootstrap is mostly just to have a simple docker-compose demo. Typically you wouldn't need to run that again for just an update.

docker-compose-bootstrap basically runs until promgen check 2>/dev/null in a loop, to check the database connection before continuing. I would check your database settings (Check DATABASE_URL environment variable)

Upgrades should generally just be

  1. Pull and run new image
  2. Run docker exec -it <container image> promgen migrate to update database
mysterious-maniac commented 4 months ago

Thanks for the response, @kfdm!

The command to update the database after the new image is pulled and run works flawlessly. I think the upgrade instructions should be added somewhere in the README.

I still have issues with the python3.9 versions (0.58+), but will submit another issue for that.