olcf / olcf-test-harness

OLCF Test Harness
https://olcf.github.io/olcf-test-harness/
Other
12 stars 4 forks source link

Refactor database logging backend #164

Open hagertnl opened 5 months ago

hagertnl commented 5 months ago

To prepare for other database logging, we need to abstract the InfluxDB logging code out of the harness core code.

Initial thoughts are that we can create a class called something like db_logger, which can enable multiple database backends simultaneously, and receives a dictionary of the test data (or metrics or nodescreen) that it logs to the respective databases.

This will enable us to go and add a Kafka database backend.

hagertnl commented 5 months ago

Note -- when we get to the metrics & node screen data, can we try to bundle messages together please? :) We did not in the original implementation, but when we run a 1000-node node screen, it currently has to log 1000 messages to InfluxDB, instead of 1 large message that has 1000 insertions. I don't know the API for this off the top of my head, but it should be possible.

hagertnl commented 1 month ago

Working to restructure with the following hierarchy:

libraries
|- rgt_database_loggers (dir) # akin to rgt_loggers
    |- rgt_database_logger.py # has a list of loggers to call for each log instance. Can support multiple simultaneous InfluxDB instances, or InfluxDB+Kafka
    |- rgt_datbase_logger_factory.py
    |- __init__.py
    |- db_backends (dir)
        |- base_db.py # defines an API used by specific databases
        |- influxdb.py # backend-specific