Open hagertnl opened 5 months ago
Note -- when we get to the metrics & node screen data, can we try to bundle messages together please? :) We did not in the original implementation, but when we run a 1000-node node screen, it currently has to log 1000 messages to InfluxDB, instead of 1 large message that has 1000 insertions. I don't know the API for this off the top of my head, but it should be possible.
Working to restructure with the following hierarchy:
libraries
|- rgt_database_loggers (dir) # akin to rgt_loggers
|- rgt_database_logger.py # has a list of loggers to call for each log instance. Can support multiple simultaneous InfluxDB instances, or InfluxDB+Kafka
|- rgt_datbase_logger_factory.py
|- __init__.py
|- db_backends (dir)
|- base_db.py # defines an API used by specific databases
|- influxdb.py # backend-specific
To prepare for other database logging, we need to abstract the InfluxDB logging code out of the harness core code.
Initial thoughts are that we can create a class called something like
db_logger
, which can enable multiple database backends simultaneously, and receives a dictionary of the test data (or metrics or nodescreen) that it logs to the respective databases.This will enable us to go and add a Kafka database backend.