gabb1er
commented
8 months ago - added migration scripts to create new table for storing metric errors.
- added new enabler in storage configuration to control whether metric errors need to be saved in DQ Storage.
- added new enabler in encryption configuration to control whether data excerpts in metric errors should be encrypted.
- modified class storing metric errors and created its Slick table representation. Added error hash field that is used as part of unique key to deduplicate error records. Error hash is and MD5 hash string that is always computed using raw data excerpts even if it is encrypted later.
- added deduplication of finalized metric errors by their unique constraint in order to avoid sending repeating data.
- modified encryptor to make it faster (do not re-generate AES key and initial IV vector for every encryption attempt).
- fixed partition values filter in HiveSourceReader
- added more verbose logs for case when some of the source keyFields are not found within dataframe columns.
- enhances JDBC manager upsert method to avoid stack overflow errors when saving large sequences of results (quite usual case for storing metric errors).
- added Source validation on read: checks that all keyFields are presented in datafarame.
- refactored application settings logging.
- added implicit class to convert Option instances to Result instances.
- documentation is updated to reflect the changes.
cibaa-team-user