Read about enrichment techniques when it comes to messages in Kafka. This includes various internet articles regarding implementation of a broker as a proxy which modifies the messages before they are received by the consumer. This idea did not have that many resources online(at least until 03.05 I couldn't find anything useful), thus I shifted my focus on finding another approach which may work and then I read about Kafka Streams and a bit more in detail in the book Kafka: The definitive guide.
Updated README and some small things in the repo (protecting the main branch, added linter as a job for checks and also creating the issue for the dev diary)
Containerized the application. Now one only needs to run the Dockerfile
Made some changes regarding pom.xml file and linter jobs
Had a look at possibilities of binding Python files in Java. The only alternative is to use Jython which is unfortunately outdated since it only works with Python 2.7. We thought about this idea because some projects on Github already implement what we want to do(our anonymization methods) thus we thought of binding the files with our project.
Checked whether the algorithm works fine, and it does. The problem was the small dataset.
Implemented the state store for the records. Now records (before they are anonymized) are going to get saved in the state store first, and each time a new record comes, the whole entries in state store are going to be fed to the Doca algorithm and they are going to be anonymized and written to the output topic.(TODO figure out the best way to feed the data to the algorithm and implement it)
Config file for Delta Doca where the user can input the fields he/she wants to anonymize(we'll have a look whether this should be changed or not) and also the parameters for it
This is the dev diary of @fjoniyz