DevDiary: Maxim

[nomorehumor]

01.05-07.05

• Researched more into the topic of Kafka broker, to get a deeper understanding of how messaging and topics works in the framework. Some knowledge was missing, since the first week I tried to just research broadly, what different approaches can be used to send information in the cluster (Kafka Clients, Connect, Streams)
• Researched the topic of Interceptors (ProducerInterceptor, ConsumerInterceptor). Found not much information online, only the official documentation and one example in a Medium article.
• Set up a development environment (IntelliJ IDEA, Kafka Docker containers, Maven)
• Created small example of a producer and ProducerInterceptor that mutates the sent message #14

[nomorehumor]

08.05 - 14.05

• Reviewed PRs #8 and #10
• Analyzed provided data schemas from Ampeers
• Implemented producer #26 and connected it to data generator implemented by @sofia-001. Also created application architecture.

[nomorehumor]

15.05 - 21.05

• Implemented interfaces for #32 ticket. We discussed with Sofia the interfaces that are needed here in context of clustering usage. Turned out that the #32 was not the best to start with in implementation of CASTLEGUARD alogrithmus, since all issues depend on this particular.
• Documented and applied changes to data schema received from Ampeers.

[nomorehumor]

22.05 - 28.05

• Preparation for Ampeers meeting
• Implemented outputCluster(), delayConstraint() and finished insertData() with usage of functions implemented by @sofia-001 and @ingastrelnikova

[nomorehumor]

29.05 - 04.06

• Refactored data generator and added streaming mode that allows streaming of data directly to Kafka. Merged the PR #55
• Discussed with Flo and Fjoni the architecture details of anonymization nodes. Discussed the problem of storage of non-anonymized values, since Delta-DOCA and CASTLEGUARD require prior values for generalization

[nomorehumor]

05.06 - 11.06

• Merged CASTLEGUARD changes of Inga and Sofia
• Fixed bug in CASTLEGUARD output function related to modification (through item generalization) of clusters during iterating through them
• Implemented model for CASTLEGUARD input/output

[nomorehumor]

12.06-18.06

• Reviewed PRs #31 and #66
• Fixed bug in Castleguard Item class
• Added executable for first tests of Castleguard

[nomorehumor]

19.06-25.06

• Designed and created ClusterManagement for castleguard together with @Ralfons-06. The goal was to decompose the operation with clusters, such as merging, as well as the clusters collections to a separate class
• Refactored Pipe by decomposing it in different functions to preserve simplicity and maintainability.
• Changed code that handles redis storage, added DataRepository as an abstraction for access to the redis storage. There was a problem previously that the keys of entries in the storage couldn't be retrieved in the runtime directly, but it is possible now

[nomorehumor]

26.06 - 02.07

Started working on the benchmarks. Created a plan to implement the monitoring module inside our system and the scenarios for testing. Researched some ways of benchmarking kafka. It turned out that Kafka can't provide any record timestamps, so direct testing of our Kafka Streams application became impossible

[nomorehumor]

03.07 - 09.07

Gathered with @ingastrelnikova and @overflw to discuss the benchmarking and its possible integration with visualization framework. We came up with some ideas and took a look at the OpenMessaging project that may help with the benchmarking. However, we decided that it will need to much effort to use it right now.

[nomorehumor]

10.07 - 16.07

• Implemented some metrics collection functionalities together with @ingastrelnikova and added timestamp creation in the pipe
• Reviewed some of the old PR with anonymization interface
• Changed return type for delta doca to a list of maps (header to value). Also changed the ways values are stored in redis from separate entries to hashmaps. There were several itereations to make both doca and castleguard conform to a single return type. Doca needed some preprocessing of values to cast them into double 2D array.
• Integrated Castleguard into pipe by applying anonymization interface to it.

[nomorehumor]

17.07 - 23.07

• Executed benchmarks on different algorithm configurations with Doca and Castleguard.
• Added new objects to castleguard. It was crucial for the benchmarking that ids of entries are known after their output from alogrithm. Therefore the anonymization interface was changed again and now both algorithms take lists of Item that stores id and the data. Afterwards the algorithms convert data to their appropriate type
• Helped @Ralfons-06 with creating plots for the presentation

[nomorehumor]

Lecture Free Period (24.07-31.08)

• Implemented monitoring module: LocalMetricsCollector, RemoteMetricsCollector and BenchmarkConsumer. The idea was to create a framework for application benchmarking that could be executed on a separate node and would not affect the performance of the application itself. This took a lot of time, since at first I tried to implement communication between different Java processes using NIO2 asynchronous sockets, which is a low-level solution (although for some reason is recommended by many internet sources). In the end I decided to use zeroMQ library that is already known to me and is much easier to use. This way I was able to implement the communication between modules
• Executed benchmarks on Google Cloud Platform. With help of @ingastrelnikova, who set up cloud environment, we deployed different components (Pipe, data generator, remote metrics collector/consumer, kafka broker) on different VMs. Then I executed pipe with castleguard/doca algorithms and different data generator throughput. Created plots.
• Started writing report, specifically performance assesment part.
• Refactored code repository and helped with several PR reviews