Week 7: Presentation proposal

[Glassar]

Assignment Proposal

Title

AI Anomaly Detection for log monitoring

Names and KTH ID

• Melvin Jakobsson (melvinj@kth.se)
• Jonatan Tuvstedt (jtuv@kth.se)

Deadline

• Week 7

Category

• Presentation

Description

In this presentation we will first describe traditional methods of log monitoring. We will then describe the current state of automatic AI log monitoring, including the underlying operating principles of them. Finally comparing and contrasting these two methods.

Relevance

This topic is relevant because traditional monitoring is limited to act on either predefined anomalies, or for a predefined limit of when the logs are considered an anomaly. In contrast AI anomaly detection tools are a new technology that is rapidly advancing, and which could potentially make log monitoring much more effective.

[algomaster99]

Looks good!

[TobbeCarlsson]

Generative AI:

I/We certify that generative AI, incl. ChatGPT, has not been used to write this feedback. Using generative AI without permission is considered academic misconduct.

High-level strengths:

• Good looking slides/structure: The slides look appealing and professional. Almost feels like something you could encounter at a Tedx talk :D The theme is also very futuristic which makes sense when having an AI related presentation. The structure is also clearly stated in the beginning which makes the presentation easy to follow.
• Good topic for devops: Log monitoring is an extremely important part of large scale systems and requires a large amount of data handling which AI is perfect for.
• Good reflection on the drawbacks: You have a clear section that discusses the potential drawbacks of AI in log monitoring which clearly showcases that the solution is not perfect. You also didn’t get stuck in only a single aspect of AI for log monitoring, you discuss strengths and weaknesses in regards to data security, performance, blind spots as well as interpretability.
• Good flow: Each of your slides and how you transition between them with your script flows together well and leaves us with an engaging presentation throughout.
• Competence: Our general impression was that you had performed more than enough research within the area, and that they knew what they were talking about. This further increases the validity and level of the presentation.

High-level weaknesses:

• Questionable color choice: Whilst the slides look very nice, it might be hard to distinguish blue text from a blue background. Especially when viewed in a lecture hall, this might provide a challenge for the audience. You got to remember those poor color blind people out there!
• Slide headers and font sizes: There seems to be different fonts and/or varying font sizes between different slides, whilst this is a minor aspect which is difficult to notice it is still worth thinking about. Also, each header is placed in a new location for each slide, which affects the smooth appearance of the presentation in a possibly bad way.
• Take home: It would be interesting to talk additionally about how companies could incorporate AI monitoring in addition to traditional monitoring. The take home is that the two can complement each other, but the general content of the presentation does not really reflect that message.
• Slide complexity: This is not necessarily a weakness but rather an important note. There are a quite substantial amount of things happening on the slides, so make sure that you actually take your time to explain clearly. Alternatively, if you perhaps do not have enough time to talk about all aspects mentioned in the slides, then it might be better to remove those parts. Doing this would make it easier to know what to look at as well.

Specific Strengths:

• Structure and content clarity: The initial walkthrough of the structure of the presentation was great! It provides the audience with clear expectations in terms of what's coming.
• Monitoring Comparisons: We truly appreciated the comparison between traditional and AI log monitoring. This gives the audience a clear understanding of the most important concepts of the presentation, whilst also putting these two in contrast to each other. To understand AI log monitoring, one must of course first understand AI-less log monitoring ;).
• Model Comparisons: Another aspect providing much information in a short span was the comparison between various models in terms of different points such as security and interpretability. This was a great way to tie the knot in terms of AI log monitoring tools, and increased our understanding of the various potential options out there.

Specific Weaknesses:

• Problem statement: It feels a bit like the problem statement is more related to why we need logging and monitoring when it could be even better if it’s related to traditional monitoring vs AI monitoring. There is a quite interesting article that discusses the problems with traditional log analysis and showcases why ML log analysis is the solution. This could be a good inspiration for you. This would also then tie better into your take home message.
• Technical content: During the walkthrough of DeepLog you could have included arrows that referred to the different aspects you described. The schemas themselves are pretty difficult to grasp on their own, especially for those with no experience in Deep Learning.
• Presenting effectiveness of AI monitoring: Would’ve been nice to get some additional information on the performance of those models. It is informative to hear of current models that you mention, but without knowing their accuracy it is difficult to grasp the relevance of these models. Are they the future of log monitoring, or is it simply an experiment as of now to use AI anomaly detection?
• Detail lacking at times: Could’ve been interesting to extend on the topic of these models being able to make predictions for data it has not been specifically trained on. How is this possible? That would deepen the presentation overall, and provide a clearer picture to the audience. Also a bit about how DataDogs can predict future normal behavior solely based on your logs. But maybe this information is not public, similar to how Dynarace and Azure did not describe in detail.

Summary:

In summary it was a really interesting presentation with a topic that we think many engineers who work on the devops side of things are super interested in. It sounds like you really understand the topic but have a few things here and there that you could improve to make the presentation even more coherent and easy to follow. Good luck! Although you probably don’t need it as long as you practice beforehand.