CasperKristiansson
commented
1 month ago Feedback
By Casper Kristiansson and Muhammadsodiq Nematjanov
Code of Conduct
We certify that generative AI, incl. ChatGPT, has not been used to write this feedback. Using generative AI without permission is considered academic misconduct.
High-level strengths and Weaknesses:
High-Level Strengths
-
Presentation Structure: The presentation was well-delivered and compared - both the traditional approach in Weka and the method based on ChatGPT.
-
Understanding of Technology: Good understanding of both machine learning techniques utilized in Weka and the large language model approach used in ChatGPT.
-
Balance w/Time: Great balance within the 7-minute mark by balancing the explanations with enough technical detail to appeal to the audience.
Weaknesses
-
Paper Selection: The chosen paper did not have that much depth in view for scalability and the practical use of ChatGPT in real-world environments on a large scale.
-
Lack of Technical Detail on ChatGPT: The presentation could have gone into further detail about peculiar challenges associated with integrating log data into ChatGPT apart from the technical configurations w.r.t temperature and token settings.
-
Shallow: Weka has been explained; in that regard, it was able to connect it with other tools, which would provide a better overview of the industry.
Overall Feedback
Overall, your presentation was well-delivered and you provided a clear and structured comparison between the traditional Weka approach and the newer ChatGPT-based method for log analysis in DevSecOps. Your explanation of both approaches showed that you both have an understanding of the underlying technology, and the contrast between Weka’s established machine learning techniques and ChatGPT’s innovative large language model (LLM) approach was interesting to hear about as I was not overly familiar with the subject before.
That said, I would like to address some points. First of all, I think that perhaps the paper might not have been the best choice for this assignment. While it did cover some interesting ground—especially with its comparison of a traditional machine learning toolkit and an emerging LLM—it lacked depth in certain areas, particularly around the scalability and applicability of ChatGPT in real-world, large-scale environments, as you mentioned. The absence of detailed methodology in the paper made it difficult to evaluate how the proposed solutions would fare in more demanding contexts, and this perhaps limited the overall depth of the presentation. For future assignments, it would be helpful to take a closer look at the selected paper to ensure it not only fits within the scope of the course but also has enough technical rigor to provide a strong foundation for discussion. I believe that this would allow you to focus on more intricate aspects of the topic and ensure the presentation feels comprehensive!
On the technical side, you did an excellent job presenting the steps involved in Weka’s classification process, including supervised learning methods such as kNN, J48, and DecisionTable. Your explanation of how Weka models handle log records, classify them as either normal or suspicious, and rely on large datasets for accurate predictions was very clear. You also highlighted the fact that Weka requires re-training whenever the log structure changes, which was a key point in your comparison. I would however have liked to hear a bit more about other tools, or is Weka the only industry standard tool? It would have been an interesting point to bring up.
Another area where I would have liked to see more was in the technical depth of the ChatGPT implementation. For example, the code snippet you briefly discussed (involving the Python API for log classification) was interesting to hear about, but I would have appreciated more detail about how the log records are formatted, what specific challenges arise in feeding data into the model, and how the temperature and token settings influence ChatGPT’s performance in this context. Of course, given the time constraints of the presentation and the somewhat lacking methodology of the paper, it’s understandable that you couldn’t delve too deeply into these aspects, but I think they could have enriched the discussion even further.
I recommend looking into the paper "Incorporating AI-Driven Strategies in DevSecOps for Robust Cloud Security" [1]. While your presentation compared ChatGPT and Weka about log analysis, this paper considers a more general perspective of AI in DevSecOps. It creates insights into how AI-driven strategies for threat detection, anomaly detection, and automatic incident response can defeat some scalability and efficiency limits that you have discussed. It discusses more specifically AI's role in dynamic adaptation to emerging threats and handling large-scale cloud environments that can enable deeper insight into the limitation of ChatGPT within real-world runtime scenarios. Drawing a comparison between the role of AI in DevSecOps explored in this paper with your novel approach using ChatGPT would be an interesting perspective on how to better integrate AI into DevSecOps workflows.
As a final note, I think you handled the time limit very well. It’s always difficult to fit everything into a 7-minute window, especially when dealing with technical subjects. I believe you struck a good balance between explaining the concepts and providing enough detail to engage the audience. That being said, I would encourage you to push for a bit more technical depth next time, as long as the paper chosen allows for it!
[1] Rangaraju, Sakthiswaran, Ness, Stephanie, & Dharmalingam, Rajesh. (2023). Incorporating AI-Driven Strategies in DevSecOps for Robust Cloud Security. International Journal of Innovative Science and Research Technology, 8(23592365), 10–5281.
Assignment Proposal
Title
Machine Learning-Based Run-Time DevSecOps: ChatGPT Against Traditional Approach
Names and KTH ID
Deadline
Category
Description
We will present a recent scientific paper from this year wherein they perform a comparative study between two different approaches to classifying suspicious server log activities and detecting potential threats. Specifically, it compares the classic, traditional machine learning approach of using Weka API for classification with a new proposed novel method of using ChatGPT for performing runtime log analysis. They explore the mechanics and potential of using ChatGPT in Python where context represents labelled data and the questions themselves contain the log records which are being evaluated. Beyond exploring the possibility of it, they also analyse the viability of the novel method, and examine its potential as well as its drawbacks and limitations, and then compares it with the tried and true Weka API method.
Our presentation will begin by elaborating on why such research is important for the field of DevSecOps, as well as describing the scenario being analysed in the paper. From there, we will go on to detail the workings of both the new and the traditional approach in detail, followed by their respective strengths and weaknesses. Afterwards, we will end the presentation with an explanation of the paper's conclusions.(Machine Learning-Based Run-Time DevSecOps: ChatGPT Against Traditional Approach)
Relevance
Security integration in the DevOps pipeline is crucial for maintaining a secure development environment. This paper addresses this need by introducing AI-driven automated log analysis techniques to detect security threats during runtime. It also compares this approach with traditional machine learning practices, providing valuable insights for enhancing security and efficiency in agile development workflows.