Feedback for Group Alpha

[lbechberger]

This is the thread where all the other groups leave their feedback for the documentation of group Alpha.

[Broconuts]

Your documentation is concise and technically accurate. While it is mostly complete, an explanation for the choice of this particular approach is missing; once you elaborate the details of your approach, you could add this. In its current form, your documentation omits any benefits of the approach. For example, it is not clear why you intend to design the classifier the way you do. In addition to this, structuring the document more clearly into sections like “Goals”, “Approaches” etc. could potentially contribute to readability. Finally, we found your first paragraph to provide a very good foundation to the subject matter at hand and consider your providing of links to enable further research to be helpful and adequately used.

[cstenkamp]

Your documentation is consistent and complete with regards to what we talked about in class - you mention almost all the methods we talked about in class about how to generate a dataset, including their upsides and downsides.

Regarding the actual approach/design decisions: First of all, you didn't make a design decision yet, you simply listed the advantages and disadvantages of several approaches, all of whose I agree with. I would also vote against 1 and 2, as they don't require machine learning, or artificially introduce it. I would also strongly advice against option 3, as it introduces a strong inductive bias, and the classifier will most likely not learn more than that simple heuristics you used to create these user-profiles. Option 5 is of course an option, but as it doesn't use our dataset, I would say your best bet is option 4, though I know how annoying it is to create an artificial dataset.

Regarding the completeness of the actual documentation: As you talked about all 5 approaches from a high-level perspective, the level of detail you provide for each of them is a bit too low for me. I know you haven't decided on an approach, but because of that you also didn't go into much detail about them - How precisely is your vector going to look like? Do you plan to use all categories in one "layer" (meaning that football and basketball are both on level 1), or do you want to subdivide the layers? What is your representation of the news articles? You said you don't want to use the provided categories, but is the vector generated by these? What if the user is only interested about articles mentioning Trump, and not Hillary Clinton - the dimension "American Politics" will not help there...

Regarding Style & Readability - I think it's generally useful to have more sub-categories, maybe just taking the questions from Lucas as captions, precisely answering them afterwards... And it's useful to look at a markdown styling cheat sheet to make the documentation more visually appealing, using sub-captions or lists, for example for your five options.

Giving a few concrete examples would of course also help understanding of your approaches, and I also think it helps yourselves, as after visualizing the concrete vectors you may notice yourselves if your approaches are feasible or not.

(as there is no code yet, I can of course not talk about that)

[AnnaBruns]

Dear Team Alpha,

The description of your data set generation explains quite well how you plan to obtain the user profiles and why the previously mentioned approaches are not suitable for this project. We did not find any contradictions or technically inaccurate statements. You provided examples to clarify your statements, structured your text in meaningful sections and added useful titles. What remains unclear to us is if you want to use all subcategories or only the ones with a certain size. As an improvement, you could go into more detail in the future. All in all, we have the feeling that you have invested a fair amount of time in planning the data set structure and that you are well on track.

[bmajumderuos]

Dear Team Alpha,

Overall your documentation is detailed, consistent and complete. Following are the comments for the individual areas:

Regarding actual approach/design decisions: The approach and design decisions have been clearly detailed. Regarding completeness of the actual documentation: The documentation seems complete with all the relevant details. Moreover, the reference to the methods used in the code was helpful. Regarding Style & Readability: Overall, the classification of topics and readability is great. One way to improve it even more would be to separate the text into paragraphs under suitable sub-headings.

Good luck with dataset!

[apukropski]

Dear Group Alpha,

You wrote in an easily accessible language how you acquired your dataset and reasoned about why you left out some of the categories. The well-structured description matches your code (even though the mentioned script is named differently in reality) and your usage of comments and meaningful variable/function names made it easy to understand your approach. You did not overcomplicate the facts and the visualisation of your dataset structure was beneficial as well. It would be helpful if you could state in the future why you have 30 liked articles for each user on top of the 30 positive training examples. Besides that, you seem to be up-to-date with the tasks and did a great job in our point of view.

[pphilihpp]

Hello Group Alpha,

this is the feedback for your documentation of Session 7, 04.12.18. You have documented all the necessary parts of this week and provided reasonable statements about your decisions on how to split your dataset and on how to evaluate the performance of your classifier later on. Overall you have written a great text, but we still have some improvements for your style of documentation in the future.

Regarding the content: Your documentation is technically accurate. The values you calculated based on the metrics for the different baselines are correct. Moreover, you have written in a consistent way so that the reader always knows, what you are talking about. Your documentation is mostly complete and your Design Decisions are mostly understandable. An exception is the F2-Score and Matthew's correlation. Since we did not talk about these measures in the seminar, it was not clear to us, what these metrics describe, and we would love a deeper explanation in your documentation of it. We also did not understand, why you will use this measures to evaluate your classifier. Generally, it would be good, if you could describe briefly what the different metrics mean in your approach and how you can calculate them. This would help other readers in understanding your documentation a lot.

Regarding the code: Unfortunately, you provided no links to your source code this time. Did you already implemented the dataset split or did you just talked in your documentation about it? This was sadly not clear to us. Maybe you could provide some links to your source code in the next time. You can also improve your code quality by deleting unnecessary comments like "#length of" or "#for i in sorted cat_lens" in your dataset_generation.py file. Moreover, you did not provide a "How to..." yet. This is not an urgent issue, but you should definitely provide a "How to ..." until the end of the project. Otherwise, the reader does not know how to execute your code on his local machine.

Regarding the style: We liked your style of writing. The text is easily accessible, mostly grammatically accurate and well structured. Some minor mistakes like "We want the same for our classifier for News recommendation.", where a preposition for news recommendation is missing or spelling mistakes "weigth" (correct "weight") could be avoided in the future. The example calculation of the metrics for the different baselines is really good and helps the understanding of your baselines very much. Since the distribution of your labels is 50 - 50, we wondered, what the difference between your baselines "50-50" and "Label Frequency" is. They should be the same and you should just compare your classifier to one of these two baselines in the future. Instead of just providing the resulting values you could have provided an example, how you calculated, for instance, Matthew's correlation. This would help the understanding of your baselines as well.

Finally, we would say your documentation for this week was good and you explained everything you did and thought about this week on a good level. We liked your reasoning about your Design Decisions and your baselines whereas we see a lot of improvements regarding your code (links in the documentation, quality and "How to ..."). You can consider some minor improvements regarding your style and your content (deeper explanation of your evaluation metrics) in the future as well.

Good luck with the rest of the project!

[pphilihpp]

Hello Group Alpha,

this is the feedback for your last part of the documentation of Session 13. Overall you did a good job in explaining which classifiers you use, how you tuned your hyperparameters and how you handled missing data in your features.

Regarding the content: Everything you described is sound and technically accurate from our point of view. The only thing we were a little bit confused about was your headline because we think this documentation part was for Session 13 instead of session 12. But that is of course not a big issue. Otherwise, your documentation is consistent and nearly complete. One point you could describe more in detail is the term "mutual_inf_classif" in your first paragraph. Is it a Python function? How does it work? For an outstanding reader this is not so clear, what you meant by this term. Otherwise, you described your Design Decisions very well this time! This is a huge improvement comparing to the previous documentation steps. Very well done!

Regarding the code: We are totally missing the links to the source code in your documentation. Since you did not provide any file structure, it is really difficult to see, where you implemented the different things that you have described in your documentation. Also there are still many unnecessary comments (like a lot of commented prints or similar things for example in the files classifiers.py and tc-idf-dic_creator.py). Moreover, your functions should have a short comment, what they supposed to do, so that the reader can understand the source code more easily. Maybe you could provide a clearer file structure as well.

Regarding the style: Most of your text is written in a simple way and easily accessible. Sometimes you used uncommon words, for example the word "elevated" in the last paragraph. This uncommon words were a little bit confusing for us and made the text minimally more difficult to read. But excepts these unusual words everything else was very good readable and due to your different headlines very well structured. Sometimes you did some grammatical mistakes, for example an "s" behind classifier is missing in the first sentence of your "Classifier selection" part ("We decided to use the classifiers random forest and maximum entropy because these classifier[s] yielded the best results when ..."). But most of the rest of your documentation is grammatically accurate. This time, it was not easily possible to provide good examples and visualizations, but the listing of the different values you obtained for using different classifiers in your documentation helped the understanding of your previously written text a lot.

Summing it all up you did a good job with your last documentation part! Your documentation is quite good, but your code lacks a bit of quality. Try to consider our little suggestions for an improvement of your documentation and then we think, that there is no doubt that your final, overall documentation will be good as well.

Good luck with the last step of this seminar!