Feedback for Group Epsilon

[lbechberger]

This is the thread where all the other groups leave their feedback for the documentation of group Epsilon.

[apukropski]

Your introduction describes comprehensively your project idea by providing good explanations of the questions you want to answer and why using classification is a possibility to solve the problem. Furthermore, you state the difficulties you might face, used technically accurate terms without contradicting yourself, list the databases you are planning to use and alternative solutions. Reading your text was easy because of your usage of sections with meaningful titles and examples to clarify your ideas whenever necessary. You explained your abbreviations most of the time. In general, we like the approach of splitting the work between you and the other groups.

All in all, we have only small remarks that could improve your description: 1) It would have been nice if you would have additionally listed the mentioned alternatives (you only indicate them by writing "one possible choice […]"). 2) You assume in "Choice of Input and Target" that the reader/user will always ask questions concerning a relation y, but did not explicitly state if it is possible to ask questions in the form of "what is x?" or what happens if the user asks such a question. Further information on this in the future would be helpful. 3) The abbreviation "QA system" is not explicitly explained, but given the context the meaning was quite clear.

[ljscfo]

It's great that you have described in detail to what extent your data set fulfills the desiderata. In addition, your two funny examples increase the comprehensibility.

Nevertheless, our task is to contribute with, admittedly, superficial criticism.

First thing about the grammatical accuracy: There are some typos and wrong sentences: "...the answer the question..." - probably you mean "...to answer the question..." "trainings data sets" -> "training data sets" "algorythm" -> "algorithm", I never got how to spell rhythm correctly either (sorry..) ... Double check your final documentation for those.

One little thing about consistency: Make sure that you use the same spelling for names every time ("KnowledgeStore" in this case (middle of first paragraph)).

Readability: No big deal at all, but please don't mix running text with bullet-point-like structures ("Objects to label: Words.." towards the end of the second paragraph.)

One last thing about completeness: Yesterday, I was relatively confused about why you only talk about the relation and RelationMentions in your documentation, but understood during the seminar today that each QA-group wants to focus on one different question type. In retrospect, you name it explicitly enough in your text, but nevertheless it would have helped me if you said something like: "Our group focuses only on the 'What is the relation between...?'-questions."

[junbohuang]

our dear group Epsilon,

well done on your code and documentation! they seem promising to get a good result based on your constraints.

For the content: technically accurate. good awareness on the term you used(agent and patient). Although you change them from subject/object to agent/patient. I would say it's still consistent because you adapt to new situation quite quickly. Also you explained the reason why you change the terms, meaning of the terms pretty in detail. However, you didn't explain how you retrieve the agents and objects in your documentation (although it can be easily seen from the code). What is the general pipeline getting these two entities and how their relation is extracted? Readers might not be able to get this information by purely looking at the documentation. This can be improved to be complete. As for the decision making processing, i would say you did a really good job on it, as the only thing you need now is the time to run your code.

For the code: well structured. well documented. easy to understand. intuitive variable and function names. most important thing is, it is simple. really good job!

For the style: grammatically correct and easily accessible. quite easy for readers to understand what you wrote. Separation of session is also clear (Terminology, Current State of the Data, Code). Consistent example (Mr Bigshot owns the Company BigCompany). There is no visualization, but i don't think is is a MUST in your last documentation. Good style in general!

Overall, i enjoy reading your documentation and your code. As a reader i feel myself in a flow reading your documentation and your code with high quality attention, because it's fluent, compact and delivers lots of information.

[pphilihpp]

Hello Group Epsilon,

this is the feedback for your documentation of week 26.11 - 02.12. Overall you provided a good documentation this week and it is clear, what the current state of your project is. But we also identified some weaknesses that we want to share with you.

Regarding the content: Your documentation is technically accurate and we enjoined your detailed description of the pandas data frame with its respective columns. You used your terms consistent through your whole text. Your documentation is almost complete, but you could have described the way, how your code works within the files data_generator.py and explorer.py a little bit more in detail. All of your Design Decision are reasonable and well explained and you should stick to such great explanations in the future as well.

Regarding the code: You linked your documentation only a few times to your source code, e. g. data_generator.py and explorer.py. We would have liked some more references and explanations of your source code in your documentation. Maybe you can create a better connection of your documentation to the source code in the future. The code quality seems fine. We are missing a little bit the "How to ...", so in the future, you should definitely include this in your documentation as well.

Regarding the style: Most of your text is easily accessible, grammatically accurate and well structured. You have some minor mistakes, e. g. the first sentence of your "Data Set Generation Process"-Paragraph "... generate rows for the pandas data frame discussed from each article in the table ..." is grammatically a bit weird. Due to the more or less complicated description of your Data Set, it would be really helpful for the reader, if you could provide some examples or visualizations. This would make it much easier to understand your Data Set and your approaches in the future.

Summing it all up, your documentation of this week is understandable and easily readable. Your biggest strength this week was the good description of your Design Decisions and you should definitely keep up these very good descriptions in the future. You can improve further parts of your documentation by linking your documentation better to your source code, by providing a "How to ..." that the reader knows how to execute your code and by explaining your concepts in examples and visualizations. Everything else is on a good level!

Well done!

[ljscfo]

It's good that you structure your documentation in a way that makes it easy to follow for a reader. That also holds for your argumentation for precision and why you use the baseline of always false instead of just using all or most baselines and metrics. Nevertheless, you could explain why the other metrics aren't interesting for you. Isn't the precision for the baseline "always false" just 0? Or 0/0? With that, you'd have a good metric, which score would increase from 0 if you later train a classifier.

Just from the last part of your documentation (part Data Set Exploration), it's difficult to understand what you classify. Are these articles which contain the answer for the question? We also were a bit confused about the True and False classifications. One could think that the dataset already contains a classifier which has some metric of low 0.597 %. But you're probably just referring to the ratio of positive and negative examples.

One thing about your argumentation for crossvalidation: Isn't that method especially good for small datasets? (With the downside of being computionally expensive.)

You said that you don't want the triples from one article only to be in one split. However, in a real-life application, you would have new articles to derive answers from. So it may be good for test and later validation to be done on triples of articles that were not used for training. So the better way might be the opposite - ensure, that each split contains one or more articles that aren't in any other split.

Three typos: SInce - since classifcations - classifications inbalance - imbalance (though inbalance is found often enough)

Nevertheless, the level of the language you use is impressing. Stay there.

[AnnaBruns]

Dear Group Epsilon,

Your documentation on the evaluation procedure of your news reader relation classifier was structured vey well and was very informative. There were no grammatical or spelling mistakes, you wrote in a accessible and clear manner and you used consistent terms. It was nice that you explained the different versions of your dataset and referred back to them throughout the documentation. You could have explained a bit more why you made certain decisions, e.g. why you trained those three classifiers and why it made sense to compare them by means of precision (or recall) and not on a different metric. We liked the sample predictions of your RF classifier a lot, because it showed that the results were actually sensible. Your code in classifier.py is clearly structured and it was comprehensible because you gave the functions and variables meaningful names. If you want to improve the code, add comments, so it becomes even more comprehensible.