Review Ticket: Getting Started with Word Embedding in R

[walshbr]

The Programming Historian has received the following tutorial on 'Getting Started with Word Embedding in R' by @SaraJKerr. The lesson is under review and can be found at:

http://programminghistorian.github.io/ph-submissions/lessons/getting-started-with-word-embeddings-in-r

Please feel free to use the line numbers provided on the preview if that helps with anchoring your comments, although you can structure your review as you see fit.

I will act as editor for the review process. My role is to solicit two reviews from the community and to manage the discussions, which should be held here on this forum. I have already read through the lesson and provided feedback, to which the author has responded.

Members of the wider community are also invited to offer constructive feedback which should post to this message thread, but they are asked to first read our Reviewer Guidelines (http://programminghistorian.org/reviewer-guidelines) and to adhere to our anti-harassment policy (below). We ask that all reviews stop after the second formal review has been submitted so that the author can focus on any revisions. I will make an announcement on this thread when that has occurred.

I will endeavor to keep the conversation open here on Github. If anyone feels the need to discuss anything privately, you are welcome to email me. You can always turn to @ianmilligan1 or @amandavisconti if you feel there's a need for an ombudsperson to step in.

Anti-Harassment Policy

This is a statement of the Programming Historian's principles and sets expectations for the tone and style of all correspondence between reviewers, authors, editors, and contributors to our public forums.

The Programming Historian is dedicated to providing an open scholarly environment that offers community participants the freedom to thoroughly scrutinize ideas, to ask questions, make suggestions, or to requests for clarification, but also provides a harassment-free space for all contributors to the project, regardless of gender, gender identity and expression, sexual orientation, disability, physical appearance, body size, race, age or religion, or technical experience. We do not tolerate harassment or ad hominem attacks of community participants in any form. Participants violating these rules may be expelled from the community at the discretion of the editorial board. If anyone witnesses or feels they have been the victim of the above described activity, please contact our ombudspeople (Ian Milligan and Amanda Visconti - http://programminghistorian.org/project-team). Thank you for helping us to create a safe space.

[walshbr]

@SaraJKerr looking at your markdown syntax issues now - sometimes when you link to something you have a space between the brackets and the parentheses, as in paragraph 3. Getting rid of the space between brackets and parentheses will make the links actually parse as links. And then the other main syntax issue I see is the footnotes - some of the footnotes look like they have extra line breaks in them. The parser expects any note to only exist on a single line, so a line break will make it confused and think it's moved onto a new note. If you can clean up those things I'll take a look at the content of the lesson itself! Let me know if you have any problems with this or with getting Jekyll running - happy to help out.

[walshbr]

Ah and I just added a folder for the lesson in the assets folder for you @SaraJKerr - you'll need to upload the Raw_pepys.txt file in there when you get a chance. I should be able to finish reading this and commenting by next week, though I've started already.

[SaraJKerr]

@walshbr I have uploaded the Pepys_raw.txt file. I have also set up Jekyll to run on my laptop but it throws an error with the images "Could not locate the included file 'figure.html' " so I think that is an issue with my local folder set up. I have corrected the markdown for the hyperlinks and a couple of the images where there were missing speech marks. Should I upload it directly or create a branch?

[walshbr]

@SaraJKerr - we don't employ a branching strategy on this staging repository so you can go ahead and upload directly.

I can also try to troubleshoot the include issue - it's not coming up when I run ph-submissions locally, so it does sound like an issue with your local setup. If you want to send along a screenshot of the error message in your terminal and your folder setup I can take a look. Happy to keep talking troubleshooting on the review ticket here or take things back to email if you'd like.

[SaraJKerr]

@walshbr - I have uploaded the corrected version, but it is still bringing up a 404 when I try to open it on GitHub pages - so there must be something I have missed.

I'll try to sort out the local set up, but thanks for the offer of help.

[walshbr]

It's possible you have the wrong link? The staging version is here (coming up for me) - http://programminghistorian.github.io/ph-submissions/lessons/getting-started-with-word-embeddings-in-r. If that's not coming up though it might be a caching issue - I'd try clearing your cache. But if that doesn't work let me know and can try to troubleshoot more.

I cleaned up the footnotes for you. I'll take a look in the next week.

[SaraJKerr]

Thank you - that works. I think I must have missed a letter somewhere.

[walshbr]

Had some time, so I took at this earlier than anticipated. I think this is great in a lot of ways and headed in the right direction! It will be a great addition to the text analysis and R offerings that we have. The recommendations I have below are almost entirely requests for more explanation, analysis, and narrativizing throughout the piece. In particular, I think more explanation of the concepts underlying the technique and more engagement with the Pepys material will help gear the piece towards a humanist audience and give a clearer sense of how people could use this in their own research. I've added all my running notes from the read through as checkboxes so that you can mark things off as you go. Let me know if you have any questions or want to discuss anything. I think it makes sense to make these revisions before sending out to reviewers, as what I'm suggesting below would add a fair amount of text.

• [x] p. 1 - "They require relatively little preprocessing, so [they]…" seem to have a word missing
• [x] p. 3 - Is there a particular reason that R studio is used? Are there specific technical reasons, or is that just a general suggestion for lowering the difficulty of the lesson? We're in the process of working up some sustainability guidelines for authors and reviewers, so it might be worth taking a look at the preview we have on a pull request (soon to be implemented on the live site itself). Check out all three segments - they might prove helpful.
• [x] p. 5 - "The underlying structures created to enable computers to extract information from texts, for example for search algorithms, have increasingly been used explore meaning circulation within and between texts." I think there are a couple typos here, but also I might remove the aside about search algorithms to help the clarity.
• [x] p. 5 - I might bold those terms that are particularly important for the lesson to come here.
• [x] p. 6 - I might not include the image here in the middle of a sentence. I'd move it to the end of the sentence.
• [x] p. 6 - "as a multi-dimensional structures, " no plural here I think.
• [ ] "What are word embeddings?" - The section on word embeddings needs a little bit of massaging and explication to convey the topic. I think p. 6 and p. 8 work well with the Rubix example. p. 5 is a bit confusing, as it jumps from abstract concept to abstract concept, laying on the definitions of terms pretty thick. One way to restructure it might be to put the Rubix example first, then introduce definitions based on that. And then I think introducing a real-world example might help the section cohere, even at this early stage. Maybe one that foreshadows the kinds of case studies that you'll be doing later in the tutorial? Particularly since you list familiarity with R but not familiarity with any text analysis or statistical work - it might be a bit difficult for readers to follow.
• [ ] p. 8 - what do the numbers in the figure here represent? I.e. - how are they calculated, or what do the numbers represent about the relationships? Again, working with a more concrete example might help here.
• [ ] p. 10 - you'll need to link to the Pepys file.
• [ ] p. 12 - You might check out the other lessons we have on R to see how others make use or not of R Studio. The one on text processing in R, in particular, does a good job of introducing R Studio. Even though it recommends using R Studio, it is not necessarily integral to the lesson's instructions themselves. This is all to say that I would remove the screenshots of Studio itself - they're good for those following along using R Studio, but they're most likely to change if the GUI for R Studio changes over time. The best approach might be to offer the steps that one might need to set up R Studio in an aside for users with that setup but recognize that not everyone will be working in this way.
• [ ] p. 13 - again would include the figure at the end of the sentence here.
• [ ] p. 16 - if you're installing these from the console (not sure about in RStudio) you'll need to provide a mirror to use for the download. for example - install.packages('devtools', repos='http://cran.us.r-project.org'). Also I'd add a couple more sentences explaining what these lines are doing.
• [ ] p. 16 - Ben's github is bmschmidt, so the line here needs to be corrected accordingly to point to the right repo.
• [ ] p. 17 - I'd lowercase the first letter of the Pepys filename here and throughout.
• [ ] p. 17 - maybe worth mentioning that this sort of preprocessing is something that you do in virtually every text analysis context, and R is making your life easier.
• [ ] p. 17 - in general, I think a problem in the lesson so far is that it jumps in a bit too far without giving a grounding for what you're doing. I.e. - here's a sentence explanation and then how to do it. I'd narrativize a little more - offering a sentence or two here about the pipeline you're taking us through. Giving a sense earlier on of what the case study is that you'll be working with might help. And then you'll have that example in view that you can refer to along as you go.
• [ ] p. 19 and 20 are good examples of instances where it feels like a little more narrative could help. Maybe just a couple sentences reiterating where you are in the pipeline.
• [ ] p. 21 - maybe explain what you mean by model here? and what you mean by training? Or are you expecting readers to have that knowledge?
• [ ] p. 21 - explain what a skip-gram model is?
• [ ] p. 21 - might be worth noting that this particular model will take a while to load so that readers don't think their machine is hung up. I.e. - the comment about the length of training time is not hypothetical but something they're about to experience.
• [ ] p. 22 - I think vectors might be self-explanatory since you touched on it earlier, but might be worth reiterating. And I think window/context won't be self-explanatory for someone who is working with this for the first time. And what about the threads parameter?
• [ ] p. 26 - I think this is headed in the right direction but could use expanding. A particular example might help. An example of words compared based on the 1, -1, 0, etc. basis. What is an example of semantic and contextual similarity? Just a few more sentences would help round out the theory for the reader.
• [ ] p. 27 - again, figure after sentence ends.
• [ ] p. 28 to 30 - another good example of how you can narrativize more. Say something about the results, so that the reader gets a better sense of what the output is showing and how to interpret it / why it is useful.
• [ ] p. 29 - might be worth noting how the output will vary and why this might be the case for the reader's particular use. Ie prophecy is much further down on my list and Germany doesn't appear at all.
• [ ] p. 31 - another example of something that could use more explanation. T-distributed Stochastic Neighbour Embedding is probably not something the average reader will be aware of. Also worth explaining why you might want to plot these things. And whether or not this kind of plotting is the most useful way to examine them. Also explain what it means to reduce the dimensions to two and why that needs to happen?
• [ ] p. 33 - I think the graph could use a little parsing. Mostly just the part about how they relate to the word England. What are the axes showing? And how do we understand the terms in relation to one another as well as to England? Does the graph also serve as a reasonable representation of how similar they are to each other? Or just how similar the contexts are in relation to England.
• [ ] p. 34 - Ah you've addressed some of the comments above here. But I think it might be worth expanding on why the plot will be different each time. And also hand waving at the significance of the axes, even if you just explain why they're not significant.
• [ ] p. 35 - Ah great you're dealing with the graph more, which addresses my thoughts above somewhat. It might be helpful to get a sense of why the particular things here are better or worse, since you offer several options at once.
• [ ] p. 39 - ditto again for the randomized nature of the output.
• [ ] p. 44 - one thing that might be worth noting - the lines that point to a particular red dot here are just helping to identify the point at which a word is plotted right? The fact that they start to look directional is perhaps a point of confusion, since you're talking about distance anyway. So might just be worth clarifying the parts of the visualization that are meant for readability and which are actually subject to interpretation.
• [ ] p. 45 - I think you should do more explaining what clustering is here, before you have the reader go ahead and do it. The connection to topic modeling could also be fleshed out more. In addition I think the code block for clustering could use some explanation - seed, centers, etc. are not self-explanatory concepts. Might be worth giving an example of how this works a la the Rubix cube analogy from above. Again an explanation of the randomness would be helpful. A common question I get from students is "well if the results are different every time, how can we make any interpretations based on them?" Also worth explaining why you're running k means clustering here on the word embeddings model, and what kind of compelling interpretations that can lead you towards.
• [ ] p. 47 - to follow on my note from p. 45, I think you should do more to follow up on the example here. Why is this significant in the context of the Pepys diary? Can you offer a bit more in walking the reader through how to interpret the output?
• [ ] p. 48 - Again, more explanation needed here I think. It's not quite clear why you turn to KWIC after working with word embeddings or how the two methods relate to one another. Is the idea that KWIC gives you a fuller way of examining and following up on the information that you're getting from the WB model? It might not be in scope for the lesson as you set it up in the learning goals, though it certainly could be if you just articulate the connections a bit more. Might be a good opportunity for explicating the similarities or differences of the two approaches.
• [ ] p. 52 - I think this code block needs commenting to explain the different pieces.
• [ ] I think you'll need some sort of concluding section offering a reflection on what the reader has learned, the utility for research, and/or next steps if they wish to read further.

[SaraJKerr]

Thank you for the feedback. I’m finishing a chapter at the moment but will get started on the list as soon as I’m done.

[walshbr]

Sounds great @SaraJKerr - let me know if you wanted to discuss anything!

[walshbr]

I've been communicating with @SaraJKerr - she's going to aim to get revisions back by the end of March. So I'll check back in around March 25th.

[walshbr]

@SaraJKerr - just checking in on this given that March 25th just passed. Do you have an update on revisions?

[SaraJKerr]

@walshbr - Hi Brandon, I have had a look at the suggestions you have made and in light of those, and further research, I want to do a major rewrite, and possibly change the focus text. I'm not going to have time to do it at the moment, but will have time in the Summer. Can we revisit it then?

[walshbr]

Yep totally! I'll close this for now. Why don't you send a new outline along when you're ready to my email, we can chat a little off github to help scope and frame, and then go from there? I'll make a note to ping you in July if I haven't heard from you. Will close for now, but we can reopen when you're ready.

[walshbr]

@SaraJKerr - my reminder went off to ping you again about this. Feel free to send something along for review when you have a chance. We can reopen the issue whenever you have a chance to direct attention towards this.