Closed wcaleb closed 7 years ago
Thanks Tommy and Stephanie for submitting such a helpful and interesting lesson!
Before submitting this to reviewers, I thought I would share a few problems I encountered while working through it. I think a few quick fixes could solve the problems and make it easier for reviewers to evaluate the lesson:
dplaLink
or link
, and also only pulled down 1084 records.letters.csv
. I renamed letters-sample.csv
to letters.csv
in order to get the map running. (I also had to interpolate these steps from the README because the lesson draft dives into "Configuring the Map" without explaining how to get the map going.)I'd also like to raise a question here that you can either address now or wait until we've had feedback from the reviewers, but in general we hope to make Programming Historian lessons relatively self-contained. In this case that might make it preferable to incorporate instructions from your repo's README directly into the tutorial. It might be too difficult to include all of your code within our assets folder, but you may also want to do a little more explaining in the tutorial about the structure of the repo that you ask the reader to download---where would they find the scripts that they might want to modify? What goes in the css
folder and what is in the js
folder, etc?
That leads to a related question for you and our reviewers, which is what level (beginner, intermediate, or advanced) you want the tutorial to aim at. Right now I would probably list it as intermediate because it assumes a lot of facility with command-line tools. But it may be possible to reach a "beginner" audience with a little more explanation for some of the steps in the lesson, like getting the DPLA key, using GPS Visualizer, and helping the reader deal with minor problems like trying to install or run npm
without root permissions.
I'll await your initial response to these thoughts before contacting reviewers---the numbered issues are the only things I think need to be done pre-review, so it's up to you whether you want to address some of the other ones first or wait for additional feedback from outside reviewers.
Tommy and I have made the suggested changes to the lesson and fixed the problem with the parser not pulling down links. The README file should no longer be necessary. The number of results issue appears to be with the DPLA, they are missing about 500 records that were there when we began this project. I'll write to their staff and find out why those letters are missing as they are still available via the Internet Archive. In the meantime, I just corrected the number of files in the body of the lesson. As for level, we think with the more detailed explanations, an advanced beginner should be able to handle the lesson. We've added some links to basic instructions like how to use the command line and suggestions for dealing with installing software without root access.
Hi @sjrichmond, sorry for the slow reply to these revisions! I wanted to update you on where things stand. I have recruited one peer reviewer who will be posting his comments on this issue soon, and I'm currently working to secure a second peer reviewer. I will keep you updated as things develop!
Just an update that I have found a second reviewer for the lesson. Hopefully comments from both reviewers will be available here by the end of the calendar year. I will post any updates here as I have them.
Thanks! That is wonderful.
Here's my review of the lesson. Apologies for the long delay in getting them posted.
Overall, I think this is a fine addition to PH. The goal of creating an interactive map through custom code is an attractive and desirable task for many projects, and lessons that can unveil the process of creating such maps are welcome additions to the field. I think the authors do well in addressing their main goals in the lesson. As an audience of intermediate or advanced users seems like the desired target given the deep involvement with JavaScript that's required, although that tone occasionally jumps between advanced and beginner (for example, would a JavaScript developer need a section on "What is JavaScript" and "Why use Javascript"?)
Following through the instructions for installing Node, the required packages, and getting the project running were clearly explained and worked fine for me when I tested this on my system. Perhaps a bit more here about what users are installing (e.g., why they need to install jitsu
and what it's doing).
Since data collection and curation is a key component of the lesson, I think the explanation of acquiring data via an API and the justification behind storing the data as CSV is thorough and, I believe, exactly on point for such projects. I think the authors are exactly right to point out the durability of text-based data formats, as well as the ease of use that such formats allow for building interactive data visualization.
I'd like to see more explanation of the code. Even for more advanced users, the lesson itself doesn't do much to teach what's actually happening within the code to 1) parse and clean the data, and 2) how the interactive parts of the map are created and work together. In other words, if another user came to this lesson wanting to build their own map, it's not entirely clear to me how they would get started. Since part of your goal with this lesson is to explain a "rationale for simple code," it would be worth digging further into the code itself. I know this may add significantly to the length of the lesson, and maybe there's a middle ground to strike, but at the moment the code feels a bit like a black box. Perhaps that middle ground is extensive comments in the code itself that users can reference?
I would also wonder about adding a justification around tackling your own project of this kind versus using a platform like Carto or, perhaps more applicable to this project's goals, Palladio. Perhaps that's beyond the scope here, but since Palladio has mapping and filtering capabilities built into the platform to filter on various facets of the data, are there reasons why writing your own code is desirable?
The authors do a nice job explaining the analytical capabilities of the project. To further enhance this, I would suggest the authors include screenshots of the map in order to augment their claims in case users choose not to run the sample data while reading the lesson. This way, users not only get a sense of the final product being produced by the lesson, but the authors can further drive home their point by including the visualization itself. (I realize screen grabs break the point of interactive visualization, but I believe people using the lesson will want to know what they're creating).
Paragraph 2: "Many of the mistakes, misspellings and incorrect data points only became..." -- more for @wcaleb, does PH have a style guideline on Oxford comma usage?
Paragraph 4: "This lesson requires using the command line (or Command Prompt), if you have never used a command line interface before, you might want to read this getting started guide for the basics." Recommend a period after "(or Command Prompt)" to avoid run-on sentence or awkwardness with the two commas in this sentence.
Paragraph 5: "Unlike Java, JavaScript does not require installing browser add-ons or special software, it runs in all Internet browsers." Recommend replacing the final comma here with a semicolon.
More generally: are the sections "What is Javascript" and "Why use Javascript" necessary? My hunch is this is aimed at a more advanced audience anyway, who will be familiar (enough) with the language and already interested in programming. They won't need the justification for using JS.
Paragraph 7: "Before you begin, download the files from our repository on Github." Recommend a slight tweak, "download or clone the files..."
Paragraph 8: "and jQuery makes it easy to add interactive elements like the time line." Readers haven't been introduced to the time line feature yet, so this comes out of nowhere. Should it be removed, or an explanation come up early in the lesson about what exactly is being built?
Paragraph 9: "including summaries of the content, key words, links to the letter in an online repository" -- two things: 1) should "key words" be a single word? 2) the word "and" added between "...key words, [and] links to the..."
Paragraph 10: "standardized format for the MARC data" -- need to explain, briefly, what MARC is? A footnote would probably do.
Paragraph 11: "The two most popular, and arguably easiest to learn programming languages for working with data are Python and JavaScript" -- need a comma after "...working with data[,] are Python..."
Paragraph 12: "If you are using a PC, you will need to install cURL, if you are a Mac or Linux user, cURL comes pre-installed and you can skip over the rest of this paragraph." Recommend a period after "...using a PC, you will need to install cURL[.] If you are a Mac..."
Paragraph 15 and Paragraph 20: These two paragraphs are asking you to do the same step.
Paragraph 21: "...file called letters.csv We have also included a set..." Missing a period after letters.csv
.
Paragraph 21: "Return to the command window and run node get-data.js. This will download the data into a file called letters.csv We have also included a set of sample data in the files on GitHub (letters-sample.csv) if you want to test out the map without having to clean data and add coordinates yourself." Does it need to be explained how to do use the sample letters instead of the downloaded letters? Maybe note here you explain how to do this in Paragraph 32.
Paragraph 23: Need a comma, "...sent and received in Boston, Massachusetts[,] and are therefore..."
Paragraph 25: Missing comma, "...suburbs of London, England[,] in the American Midwest."
Paragraph 35: "...digital mapping projects, as Robertson points out in his essay..." -- need the full name here?
Paragraph 37: "As antislavery women moved from place to place both within a city or region, as was pointed out by Chambers in her book on the Weston sisters, but also as they moved around the Atlantic world, their correspondents shifted." -- an awkward sentence
Before I dive in, it looks like some of the lesson has been reposted under the Summary and Next Steps -- Paragraphs 40-61. Via a quick scan, I think it might be just a pasting error; they seem to be copies of the same paragraphs above, but wanted to double check.
Yes, it was a copy and paste error. It is fixed. Sorry about that!
Thanks Erin and Stephanie! And thanks Jason for your review.
Stephanie, let's wait until after the second review is complete to discuss next steps.
Sent from my iPhone
On Dec 18, 2016, at 10:43 AM, Stephanie Richmond notifications@github.com wrote:
Yes, it was a copy and paste error. It is fixed. Sorry about that!
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.
Hey folks -- apologies for the lengthy silence on this. I'm on a Windows 10 machine and I'm having some trouble installing cURL, so I haven't made it very far through the tutorial. None of the packages I've installed from https://curl.haxx.se/download.html have an actual exe included, so I'm taking to the web to see if I can find a solution.
In the meantime, it is occurring to me that the latest versions of Powershell contain a cmdlet called Invoke-WebRequest that seems to dupe some cURL capabilities. Is this a viable alternative for Windows users? There's a great Powershell tutorial already on ProgHist if it is.
Yes, you can certainly use Invoke-WebRequest to request a DPLA API key. We reference cURL because that is the tool specifically mentioned by DPLA, but any tool that can make a POST request will work.
Hi everybody!Thanks for your patience as I work through this. I did attempt to install every single Windows version of Curl from the Download Wizard to no avail. It looks like there is a version of Curl that runs on Windows10 (available here: [http://www.curl.com/release/2015/20151113.html]. I've downloaded and installed it and I get a Curl control panel. Progress.
However, at this point, I cannot get past the Get an API Key section of the tutorial because my Curl experience is entirely different than the instructions that you have here. I'm a true API/Curl beginner, so potentially someone with more experience with it could understand how to manage the control panel and proceed with your instructions. At this point, I'm afraid I'm a bit useless. I don't want to prolong this needlessly.
My recommendation is that at the very least we should note that the Windows 10 experience is different. Additionally, we can move on to another reviewer--and I absolutely would not be offended if that was the decision--or we can pause and figure out how to get the API key from this Curl control panel and I'll happily pick it up from there.
@wcaleb thoughts? How should we proceed?
Sorry that you're having trouble with cURL, @erinbush! I think for now what I will do is send you an API key that I generated myself using cURL on a Mac. That should allow you to attempt to proceed with the JavaScript parts of the lesson. (Sidenote: do any of you, Jason or authors, have a sense of whether those portions---like installing Node---will pose problems for a PC user, too?)
We can return to the cURL issue later when revising the lesson. One of the issues to decide, as I see it, is whether the main thing you want to teach here is how to get data from the DPLA using its API, or how to clean the data once received and generate a network map from it. If the latter, then I think Jason is right that you'll need to spend more time opening the "black box" of how the map gets made. But perhaps to make room for that, you could eliminate the parts of the lesson where the reader generates the CSV file (or at least make that portion "optional" for beginning users by providing them with a sample CSV file that has already been cleaned and prepared in the way described in the lesson).
The technical hurdles that Erin is running into may also speak to the question I and Jason have raised above about whether this is a "beginner" or "intermediate" lesson. Erin, we appreciate your helpful testing on all of this! If getting through the rest of the lesson proves too difficult with the level of experience you have (as described to me in a separate correspondence), then at that point we may want to pause and consider next steps again.
Thanks all for your help---let me know if anyone has questions or comments at this juncture.
@wcaleb I can't say what caused the issue in this specific case, but in general there shouldn't be any issue with running this in Windows 10. All the software, including Node, is available. We tested it on our end to be sure. cURL is not the most user friendly of programs and it can be confusing to find the right version on their website.
We could offer an alternative suggestion to the DPLA method of requesting a key. For instance the Postman extension for Chrome or RestClient for Firefox can be used to make a POST request using a visual interface. There might be another more universal, cross platform program we could suggest or one of the websites that offers a form such as Hurl.it.
We do offer a sample file in the repo that should let readers get the map running with limited data. We could highlight this more clearly in the lesson so that readers can start exploring the tool quicker.
Thanks @wcaleb and @ttavenner. The API key helped! I do think including an alternative method of obtaining the key would be very helpful. @hepplerj was quite thorough in his review and instead of just echoing him, I am going to add just a few more comments from the perspective of a more advanced beginner.
As a visual learner, I also recommend screenshots to help those who may not have a visual in mind when they begin. A view of the finished product and any possible steps along the way could help with navigating through the lesson. I did successfully make a map, but I have no idea if it's the map that you intended.
The explanation here on how to parse and clean the data, how to gather and organize the needed coordinates, and modify the file (with caveats about Excel) are all truly valuable, especially for those embarking on their own projects. That said, it's not always clear which are explicit instructions for this lesson and which are just general "if this is your own project" type of information. I think the simple fix is to clarify what you mean when you say "this data" and "your data" which could help stem any possible confusion.
Similarly, in the section on Setting up the Map, it's not clear which files and which folders you are discussing. I recommend explicitly stating which files to move via FTP and which folders contain the project for those using a local server.
Lastly, I will admit that I did nothing with the vector tiles in the Configuring the Map section. As written, there are no explicit steps or instructions. If the intent was to share an awareness that you can alter the look of the map, then no changes are necessary. An advanced beginner may not know what to do at this stage. Are there very basic configuration changes you can show here to help a user practice these skills? Or is there another existing lesson to link to?
I also want to echo that the Analysis section is quite useful and I'm happy to see it included here. Too often we discuss the making of "stuff" but don't spend time discussing what to do with it once it's built. Thank you for including that.
Thank you @hepplerj and @erinbush for your thoughtful reviews of this lesson. At this stage, I'd like to invite @sjrichmond @ttavenner to respond with your thoughts on the reviews and your plans for any revisions.
Like the reviewers, I believe this lesson will be highly valuable to Programming Historian readers. But my sense is that it needs some paring down in some places and some expansion in others. To me the core of this lesson is how historians who work with correspondence might create a network map from a CSV file of the letters and why they might want to. I'd like to propose the following and see what the authors think:
In addition to that overall revision plan, both reviewers asked for some screenshots, which I think would help, too. And I like Jason's suggestion of pointing the users to some other platforms like Carto and Palladio into which they could conceivably plug a CSV file of letters. Doing that would once again keep the focus of the lesson on the analytical upshot of a map and on the preparation of geospatial correspondence data, rather than on this particular Javascript implementation of a map.
Those, at least, are the suggestions that I came up with after reading through the reviewers, but I am eager to hear from you both (and the reviewers too if you want to chime back in) about what you think. Please feel free to make different suggestions or raise questions about what I've said if you think the lesson would be better for it.
Thank you all for your feedback, they are really helpful. Thank you @hepplerj for the detailed line edits. @erinbush I'm sorry you had so much trouble with the parser, thank you for being so determined!
After discussing all of the comments, we have determined that this is probably an intermediate level tutorial. We will remove a lot of the basic discussions of javascript and the section on parsing data and replace that section with a shorter description of how we got our data. We can certainly leave the javascript parser in the files and put a README in the folder with explaining how to use it if @wcaleb thinks that would be useful for PH readers. If not, we are fine with removing it entirely.
More specific revision plans:
We will expand the section on how the code works and how the map is constructed and add examples of how the code can be altered to fit a reader's own data and project needs. We will also add some still photos using the sample data that show what each part of the code does. We can also, if it is okay with the PH editors, add a link to our project which will allow users to play with the fully interactive map with a decent amount of data loaded into it.
In order to both explain how the code works and make sure it is clear how to get the minimal working model up and running, we will separate the "get the software up and running" part of the directions from the description of how it all works and how to alter the code to suit your project, with the get it working directions first.
We will address the question of why you might want to use your own script rather than Carto or Palladio in more detail. The rather basic answer to why I didn't use either myself when I started this project was that Palladio wasn't out yet when I started working on this and then it wasn't stable enough (I tried loading my data in and it would just crash my browser). Carto and ArcGIS had really steep learning curves and were too expensive for using for a long term project.
Here is our plan for how the lesson will be laid out to accomplish the goals of getting a minimal working example, explaining how the script works and what to change to tailor it to your own project, and keep the analytical section.
Introduction Lesson Goals Set Up Project Structure Collecting Data (compressed) Cleaning Data and Collecting Coordinates Setting Up the Map (expanded with screenshots) Customizing the Map (expanded with explanations of how the script works) Analysis Summary and Next Steps
Thanks for the update @sjrichmond. I think your plan sounds great, so go ahead and proceed with making your revisions directly to the file in this repository.
Re: whether to include the parser, I think putting a link to your other repo in a footnote would be appropriate.
As a general rule, though, we'd prefer to keep any code that is essential to the completion of the lesson (which in this case wouldn't include the parser) bundled together within the Programming Historian site, either as codeblocks in the lesson or (in the case of longer code bases) in our assets
folder. If in your revision you decide that even some things essential to the lesson must remain in your other repo, let's talk about that. But as much as possible, we'd like Programming Historian to be a standalone resource.
@wcaleb We've made our revisions and updated the lesson file here. We also made some changes to the scripts (found a bug) in our repo, so I can re-send or you can download the updated scripts from there.
Thanks, @sjrichmond. I will try to take a look at the revisions later this week and get back to you ASAP.
@sjrichmond @ttavenner Please accept my sincere apologies that it has taken me so long to turn back to your lesson. I really like the changes you have made and believe that we should be able to move this to publication soon.
I have just made a pull request at #65 that includes a variety of small suggested changes: the most substantial of them have to do with suggested links to other Programming Historian lessons. Please take a look at the pull request and let me know if there are any changes I've made that you disagree with. (You can click on individual commits to get a clearer picture of what changed: I tried to make the commit messages very descriptive, too.)
If those look good, I will merge the pull request so that we can get an updated "live preview" with paragraph numbers. At that point, I will mention a few other (five or six) questions about minor things that I didn't want to just change myself.
After all that, there are two other things we will need to adjust: first, we would prefer that the files used for this lesson be placed into the assets
folder as discussed above. It may make sense to create a "zip file" with all of the files in your other repo that we can store on our site. That would make it easy to ensure that the directory structure is preserved when the user downloads the file.
Second, you will need to adjust the way you have named your screenshot images and also follow the formatting instructions for figures as described in our style guide.
Once we've done these things, and assuming you don't have other changes you want to make or questions to raise, I'll ping our reviewers one more time to glance at the live preview and see if they notice any final problems. But I think you've done a great job addressing all the previous concerns, so hopefully that second review will be minimal and mainly focused on catching any small errors that the three of us have missed.
Hi Caleb, @ttavenner and I reviewed the suggested changes and they are all fine with us. I have fixed the image names and linked them according to the style guide and uploaded the zip file. I don't think I see any other changes or questions. Thanks!
Thanks @sjrichmond and @ttavenner! Now that those changes have been made, I see just a few more things to adjust / suggestions to make. I'll be using the "paragraph numbers" displayed in the live preview to reference specific places below.
In paragraph 7: You'll need to adjust the link and paragraph to instruct readers to download the "zip file" that @ttavenner provided.
I've recently become aware that the BPL is beginning a transcription project with the Anti-Slavery Collection and so has images of the letters hosted (in addition to IA and DPLA) at Digital Commonwealth. Would it be worth adding the link to that version of the collection either in paragraph 12 or footnote 3? (That interface also has a way to filter letters by place.)
In paragraph 43 and 44, could you make clearer where the reader would find the portion of map.js
where they would actually insert the code from the codeblock? Perhaps something like "Find the line that says makerlist
[or whatever it actually says in the code] and replace [x] with [y]."
After paragraph 5, when you discuss why you didn't use Palladio or another service, perhaps you could add a paragraph telling the reader why they might still find your lesson valuable even if they do decide to use one of those services? You could explain that even if they decide to go with one of those services, your experience with cleaning data and analyzing the map to draw historiographically significant questions might still be useful. (And you could link the text to those specific sections.) That would help readers see that one of the unique things about your lesson is the attention it pays to what someone might do with a map once it is made, either by Palladio or by the method you used.
@hepplerj @erinbush The lesson that you reviewed for Programming Historian is nearing publication! If you have time this week to take a quick look at the live preview for the revised lesson, you may be able to help us catch any errors that I and the authors may have missed in revision---and also call our attention to any parts of your reviews that we have missed.
Since we'd like to move this to publication soon (after a long delay in the process that was entirely my own fault), I'd be grateful if you could respond by the end of this week---even if only to say quickly that everything looks good or that you won't have time to look very closely. We will then be able to publish early next week!
I've made the four suggested changes and fixed one additional problem (footnote 3 was deleted in the editing and the remaining footnotes needed to be renumbered).
Just a note that I've looked everything over and it looks great. I have one minor correction: the authors state that Palladio "still requires that you load your data onto a server without much documentation about privacy." This isn't what's happening -- all of the work that Palladio is doing is done in browser memory. Nothing ever goes to a database.
Overall, however, great work! Excited to see it go live, and I love that this lesson not only talks about doing the technical work but also focuses on an analysis of the completed project.
@sjrichmond I've just created a pull request #68 with a few final suggested edits, one of them in response to @hepplerj comment above.
If you approve of those changes, I think we are ready to publish!
Also, let me know what you think of this image for the icon to associate with the lesson on our directory page. I was thinking that the "correspondence" part of the image was a nice touch.
I approve, and I like the image, it speaks well to the correspondence aspect and the internationalism of the project.
Great! One last thing I need is a brief bio for @sjrichmond and @ttavenner --- usually just a brief, one-sentence bio that will be used at the bottom of the lesson. You can put yours in a comment here and I'll get it transferred to the lesson.
Stephanie J. Richmond is an assistant professor of history at Norfolk State University in Virginia and a historian of gender and race in the Atlantic world. Tommy Tavenner is a developer and data architect working in open data and civic engagement.
Now published at http://programminghistorian.org/lessons/using-javascript-to-create-maps !
Thanks @sjrichmond @ttavenner @hepplerj @erinbush for all your hard work!
The Programming Historian has received the following tutorial on 'Using JavaScript to Create Maps of Correspondence' by Stephanie J. Richmond and @ttavenner. This lesson is now under review and can be read at:
http://programminghistorian.github.io/ph-submissions/lessons/using-javascript-to-create-maps
Please feel free to use the line numbers provided on the preview if that helps with anchoring your comments, although you can structure your review as you see fit.
I will act as editor for the review process. My role is to solicit two reviews from the community and to manage the discussions, which should be held here on this forum. I will also provide feedback in this space.
Members of the wider community are also invited to offer constructive feedback which should post to this message thread, but they are asked to first read our Reviewer Guidelines and to adhere to our anti-harassment policy (below). We ask that all reviews stop after the second formal review has been submitted so that the author can focus on any revisions. I will make an announcement on this thread when that has occurred.
I will endeavor to keep the conversation open here on Github. If anyone feels the need to discuss anything privately, you are welcome to email me. You can always turn to @ianmilligan1 or @amandavisconti if you feel there's a need for an ombudsperson to step in.
Anti-Harassment Policy
This is a statement of the Programming Historian's principles and sets expectations for the tone and style of all correspondence between reviewers, authors, editors, and contributors to our public forums.