search

programminghistorian / ph-submissions

The repository and website hosting the peer review process for new Programming Historian lessons
http://programminghistorian.github.io/ph-submissions
139 stars 114 forks source link

Lesson proposal: Creating Deep Neural Networks for Regression Analysis Proposal (PH/JISC/TNA) #414

Closed tiagosousagarcia closed 1 year ago

tiagosousagarcia commented 3 years ago

The Programming Historian has received the following proposal for a lesson on 'Creating Deep Neural Networks for Regression Analysis Proposal' by @nabsiddiqui. The proposed learning outcomes of the lesson are:

In order to promote speedy publication of this important topic, we have agreed to a submission date of no later than 24/01/2022. The author(s) agree to contact the editor in advance if they need to revise the deadline.

If the lesson is not submitted by 24/01/2022, the editor will attempt to contact the author(s). If they do not receive an update, this ticket will be closed. The ticket can be reopened at a future date at the request of the author(s).

The main editorial contact for this lesson is @tiagosousagarcia.

Our dedicated Ombudsperson is (Ian Milligan - http://programminghistorian.org/en/project-team). Please feel free to contact him at any time if you have concerns that you would like addressed by an impartial observer. Contacting the ombudsperson will have no impact on the outcome of any peer review.

drjwbaker commented 2 years ago

@svmelton and I discussed potential editors for this article. Sarah will confirm if the preferred editor has capacity.

svmelton commented 2 years ago

@scottkleinman will serve as the editor for this piece.

anisa-hawes commented 2 years ago

(I am co-assigning myself here so that I can shadow the editorial process).

drjwbaker commented 2 years ago

@scottkleinman so pleased to have you as editor on this. Note that this article is part of a special series for which we have funding. As a rresult @tiagosousagarcia and I will be offering additional support. For example, Tiago will do the technical edit and has identified potential peer reviewers. So do write to us https://programminghistorian.org/en/project-team when you are ready!

tiagosousagarcia commented 2 years ago

This lesson has now been submitted and staged here: https://programminghistorian.github.io/ph-submissions/en/drafts/originals/image-classification-neural-networks, and I will be doing a technical review shortly. Many thanks to @nabsiddiqui for submitting on time, and my renewed apologies for the delay in getting this up.

tiagosousagarcia commented 2 years ago

@nabsiddiqui, congratulations on a brilliant tutorial -- I think this is something that will be of interest to many PH readers, and it offers an excellent explanation of some core Machine Learning Concepts.

Apologies for the delay in moving your article forward -- as you know, there were a few personal reasons that kept me away from this for a while.

I've made an initial technical review 78f5984, making small changes as I went along. A summary of these are:

Changes made

Additionally, I would also suggest you consider the following before we send your article for peer-review:

Suggestions

Once again, my apologies for the delay in getting this staged and reviewed. lf my suggestions make sense and you can implement them within a couple of weeks, @scottkleinman and I can start contacting potential peer-reviewers.

Thank you again for submitting!

scottkleinman commented 2 years ago

So happy to be on board! Thanks, @drjwbaker and @tiagosousagarcia for the extra support. I'll have some first feedback towards the end of the week.

nabsiddiqui commented 2 years ago

Thank you @tiagosousagarcia. I will go ahead and start making changes when I receive @scottkleinman feedback.

scottkleinman commented 2 years ago

I've made some small editorial changes summarised below. I'll get my larger suggestions up in a separate post as soon as I can.

scottkleinman commented 2 years ago

@nabsiddiqui, this was a wonderful tutorial, and I really learnt a lot (despite the fact that the resulting model thinks a picture of me and my wife is a "boat"!). I second all of @tiagosousagarcia's suggestions, although I think it might be a good idea to add a new "Setup" section in which you instruct the user to create something like a project folder, copy in all the necessary files, and start a server running in that folder. Then there would be no need to interrupt the main flow of the discussion with such details.

Here are some other suggestions:

let confidence = " (" + (results[0].confidence * 100).toFixed(2) + "%)";
let label = results[0].label;
text("Most Likely " + label + confidence, width/2 , height/2+200);

You could make this code more elegant, but the basic idea is to output the confidence score to the screen as well as to the console.

Hopefully, these are things you can implement without too much extra work. If you think it might take more than a couple of weeks, I can let you know what I think would be the top priorities.

nabsiddiqui commented 2 years ago

Thank you @scottkleinman and @tiagosousagarcia. Do we have a strict deadline on when this needs to be completed? My March is shaping up to be very busy, but I have a much more relaxed April. I will try to work on it in March when possible, but it may be April when it gets done. I hope this is ok.

tiagosousagarcia commented 2 years ago

Hi @nabsiddiqui -- I don't think we have a hard set deadline for this, though we will hold off on peer-review. If we could get them by early April, that would be great. How does that sound?

scottkleinman commented 2 years ago

Yes, that timetable is fine for me as well. Let us know if you have any questions.

tiagosousagarcia commented 2 years ago

Dear @nabsiddiqui and @scottkleinman -- @drjwbaker and I were just talking, and we came to the conclusion that, to make sure we keep this tutorial moving swiftly, it might be helpful to have a more concrete deadline for these corrections. With that in mind, would it be possible to to have them by April 6?

nabsiddiqui commented 2 years ago

@tiagosousagarcia Yes that should be doable.

scottkleinman commented 2 years ago

@nabsiddiqui, In reviewing the submission, I put together a set of instructions for making clear how to use Teachable Machine; however, I then lost it when posting my suggestions above. I've found it again, so I am pasting it here in the hope that it's of some use to you.

Procedure for working with Teachable Machine:

  1. Click "Get Started".
  2. Click "Image Project".
  3. Click "Standard image model".
  4. Under "Class 1", click "Choose images from your files, or drag & drop here".
  5. Select the "aircraft" folder from the dataset and drag it into the Teachable Machine window.
  6. Click the pencil icon next to "Class 1" and change the name to "aircraft".
  7. Repeat this process for the other folders in the dataset. After the second time, you will need to click "+ Add a class" for each new folder.

Once the model is trained, Teachable Machine will display your webcam and continuously predict the likelihood that what it sees belongs to any of the classes the model has been trained on. If you don't like being told how much you look like a boat, turn this off by clicking the "Input" toggle switch.

To export the model, click "Export Model". In the "TensorFlow.js" tab, click the "Download" radio button. Then click "Download my model".

nabsiddiqui commented 2 years ago

Thank you, @scottkleinman. I will keep these in mind as I revise.

nabsiddiqui commented 2 years ago

Hey @tiagosousagarcia and @scottkleinman. I have updated the lesson based on your requests. Let me know if you need anything else from me.

tiagosousagarcia commented 2 years ago

Many thanks @nabsiddiqui! @scottkleinman, once you're happy, we can move it to peer-review

scottkleinman commented 2 years ago

I've made a few more copy edits, but, otherwise, I am happy to move this to peer review.

tiagosousagarcia commented 2 years ago

@melvinwevers has kindly agreed to be one of the reviewers for this tutorial. We can expect his review by 17th of June. Many thanks!

tiagosousagarcia commented 2 years ago

@zentralwerkstatt has kindly agreed to review this tutorial. We can expect his review by the 30th of June. Many thanks!

scottkleinman commented 2 years ago

The Programming Historian has received the following tutorial on 'Creating Deep Neural Networks for Regression Analysis' by @nabsiddiqui. This lesson is now under review and can be read at:

https://programminghistorian.github.io/ph-submissions/en/drafts/originals/image-classification-neural-networks

Please feel free to use the line numbers provided on the preview if that helps with anchoring your comments, although you can structure your review as you see fit.

I will act as editor for the review process (with help from @tiagosousagarcia and @drjwbaker). My role is to solicit two reviews from the community and to manage the discussions, which should be held here on this forum. I have already read through the lesson and provided feedback, to which the author has responded.

Members of the wider community are also invited to offer constructive feedback which should post to this message thread, but they are asked to first read our Reviewer Guidelines and to adhere to our anti-harassment policy (below). We ask that all reviews stop after the second formal review has been submitted so that the author can focus on any revisions. I will make an announcement on this thread when that has occurred.

I will endeavour to keep the conversation open here on Github. If anyone feels the need to discuss anything privately, you are welcome to email me.

Our dedicated Ombudsperson is (Ian Milligan - http://programminghistorian.org/en/project-team). Please feel free to contact him at any time if you have concerns that you would like addressed by an impartial observer. Contacting the ombudsperson will have no impact on the outcome of any peer review.

Anti-Harassment Policy

This is a statement of the Programming Historian's principles and sets expectations for the tone and style of all correspondence between reviewers, authors, editors, and contributors to our public forums.

The Programming Historian is dedicated to providing an open scholarly environment that offers community participants the freedom to thoroughly scrutinize ideas, to ask questions, make suggestions, or to requests for clarification, but also provides a harassment-free space for all contributors to the project, regardless of gender, gender identity and expression, sexual orientation, disability, physical appearance, body size, race, age or religion, or technical experience. We do not tolerate harassment or ad hominem attacks of community participants in any form. Participants violating these rules may be expelled from the community at the discretion of the editorial board. Thank you for helping us to create a safe space.

melvinwevers commented 2 years ago

First of all, I welcome tutorials on image analysis, especially those aimed at (art) historians. Some disclosure is at place here. I am a co-author on a lesson that has been submitted to Programming Historian on neural networks for image analysis. I believe this lesson is still under review. Especially in terms of theory there's quite some overlap.

It might be a good idea to see where the overlap exists and see if these lessons can be bundled.

In what follows, I have noted some general and specific points of feedback. Overall, I think the article needs to be better aimed at historians and focused on transferring re-usable skills. Too much time is spent on JavaScript and presenting the results in a browser. I believe a python based approach using notebooks or Google Colab (with GPU access) would be much more efficient in conveying how neural networks coudl be used.

General and Major Comments

Minor comments

Typos

Direchlet -> Dirichlet Paramaters > Parameters outptuts > outputs

zentralwerkstatt commented 2 years ago

Major

Framing: I appreciate the framing of technical knowledge as a prerequisite for the critical analysis of machine learning systems. This perspective could be even further emphasized, for instance by pointing out potential "injection points" for bias in the technical part.

Structure: This is an inspiring tutorial that pays excellent attention to detail without losing focus on its larger pedagogical goals. It is well written, clearly structured, and proceeds nicely from a theoretical reflection of machine learning principles to questions of implementation.

Implementation: As @melvinwevers has pointed out, the tutorial builds upon a specific toolchain that includes, on the one hand, a proprietary Web application by Google, and on the other, a set of open-source JavaScript libraries. I certainly see the pedagogical value of the Teachable Machine which beautifully demonstrates the power of transfer learning. Still, I feel that the tutorial spends way too much time on "shallow" technical knowledge, like setting up a highly idiosyncratic chain of JavaScript libraries (tensorflow.js + P5.js + ML5.js). At times, the tutorial reads like a P5.js tutorial focused on machine learning rather than a machine learning tutorial using P5.js. Today, other than in the early days of machine learning, the (open-source) toolchain for both academic and commercial machine learning work is pretty much agreed upon, with only very few real "choices" (e.g. PyTorch or JAX) remaining. Any deviation from this toolchain comes with a cost that, in my opinion, needs to be well-justified. Especially as the tutorial is aimed at people without much technical knowledge, the existence of documentation and support beyond the tutorial is vital. This documentation and support is simply not given for the chosen toolchain, beyond very simple example applications (i.e. more of the same). All that being said, I am aware that the tutorial is built around this specific toolchain which makes it difficult to argue for a fundamental revision. A potential compromise would be to a) discuss the "industry standard" somewhere in the tutorial and focus on the pedagogical benefits of the Teachable Machine (i.e. focus more on demonstrations of the theoretical concepts discussed in the tutorial, rather than creating a "reusable" model), or b) supply additional technical content in the form of Python code contained in Jupiter/CoLab notebooks. Again: I am not arguing for a fundamental revision of the tutorial. Rather, as the tutorial is aimed at beginners, I am arguing for a clear delineation between exclusively pedagogical and actually reusable and easily extendable content.

Minor (paragraph-specific)

3: This tutorial only discusses CNNs, which should be made clear at the beginning, especially due to the current omnipresence of transformers.

11: I very much appreciate the comparison with iterative programming. This could be extended to other parts of the tutorial, as readers might have some experience with iterative programming even if they are new to machine learning.

18f: The difference between biological and artificial neurons should be made even more explicit. It would also be advisable to briefly discuss the "spatial" language used throughout the tutorial (neurons, layers, networks, etc.) vs. the underlying mathematics.

20: Should mention some examples of "mathematical properties" (functions?) applied to the dataset.

21: Neural networks famously do not neatly disentangle features in a hierarchical manner but rather develop quite idiosyncratic ways of seeing. While this is not necessarily relevant to the audience of this tutorial, it should be mentioned if the tutorial's goal is indeed to support the critique, and not only the application of, machine learning methods.

37: To again echo @melvinwevers' review, the Teachable Machine is indeed intentionally designed to overfit to the user-supplied data, this should be made explicit.

62: I think the "training image" should be "testing image"? In general, the tutorial would benefit from a brief discussion of training vs. testing, and the importance to keep training, validation, and testing data separate.

scottkleinman commented 2 years ago

Thanks to our two reviewers, @melvinwevers and @zentralwerkstatt. There are a lot of good suggestions there. @nabsiddiqui, I'll be taking stock over the weekend, and I'll try to get you some next steps in the next couple of days.

scottkleinman commented 2 years ago

After re-reading the tutorial and our reviewers' thoughts, I think the biggest issue is the choice of Javascript as the language for exploring the model. This has implications for both the audience who will be best served by the tutorial and the tutorial's wider applicability for scholarly work using image classification. My sense is that there is a place for image classification in the browser and that the use of P5js and ML5js operate as abstractions for the tutorial's purpose so that the user does not have to begin by learning TensorFlow.js. This points to an entry-level audience: someone who not only wants to understand how image classification works but who is also interested in the possibilities of dissemination or interaction on the web. However, as the reviewers point out, the discussion of P5js and ML5js makes it seem like the focus of the tutorial is on those libraries and maybe even gives a misleading impression of their use in the field.

So I have the following suggestion. Why not make the entire project library (index.html and sketch.js) downloadable, along with the training data, and instruct the user to deposit their exported model file into the project folder. There is then no need to go into the ins and outs of the Javascript libraries and the various functions in the main body of the tutorial. This is not to say that this discussion should be removed, as it is still valuable. I am proposing that it go in an appendix, where anyone interested in, say, customising the output can gain a better understanding of what is going on.

This would leave more room for some discussion of the current environment for image classification. In place of the Javascript explanation, you could provide one or two paragraphs that (a) indicate the relationship between CNN and transformer models, and (b) give a sense of the environment for image classification by giving a sense of what might lead the user to explore PyTorch or TensorFlow. I think some of that material is in the bibliography, but it may need some more dedicated discussion in the main body of the tutorial. This is what I think Fabian means by discussing the "industry standard" and the pedagogical benefits of the Teachable Machine. That's only a partial fix of the sustainability issue surrounding Teachable Machine, but I think it gives the user enough to go on if the tool goes away.

One alternative (or additional) approach might be to place a "What next?" discussion in the conclusion. Here might be a reasonable place to reference Melvin's tutorial, which would be a nice place to go next. Even if this tutorial is published first, you could always go back and add a reference.

I also want to address a few more questions about the audience. Like Melvin, I wondered about the engagement with activists. I would drop this or else place it in the context of a more expanded discussion of the ethics of image classification. How do this without derailing the entire tutorial? The point Fabian makes about Teachable Machine being designed to overfit may be the way to approach this. Perhaps raising this in the conclusion would be a good way to return to the comments about ethics at the beginning of the tutorial.

Both reviewers recommend including Python code as a way to make the tutorial more applicable, but I think this would require too thorough a revision of its scope. Perhaps moving some of the links in the bibliography to the main body (as well as links to other PH tutorials) would be sufficient. I am on the fence about Melvin's suggestion to make "a tutorial specific for historians [by] incorporating a relevant research question, actual data, and showing how to analyse the results." I think this would be useful for fostering engagement with the ArtUK dataset, but it might stretch the tutorial beyond acceptable limits. Perhaps there is some scope in one of the suggestions made in paragraph 10? For instance, if there are untagged images in the collection, you could illustrate how well the model tags them. But, without knowing the data, I hesitate to send you down this path.

At this stage, I don't want to address the reviewers' more minor suggestions, as I think you will incorporate them as you go about revision. One thing that I think should be done is that CNNs should somehow be incorporated into the title to indicate that this is the methodology the tutorial employs.

Whilst I think everyone is trying not to propose changes that are too extensive, this already seems like a lot. So I am going to stop here to see whether others have feedback and to ask @nabsiddiqui whether any of this is valuable going forward.

nabsiddiqui commented 2 years ago

@scottkleinman, @zentralwerkstatt , @melvinwevers

Thank you for the comments and the kind words about the tutorial. They are really helpful, and I am happy to incorporate your suggestions into the lesson to the best of my ability. Here are some responses to the initial review and my general comments. My current goal is to have this done by lets say end of September?

General Comments

Response to @melvinwevers

Response to @zentralwerkstatt

Response to @scottkleinman

nabsiddiqui commented 2 years ago

@tiagosousagarcia and @scottkleinman I have updated the tutorial now with the revisions that were requested. Please let me know what else you need on my end or any comments you believe I have not addressed.

scottkleinman commented 2 years ago

Thanks, @nabsiddiqui. I should be able to get to this by Monday and will update this issue then.

nabsiddiqui commented 2 years ago

No worries @scottkleinman. Take your time.

tiagosousagarcia commented 2 years ago

Hello all -- just a quick note to say that this is my last week working for PH. It's been an absolute pleasure working on this, and I'm only sorry I'm not going to be around for its publication (from this side -- I'll definitely be reading and using it as a regular joe). Big thanks to @nabsiddiqui for writing it, and @scottkleinman for taking it forward. Well done everyone!

scottkleinman commented 2 years ago

@tiagosousagarcia, Thanks for all your work in helping me to hit the ground running.

scottkleinman commented 2 years ago

@nabsiddiqui, I have some quick questions about a couple of paragraphs in the tutorial.

nabsiddiqui commented 2 years ago

Hey @scottkleinman,

Thank you for taking the time to make the copy edits. For paragraph 26, things look fine. I have gone through paragraph 88 and updated the language accordingly. The change to projects is fine. Let me know if you need anything else on my end.

@tiagosousagarcia. Thank you for all your work on this. Look forward to when our paths cross again.

scottkleinman commented 2 years ago

@nabsiddiqui, Sorry to throw this back at you, but I've found an odd discrepancy when running a final test on your code. When I use angeldemon230.jpg as my test image on Teachable Machine, I get a 92% chance that it falls into the housing category (which is wrong, but that may be just because the model is not trained well). When I export the model and use the same image in the Javascript, I get a 92% chance that it is classified as horseracing. Can you explain this inconsistency?

nabsiddiqui commented 2 years ago

@scottkleinman The reason this is happening is because the model is not trained that well. When you run the image on Teachable Machine, it tries to classify the image through the neural network. If you run it again, it will likely not get the same result since it is running the calculation again. There are simply not enough images or training data to make it consistent. So, this is actually working the way it should.

scottkleinman commented 2 years ago

OK, I was surprised that the 92% was consistent. Maybe you should add a note about the likely performance on the test data, perhaps around paragraph 44.

I am going to test the code one last time and suggest some final edits, and then I think we are ready to move forward.

nabsiddiqui commented 2 years ago

Alright I will go ahead and add that along with any final edits you suggest.

scottkleinman commented 2 years ago

OK, here are some minor edits:

The more substantial issue is that I struggled with the Javascript when I had to run a function and then replace it with a modified function or add new functions. I couldn't always figure out what the code was supposed to be when I saved the file at each stage. As a result, I couldn't, for instance, reproduce the output shown in paragraph 81 -- even though I could reproduce the final output. This leads me to believe that it really would be worth it to provide the final form of the sketch.js file at the very end so that, if the user gets lost in the individual steps, they can see more where they got lost.

scottkleinman commented 2 years ago

@nabsiddiqui, my post above should be the final edits before we move the tutorial to the next stage (unless we need to supply some last-minute links to the forthcoming image recognition tutorials).

Can you give me a timetable for getting these final revisions done?

nabsiddiqui commented 2 years ago

Hey @scottkleinman. Sorry these have been on my todo list for a while, but I haven't gotten around to them. Would end of next week be ok?

scottkleinman commented 2 years ago

Yes, that would be perfect.

nabsiddiqui commented 2 years ago

Hey @scottkleinman

Here are the edits I have made.

I think that covers everything but let me know if I missed something.

scottkleinman commented 2 years ago

Thanks, @nabsiddiqui. I have added the link, and I think we're ready to move to the next stage, with thanks to our two reviewers. I'll try to get to this over the weekend.

scottkleinman commented 2 years ago

Metadata Summary

nabsiddiqui commented 2 years ago

I the author hereby grant a non-exclusive license to ProgHist Ltd to allow The Programming Historian English|en to publish the tutorial in this Issue (including abstract, tables, figures, data, and supplemental material) under a CC-BY license.

anisa-hawes commented 2 years ago

Thank you, @scottkleinman !

I've updated the YAML header with a few tweaks. I've also added in an alt-text field for the lesson image you've chosen. I had a look at the image, but I don't know what it depicts ! Can you add a short description into avatar_alt:?

The next step of the workflow will copyediting, followed by typesetting and replacing any external links to the live web with perma.cc archival links. I will post a comment here to update you and @nabsiddiqui when the copyedits are complete.

When the copyedits have been agreed, I'll send Nabeel an authorial copyright declaration form to sign (this form replaces the Permission to Publish statement we've used in the past).

I'll then tag Alex to let him know that he can move forwards with his final read-through and checks.

Because Nabeel is already a member of the Project Team, we don't need to add a new entry in the ph_authors.yml file this time, but I will ask Alex to add in the orcid: 0000-0002-6126-5833 which isn't included at the moment.

@nabsiddiqui, we stick to very simple and short (1-2 sentence) bios in ph_authors, and don't tend to include recent publications or details of current activities. Your bio currently reads: Nabeel Siddiqui is an Assistant Professor of Digital Media at Susquehanna University. Please let us know if you'd like us to adjust it.

scottkleinman commented 2 years ago

Thanks, @anisa-hawes. I have added the avatar_alt text, but feel free to change it to something that fits the usage elsewhere. You can see the original image and its metadata here.

I just copied Nabeel's bio from our proposal for the workshop and the DH conference, so, @nabsiddiqui, feel free to make any appropriate changes.

nabsiddiqui commented 1 year ago

Hey @scottkleinman everything looks good. Anything else I need to do on my end?