Paper Proposal

[JonathanChiang]

Overview

The Course Project is an opportunity for you to apply what you have learned in class to a problem of your interest. Potential projects usually fall into these two tracks:

• [x] Applications. If you're coming to the class with a specific background and interests (e.g. biology, engineering, physics), we'd love to see you apply ConvNets to problems related to your particular domain of interest. Pick a real-world problem and apply ConvNets to solve it.

• CVPR: IEEE Conference on Computer Vision and Pattern Recognition

• ICCV: International Conference on Computer Vision

• ECCV: European Conference on Computer Vision

• NIPS: Neural Information Processing Systems

• ICLR: International Conference on Learning Representations

• ICML: International Conference on Machine Learning

• Publications from the Stanford Vision Lab

For applications, this type of projects would involve careful data preparation, an appropriate loss function, details of training and cross-validation and good test set evaluations and model comparisons. Don't be afraid to think outside of the box.

[JonathanChiang]

Project Proposal

The project proposal should be one paragraph (200-400 words).

• [x] What is the problem that you will be investigating? Why is it interesting? ` Translating neural networks from theory to clinical practice has unique challenges, specifically in the field of neuro-imaging. I am looking to expand the deep learning framework DeepNeuro, which applies deep learning algorithms for neuro-imaging with the goal to improve Physician Computer Interaction for Interpretability, IT implementation and ease of use. With the emphasis on clinical triaging

Deepneuro has proven to help design and train neural network architectures, as well as modify state-of-the-art architectures in a flexible and intuitive way.

Another exploratory application is apply federated or model serialization to ensure consistent performance of networks across variable users, institutions, and scanners across different institutions to promote interoperability and patient confidentiality.

This introduces an IT implementation and engineering solution that could enable training models across sites and institutions. This additionally introduces deep learning coding design patterns for security across institutions. `

• [x] What reading will you examine to provide context and background? `
• pysyft
• deepneuro
• federated learning
• model serialization (onnx)
• patient interoperability
• transfer learning
• health IT implementation
• airplane auto pilot practices
• physician CDSS and HCI `
• [ ] What data will you use? If you are collecting new data, how will you do it?
• pysft tutorial for model encryption
• mnist
• diabetic retinopathy
• ASK KEN CHANG AND ANDREW BEERS
• [x] What method or algorithm are you proposing? If there are existing implementations, will you use them and how? How do you plan to improve or modify such implementations? You don't have to have an exact answer at this point, but you should have a general sense of how you will approach the problem you are working on.
• docker solution
• across different IT
• clinical user interface
• expand on open source projects
• [x] How will you evaluate your results? Qualitatively, what kind of results do you expect (e.g. plots or figures)?
• results on distributed versus local
• [ ] Quantitatively, what kind of analysis will you use to evaluate and/or compare your results (e.g. what performance metrics or statistical tests)?
• ASK KEN AND ANDREW
• Statistical Analysis

operating characteristic curve (AUC) and precision, recall, and
F1 scores (these metrics are defined in detail in Appendix E1
[online]). These measurements were computed by using the
scikit-learn (version 0.19) Python library and were reported for
the test set in each model. Interrater agreement was assessed with
the Cohen k statistic and was also computed by using scikitlearn software. The DeLong nonparametric statistical test (24)
implemented in the Daim (version 1.1.0) R library was used to
assess statistical differences among AUC values, wherein P values
less than .05 were considered to indicate a significant difference.
The method described by Hanley and McNeil (25) was used to
compute 95% confidence intervals for AUC values.
To enable accessible analysis of CNN model results, we created class activation maps (CAMs) that show the areas of a given
image that are most responsible for its CNN classification (26).
The CAMs we used were slightly different from those used by
Zhou et al (26) because of the sigmoid nonlinearity on the final
layer```

[JonathanChiang]

Final Report:

Your final write-up is required to be between 6 - 8 pages using the provided template, structured like a paper from a computer vision conference (CVPR, ECCV, ICCV, etc.). Please use this template so we can fairly judge all student projects without worrying about altered font sizes, margins, etc. After the class, we will post all the final reports online so that you can read about each others' work. If you do not want your writeup to be posted online, then please let us know at least a week in advance of the final writeup submission deadline.

The following is a suggested structure for your report, as well as the rubric that we will follow when evaluating reports. You don't necessarily have to organize your report using these sections in this order, but that would likely be a good starting point for most projects.

` Title, Author(s)

Abstract:

• [ ] Briefly describe your problem, approach, and key results. Should be no more than 300 words. Introduction (10%): Describe the problem you are working on, why it's important, and an overview of your results

Related Work (10%):

• [ ] Discuss published work that relates to your project. How is your approach similar or different from others?

Data (10%):

• [ ] Describe the data you are working with for your project. What type of data is it? Where did it come from? How much data are you working with? Did you have to do any preprocessing, filtering, or other special treatment to use this data in your project?

Methods (30%):

• [ ] Discuss your approach for solving the problems that you set up in the introduction. Why is your approach the right thing to do? Did you consider alternative approaches? You should demonstrate that you have applied ideas and skills built up during the quarter to tackling your problem of choice. It may be helpful to include figures, diagrams, or tables to describe your method or compare it with other methods.

Experiments (30%):

• [ ] Discuss the experiments that you performed to demonstrate that your approach solves the problem. The exact experiments will vary depending on the project, but you might compare with previously published methods, perform an ablation study to determine the impact of various components of your system, experiment with different hyperparameters or architectural choices, use visualization techniques to gain insight into how your model works, discuss common failure modes of your model, etc. You should include graphs, tables, or other figures to illustrate your experimental results.

Conclusion (5%)

• [ ] Summarize your key results - what have you learned? Suggest ideas for future extensions or new applications of your ideas.

Writing / Formatting (5%)

• [ ] Is your paper clearly written and nicely formatted? Supplementary Material, not counted toward your 6-8 page limit and submitted as a separate file. Your supplementary material might include:

Source code (if your project proposed an algorithm, or code that is relevant and important for your project.)

Cool videos, interactive visualizations, demos, etc. Examples of things to not put in your supplementary material:

[JonathanChiang]

https://jdunnmon.github.io/dunnmon_radiology_2018.pdf

[JonathanChiang]

Development Size and Initialization Method Test Accuracy Precision Recall F1 Score AUC Value

[JonathanChiang]

Figure 3: Comparison of, A, receiver operating characteristic (ROC) curves for DenseNet-121 (NN) and NN+PL (mean of NN score and prospective label [PL] score) classifiers and, B, area under the ROC curve (AUC) histograms obtained from a 1000-sample test set by using the bootstrap method. Each ROC curve represents the output of one representative NN model. In B, solid lines indicate mean values, and dashed lines indicate standard deviation from the mean. Data set size (K = 1000 points) refers to total size (training + development, 90-to-10 split).

[JonathanChiang]

Deep Learning for Triage of Chest Radiographs: Should Every Institution Train Its Own System?

[JonathanChiang]

Pysyft Scope: Note that we'll not just be talking about how to decentralized / encrypt data, but we'll be addressing how PySyft can be used to help decentralize the entire ecosystem around data, even including the Databases where data is stored and queried, and the neural models which are used to extract information from data. As new extensions to PySyft are created, these notebooks will be extended with new tutorials to explain the new functionality.

[JonathanChiang]

https://arxiv.org/pdf/1811.04017.pdf!

We have introduced a privacy preserving federated learning framework built over PyTorch. The
design relies on chains of tensors that are exchanged between local and remote workers. Our tensor
implementations support commands of the PyTorch API and combine MPC and DP functionalities
within the same framework

[JonathanChiang]

SyftTensors are meant to represent a state or transformation of the data and can be chained together. The chain structure always has at its head the PyTorch tensor, and the transformations or states embodied by the SyftTensors are accessed downward using the child attribute and upward using the parent attribute

[JonathanChiang]

Our main contributions are the following:
- We first build a standardized protocol to communicate between workers which made federated
learning possible.
- Then, we develop a chain abstraction model on tensors to efficiently override operations (or encode
new ones) such as sending/sharing a tensor between workers.
- Last, we provide the elements to implement recently proposed differential privacy and multiparty
computation protocols using this new framework.
By doing so, we intend to help popularize privacy preserving techniques in machine learning by
making them available via the common tools that researchers and data scientists work with on a daily
basis. Our framework is designed in a extensible way such that new FL, MPC, or DP methods can be
plugged in by external contributors willing to make their work available to the wider deep learning
community.

[JonathanChiang]

https://blog.openmined.org/upgrade-to-federated-learning-in-10-lines/

[JonathanChiang]

We detail a new framework for privacy preserving deep learning and discuss its assets. The framework puts a premium on ownership and secure processing of data and introduces a valuable representation based on chains of commands and tensors. This abstraction allows one to implement complex privacy preserving constructs such as Federated Learning, Secure Multiparty Computation, and Differential Privacy while still exposing a familiar deep learning API to the end-user. We report early results on the Boston Housing and Pima Indian Diabetes datasets. While the privacy features apart from Differential Privacy do not impact the prediction accuracy, the current implementation of the framework introduces a significant overhead in performance, which will be addressed at a later stage of the development. We believe this work is an important milestone introducing the first reliable, general framework for privacy preserving deep learning.