Paper Discussion 6b: Trustworthy Medical Device Software

[lrshpak]

• @bushidocodes Critical: Have medical devices undergone the same sort of move towards mandatory recurring subscriptions similar to John Deere tractors?
• @grahamschock : What causes the medical device industry to be so behind the ball on software safety and engineering? I thought that medical devices are heavily regulated by the FDA. Maybe there needs to not only be a culture shift in the device industry but also within the regulation authority.
• @AkinoriKahata Critical: Clarifying the responsibility and role is critical for the digital society. Because I’m not familiar with the medical sector, it is very helpful for me to clarify who are the stakeholders of medical sectors and what is the most severe problem the author expected.
• @rachellkm Critical: One of the proposed solutions was to specify “outcome measures” (section 4.1), but what exactly does “meaningful outcome measures” mean?
• @gkahl Comprehension: They briefly discussed how a lot of these systems are beginning to connect to the internet but never really discussed the wireless security of these systems. Isn't it a major risk if if someone can gain access to the devices distributing medicine/care to people?
• @s-hanna15 Critical: If the medical devices that go out are able to cause such catastrophic failure such as the Therac-25, what does the FDA actually do currently to ensure that devices are safe for use?
• @pcodes Critical: Do the authors have suggestions for how to improve interopability between devices? Requiring third-party review fixes the "poor testing" problem, but are there technical improvements that can be made to reduce compatibility problems?

[bushidocodes]

Reviewer: Sean McBride

Review Type: Critical Review

Problem Being Solved:

Provides a high-level overview of the unique attributes of software systems in a manner accessible to physicians, policymakers, and regulators.

Main Contributions:

1. Defines "trustworthy software"
2. Outlines the risks throughout the SLDC that impact the ability to create trustworthy software
3. Discusses controls for these risks.
4. Suggests policy changes that can help the FDA better review software-centric medical devices.

Questions:

1. Have medical devices undergone the same sort of move towards mandatory recurring subscriptions similar to John Deere tractors?
2. Windows has moved towards "OS as a Service." Are there any operating systems that actually support an operational life of 10-20 years for things like an MRI machine? RHEL only provides 10 years of LTS support.
3. How can a regulator balance risk controls over autonomic interventions with customers now being able to jailbreak their insulin pumps to install software like OpenAPS?
4. Does the characterization of avionics as the gold standard for this sort of development still make sense post 737 MAX?

Critiques:

1. Very anecdotal, but this is justified given the lack of data.
2. Implicitly assumes a traditional waterfall SDLC. Since 2011 (when this paper was written), government agencies such as the FAA have started to move from large upfront assessments for an authorization to operate (ATO) to "continuous risk assessment." See this FedRAMP doc for an example.
3. Briefly discusses open research and quotes a report that mentions TinyOS, but doesn't make policy suggestions around open-standards or FOSS.

[grahamschock]

Reviewer: Graham Schock Review Type: Critical

Problem Being Solved Every year software has a bigger role in critical components in medical devices. However, an increasing prevalence of software leads to software faults and failures in medical devices; the paper shows this through an analysis of recall rates over time that were attributable to software errors. One of the most infamous software errors that I have known since highschool was the Therac-25 Incident, where a radiation therapy machine gave lethal doses to patients.

Contributions “Trustworthy Medical Device Software” is different to the other IoT papers that I have read for this class. Fu takes a more systems engineering perspective to the software issues at hand. Fu details different techniques that are necessary in order to ensure that we can create trustworthy software in medical devices. Fu also describes the unique challenges of software in medical devices. For example he details how the discrete nature of software and how software is not susceptible to small errors as manufactured engineering systems are.

Questions

• What causes the medical device industry to be so behind the ball on software safety and engineering? I thought that medical devices are heavily regulated by the FDA. Maybe there needs to not only be a culture shift in the device industry but also within the regulation authority.

• I wonder how we can actually test these devices and software? I had to do some research on the avionics industry software testing and some of the testing was to physically subject planes to insane conditions. I wonder if there is a way to simulate conditions within the human body and subject medical devices to those conditions.

• What software languages are most of these devices written in? Because most wearables and embedded systems are written in languages like C, as the paper discusses these devices are susceptible to a lot of type errors etc. However, other devices like Rust might be a little more safe.

Critiques

• I wish a little more formal verification of software was discussed. There are a lot more elements of having verified software than just the language that we write in. For example, using a verified compiler and operating system can lead to even more confidence that our software does what we think it does.

• Not a lot of trade offs were discussed with these software developing models. There are some costs to requiring more testing and verification which can negatively impact the industry. Knowing these tradeoffs is necessary in order to make informed decisions.

[AkinoriKahata]

Reviewer: Akinori Kahata Review type: Critical

1. The problem being solved.
   • Although the increase of using software for medical devices has improved the functions of medical devices, it also causes negative effects on medical devices because the characteristics of software is different from hardware, which has been used for medical devices. For example, software has to be updated periodically, but sometimes updating software would cause the problem of other functions, and it isn't easy to verify previously. To keep the trustworthiness of medical devices, all stakeholder, including manufacture, medical stuff, software engineer and policymaker, should understand the difference between software and hardware, and overcome the problems.
2. The main contributions.
   • Firstly, author try to make clear the difference between software and hardware, especially what can be critical for medical functions, with examples, such as specific medical devices and aviation system. Secondly, he introduces the solutions from the technical point of view, especially software engineering approaches. Finally, the he points out the importance of rulemaking side, which including regulatory policy, concept of responsibility sharing.
3. Questions.
   • I want to know the definition of each element of trustworthy. What is the difference between dependability, reliability and trustworthy? Sometimes these words are used confusing.
   • Clarifying the responsibility and role is critical for the digital society. Because I’m not familiar with the medical sector, it is very helpful for me to clarify who are the stakeholders of medical sectors and what is the most severe problem the author expected.
4. Critiques.
   • When we talk about security issues, we have to consider what is a threat. The author introduces the damage of malware in the paper, but how to be infected malware is different from each occasion. Threat analysis is important for discussion about trustworthiness.
   • To consider the updating problem is significant for medical devices, but we have to distinguish software bugs and security vulnerabilities. In my understanding, these differences are not clearly distinguished in the paper, but we have to clarify why the update is essential or not.

[rachellkm]

Reviewer: Rachell Kim Review Type: Critical

Problem Being Solved:

Two major concerns surrounding medical device software are effectiveness and safety. Complexity of the software systems, incompatibility of software systems from differing manufacturers, and inconsideration of potential human errors during software design all contribute to disastrous effects to patients who depend on the trustworthiness of these devices. This paper attempts to provide a summary of the risks and benefits of software used within medical devices.

Main Contributions:

This paper provides a high level analysis on the various roles of software in medical devices. The author evaluates the current state of software management in medical devices and offers a few solutions that may aid in mitigating harmful accidents as a result of faulty software systems. Moreover, the author introduces important questions about the positive and negative contributions of software in medical devices that have yet to be studied and encourages investigations into these causes.

Questions:

1. One of the proposed solutions was to specify “outcome measures” (section 4.1), but what exactly does “meaningful outcome measures” mean?
2. The author repeatedly cites the avionics industry as an example where systems engineering for critical systems are considered more “robust,” but did not seem to list specific examples or inherent differences in practice. Could there be other characteristics that impede the medical industry from adopting equivalent standards, or does it really boil down to difference in “safety culture”?
3. Are there other “modern” software engineering techniques other than type checking and static analysis that are not being done for software systems in medical devices?

Critique:

1. I thought this paper would go into more specifics about what constitutes as “good” systems engineering for medical devices and detail the differences in existing and proposed solutions, but it gave a much more general overview of what that would look like. Specifics would have been nice, but I can understand why that information would not have been available.
2. The abstract states that this paper is meant to summarize what the computing community already knows about the role of software in medical devices, but it lists the lack of statistics and collected information regarding how much functional control software has on these devices as one of the problems.

[gkahl]

Reviewer: Greg Kahl

Review Type: Comprehensive

Problem Being Solved

As technology has advanced, and particularly embedded systems, naturally embedded systems and microcontrollers are being implemented in medical devices in order to bring additional features, real time monitoring, and even prescription of medication to patients automatically. Although the addition of computers into these medical devices adds great new features and monitoring, it also brings in an additional layer of error. There are new security risks you have to worry about, and even software bugs that may result misinformation, diagnosis, or even over-distribution of medicine.

Contributions

This paper explores what exactly makes these Medical Devices untrustworthy. They believe that in order to make medical devices that are trustworthy there is a certain design approach you must take in order to design a secure system. They advocated for more carefully thought out system specifications to help reduce all possible "bad cases" no matter how unlikely they are to occur. In addition to this there should be safety precautions put into place in order to reduce the chance of human error affecting the performance of the system. Finally, they advocate for more open/researchable environment in which these systems can improve and the FDA can better regulate the software being put into place.

Questions

1 - They briefly discussed how a lot of these systems are beginning to connect to the internet but never really discussed the wireless security of these systems. Isn't it a major risk if if someone can gain access to the devices distributing medicine/care to people?

2 - They discussed the development of more specific specifications and requirements for these systems. Isn't it extremely difficult to develop bullet-proof specifications for every edge case?

3 - In the past we have discussed schedulers and scheduling tasks that have direct impact on human lives as highest priority. It seems that a lot of these devices have this aspect, how does the scheduler and its security come into play when designing a trustworthy medical device?

[ericwendt]

Reviewer: Eric Wendt Review Type: Comprehension

Overview/Problems Being Solved This paper discusses the dangers of software in medical devices, and details the responsibilities of programmers and device designers to ensure that many difficult situations are accounted for. Even if undesirable outcomes of the device’s performance seem to be unpredictable, disastrous effects can occur if care is not taken to make sure that specifications are near perfect.

Contributions One contribution that this paper details is the overview of responsibilities designers and end-users must face when using software and embedded systems. These include but are not limited to: Adopting Modern Software Engineering Techniques, specifying meaningful requirements, and mitigating risks due to human factors. Another contribution is the paper’s discussion of shooting for predictable outcomes and what the desired outcomes should be, rather than extended features on new devices. It’s very important adding new features does not fault previous features or expected outcomes, such as overdosing a patient because an update interfered with working software.

Questions:

1. Is it better to have consistently updating software on embedded systems to ensure timely patches and improvements, or stable software that is effective but does not change quickly?
2. Statistics are mentioned in this paper as well as a few categories. Is there a relevant example of a type of statistic that could help a broad range of medical devices?
3. Is the use of AI practical for mitigating human errors. I’m thinking it may be useful to notify humans when a possible mistake is being made, but not actually have the AI make the definitive decision itself.

[s-hanna15]

Reviewer: Sam Hanna Review Type: Critical

Problem Being Solved: This paper talked about the medical device industry and the lack of oversight on the trustworthiness of medical devices that implement software. The medical device industry is one that has a lot of importance in terms of lifesaving medicine, but if the devices are unsafe it also has the ability to harm and potentially kill a lot of people.

Important Areas: This paper focuses on a systems engineering approach to trying to ensure that medical devices that utilize software are trustworthy. The two areas that they emphasize are having required specifications and human factors that could affect the use of the device. They also look at what policy and the FDA can do to increase the trustworthiness of these devices across the board.

Questions:

1. If the medical devices that go out are able to cause such catastrophic failure such as the Therac-25, what does the FDA actually do currently to ensure that devices are safe for use?
2. They mention that it is difficult to reproduce problems, what is a way for the people making these devices to do such a thing and fix the problems with the devices they are making?
3. Are there any privacy concerns with having software involved in medical devices?

Critiques:

1. They take a very systems engineering approach to this, which works for some part of it, but then they recommend more software developers involved in the process at the FDA and I think maybe they should go both ways, have more software developers at the FDA, but also more systems engineers involved in the whole process if that is the approach they are going to take.
2. They talk in-depth about normal factors that could affect these devices, such as bad coding practices, human error, and bad specifications. They mention security briefly, and privacy not at all. I feel like that is something that should have been mentioned at least a little more, these devices have huge privacy considerations and with increasing security concerns in terms of malware, it is something that needs to be considered when making connected devices.
3. They talk about how medical devices should have a third-party review and I think that is something that should have gotten a little more focus, there are places that review the security of products before they can be used in certain industries and it is something that seems like it could really help this field.

[chandaweia]

Reviewer: Cuidi Wei Review Type: Comprehension

Problem being solved This paper presents the challenges and defects of medical device software in effectiveness and safety aspects and try to list the solutions from techniques and policy to improve.

Main contributions This paper detailed the techniques to create trustworthiness and recommend policy for medical device software by summarizing what the computing research community knows about the role of trustworthy software for the safety and effectiveness of medical devices.

Questions 1.In the introduction part, the author presented the data about medical software failures. However, with the development of computer science and increased popularity of medical devices, is it possible that the increased recalls is due to the more many medical devices. For that case, I think the recalls will also increase. 2.Even if the medical devices have more safety and effectiveness, there is a possibility that if the medical devices down because of low power. It still also dangerous. 3.I’m confused about that why companies are less likely to perceive value from specification of requirements. According to my understanding, the companies are more likely to improve the effectiveness and safety under the driven by interests.

[samfrey99]

Reviewer: Sam Frey Review Type: Comprehension

Problem: Medical devices have potential to injure or even kill the person they are meant to help due to a simple software bug. This paper highlights how much more needs to be done to ensure the reliability of these systems.

Contributions: The paper highlights specific techniques from various fields of engineering to make safety a priority in the development of medical devices. These include using strongly typed programming languages, taking a Systems-Engineering approach to the development lifecycle, and working to prevent and eliminate user error.

Questions:

1. Does a form of formal verification exist for medical devices? A platform for this may help eliminate software issues discussed in the paper.
2. Do companies assume that medical professionals know what to do and therefore skimp on error checking, are medical professionals assuming that the technology won't allow for mistakes like improper dosing, or is it a mix of the two issues?
3. The paper frequently refers to security as a low-probability, high-consequence risk but doesn't say what to do about it. How can we improve security as these devices continue to be connected to the internet?

[pcodes]

Reviewer: Pat Cody

Review Type: Critical Review

Problem Being Solved:

Software is increasingly being combined with medicine to create smart medical devices, but a failure in one of these devices is far worse than a smart fridge, as these devices can be the difference between saving or killing someone. Testing these devices is also difficult, as they might perform fine in a vacuum, but in practice they might behave incorrectly when in communication with other devices.

Main Contributions:

This paper highlights many of the technical issues that contribute to medical device software failure, ranging from a poor spec to poor software engineering techniques. It also discusses policy improvements, such as requiring better data collection and allowing for open research in the area, as opposed to the often-proprietary nature of medical research.

Questions:

• The paper authors discuss some frequent problems that happen at a technical level, but do they have any proposals for how to fix the software update problem? This is a frequent question in other areas of IoT, and seems even more relevant when we're dealing with devices that are responsible for people's lives.
• Do the authors have suggestions for how to improve interopability between devices? Requiring third-party review fixes the "poor testing" problem, but are there technical improvements that can be made to reduce compatibility problems?

Critiques:

• The policy suggestions were interesting, but I wish they had provided a little more background on the current state of medical policy.
• No mention of privacy, which seems like it would have been a great point to bring up given that it is both a technical and policy problem.