Investigate possible bad pixels on sensors of Astrobee flight units

[trey0]

The Metis SBOT SBIR team notified us that, based on their work with Astrobee NavCam images collected on ISS, there may be dead/hot/stuck pixels or dust on some of the NavCam sensors. They noticed the problem in part because (unlike most users) they used the raw Bayer format NavCam images and converted them to RGB rather than greyscale, which makes "salt and pepper" noise stand out as brightly colored pixels, with the color depending on where the affected pixel falls in the Bayer pattern.

Radiation damage to sensor pixels was expected from the earliest phases of the Astrobee design effort, based on similar issues observed in ISS camcorder imagery. That's one of the reasons that all of Astrobee's sensors are designed to be replaceable (although the difficulty varies, and probably not all are good candidates for astronauts to replace on-orbit). However, we had not previously noted this type of damage to Astrobee sensors, and we don't have a process in place to regularly check for damage.

Further investigation needed:

• [ ] Determine which of the potential issues we're actually dealing with (e.g., dead/hot/stuck pixels, dust on sensor).
• [ ] Assess the extent of the problem (we have multiple Astrobee flight and ground units that may be affected to different extents, and this problem is likely to apply to other Astrobee sensors besides the NavCam).
• [ ] Assess the impact of the problem, with particular emphasis on localization. The SBOT team indicated they thought it negatively impacted their (somewhat different) localization algorithm because pixels that differ strongly from their neighbors can become bogus features that confuse feature tracking algorithms. If true, Astrobee's baseline localization system, which also uses feature tracking, is probably impacted as well.

Potential mitigation strategies:

• Communicate the problem to guest scientists so they can plan for any impacts on their experiments.
• For many purposes, the impact of the problem could be reduced by replacing the values of bad pixels with a best-guess value interpolated from their neighbors. This approach would work best if, within each sensor unit, the problem is limited to a small consistent subset of bad pixels that we could readily identify and correct. For best results, the correction should be applied to the raw Bayer image prior to greyscale conversion. This would be a software-only change, but would incur ongoing maintenance burden (periodic updates to a per-robot configuration file listing each sensor's bad pixels, possibly requiring specific data collection activities to inform the update). Alternatively, we could try to automatically identify bad pixels from natural imagery, perhaps at the start of each activity.
• Depending on the nature of the problem, there may be hardware mitigations to try. If dust is involved, some sensors provide self-cleaning options. There may also be "hard reset" actions that can restore temporarily stuck pixels. Unfortunately, nothing like this is mentioned for the NavCam/DockCam sensors in the DFM 42BUC03-ML Technical Reference Manual.
• Sensor replacement may make sense in some cases, especially if the problem affects Honey, which is already on the ground for repairs.

[jvbenavi]

Then another possible mitigation that may have been assumed but not mentioned explicitly, is asking crew to wipe clean the front facing surface. I vaguely recall an occurrence of successfully reducing/eliminating “stuff” in a nav-cam image after asking crew to wipe it.

[bcoltin]

Suyoung evaluated our imagery on multiple localization algorithms including ORBSLAM and did not have any issues with feature detection. We have also not seen evidence of this impacting our feature detection.