Feature Toggle cleanup

[jennb33]

User Story

As the Platform developers and managers, We need to review all Feature toggles that have been removed from yaml, So that we can reflect that Feature toggles status in the database, for parity.

Issue Description

Closer attention needs to be paid to the feature toggles that are used by other developers and teams. Currently there are feature toggles that have been removed from yaml but not from the Database, so the Platform Product team needs to determine how many feature toggles this is and remove the ones from the db that need removed.

Tasks

• [x] Create feature toggle status inventory that includes yaml and Database status. For all four environments.
• [x] Identify which entries within the yaml files have been removed
• [x] Compile documentation of this process and its success
• [x] Update any feature toggle documentation to cover the process for maintaining alignment between the yaml files and the database to avoid this from reoccurring.
• [x] Strategize on additional automation to remind developers on BE Support to delete the flag from the DB. Need to include DEV, STG, Sandbox, and PROD in the review.

Success Metrics

The Platform Product team can review the feature toggles to determine how many differences present between yaml files and the databases. Then teams will be tracked to determine how long it takes for each team to make those updates in the database to validate the yaml file status.

Acceptance Criteria

• [x] The Platform will have fewer variations

• [x] The Feature toggle process will be updated to include any tips on how to keep the yaml files and database aligned

Validation

• [x] Review new documentation: https://vfs.atlassian.net/wiki/spaces/BCP/pages/3481370733/Removing+feature+flippers
• [x] View Datadog logs.
  • [x] Datadog search for "Error processing Flipper features" - This is a new log. I haven't seen any yet, but that's okay.
  • [x] Datadog search for "The following feature flippers were added" - this is a new log. Make sure it's an array of features.
  • [x] Datadog search for "Consider removing features no longer in config/features.yml" - I changed this to only log if there were features to list. So there should be fewer logs since this work merged (~11am dev/staging. 1pm prod/sandbox 9/24)

[rmtolmach]

We do have code for this in vets-api flipper.rb. We even have a job that cleans up old features, however, for some unknown reason, it occasionally deleted features in prod (😱) that shouldn't have been deleted, so we disabled the job in 2023. The ticket around that was https://github.com/department-of-veterans-affairs/va.gov-team/issues/53538.

There is still a message that gets printed to the logs and can be found in dd here. Because of the history around these values sometimes not being accurate, I ran through the steps below manually to find the features needing removed in dev:

In a pod, run bundle exec rails c. Then:

feature_names = Flipper.features.map(&:name)
puts feature_names

Copy feature_names and format them into an array to use in a script

require 'yaml'

yaml_content = YAML.load_file('config/features.yml')

# Extract the feature names from the 'features' section
feature_names = yaml_content['features'].keys

# Format them into an array
in_argo_dev = %w[list here]

need_deleting_from_db = in_argo_dev - feature_names
puts need_deleting_from_db
# puts need_deleting_from_db.count

There are 118 flipper features that need the be deleted from dev. I will delete them tomorrow.

[rmtolmach]

✅ the 118 features in DEV that were no longer in features.yml but not removed from the db, have been removed. These 118 features were also called out in Datadog, hence the name. from_datadog was an array of feature names I pasted in the console.

irb(main):009* from_datadog.each do |feature_name|
irb(main):010*   Flipper.remove(feature_name.to_sym)
irb(main):011> end
=> 
["combined_financial_status_report_enhancements",
 "cerner_allow_partial_facilities",
 "mobile_precache_appointments",
 "cst_use_claim_details_v2",
 "decision_review_sc_pii_expunge_enabled",
 "drupal_footer",
 "hca_household_v2_enabled",
 "yellow_ribbon_search_enhancement",
 "sharable_link",
 "show_new_secure_messaging_page",
 "financial_status_report_streamlined_waiver",
 "injected_header_web_components",
 "disability_compensation_lighthouse_submit_migration",
 "hca_vaprofile_military_info",
 "profile_use_field_editing_page",
 "check_in_experience_travel_logic",
 "profile_use_hub_page",
 "my_va_lighthouse_claims",
 "profile_show_email_notification_settings",
 "mobile_v1_lighthouse_facilities",
 "pension_module_enabled",
 "medical_copays_html_medical_statements_view_enabled",
 "va_online_scheduling_request_flow_update",
 "check_in_experience_unified_landing_page",
 "mhv_secure_messaging_to_phase_1",
 "financial_status_report_streamlined_waiver_assets",
 "cst_use_lighthouse",
 "check_in_experience_pre_check_in_action_link_top_placement",
 "omni_channel_link",
 "cerner_transition_556_t30",
 "va_online_scheduling_print_list",
 "cerner_override_463",
 "pdf_warning_banner",
 "show_medical_copays",
 "income_limits_new",
 "my_va_enable_new_sip_config",
 "va_online_scheduling_clinic_location",
 "debt_letters_show_letters",
 "use_facilities_api_v2_for_cerner_cta",
 "decision_review_hlr_pii_expunge_enabled",
 "fsr_confirmation_email",
 "ezr_tera_enabled",
 "yellow_ribbon_degree_filter",
 "disability_526_classifier",
 "disability_526_maximum_rating_api",
 "reactivation_experiment_rate_limit",
 "pw_ehr_cta_drupal_source_of_truth",
 "show_meb_unverified_user_alert",
 "ezr_async",
 "simple_forms_lighthouse_benefits_intake_service",
 "profile_hide_direct_deposit_comp_and_pen",
 "decision_review_form_store_saved_claims",
 "va_view_dependents_access",
 "check_in_experience_phone_appointments_enabled",
 "va_profile_information_v3_redis",
 "va_online_scheduling_status_improvement_canceled",
 "show_new_get_medical_records_page",
 "decision_review_sc_pact_act_boolean",
 "va_online_scheduling_appointment_list",
 "show_myva_dashboard_2_0",
 "check_in_experience_travel_claim_increase_timeout",
 "use_new_get_all_features",
 "va_online_scheduling_status_improvement",
 "show_events_v2",
 "decision_review_nod_pii_expunge_enabled",
 "supplemental_claim",
 "va_online_scheduling_clinic_filtering",
 "caregiver_sigi_enabled",
 "decision_review_use_hexapdf_for_encrypted_attachments",
 "va_profile_information_v3_service",
 "va_online_scheduling_appointment_details_redesign",
 "hca_submission_failure_email_va_notify",
 "check_in_experience_pre_check_in_handoff_enabled",
 "form10182_nod",
 "va_online_scheduling_physical_location",
 "show_header_v2",
 "cerner_override_531",
 "va_online_scheduling_descriptive_back_link",
 "disability_compensation_lighthouse_rated_disabilities_provider",
 "reactivation_experiment_initial_gate",
 "profile_use_notification_settings_checkboxes",
 "cerner_override_648",
 "virtual_agent_upgrade_webchat_14_15_8",
 "mhv_landing_page_enabled",
 "va_online_scheduling_required_schedulable_param",
 "va_online_scheduling_unenrolled_vaccine",
 "check_in_experience_travel_api_v2_cutover",
 "hca_tera_enabled",
 "mhv_secure_messaging_to_va_gov_release",
 "cerner_transition_556_t5",
 "va_online_scheduling_acheron_service",
 "decision_review_supplemental_claim_pdf_v3",
 "check_in_experience_45_minute_reminder",
 "decision_review_higher_level_review_pdf_v3",
 "check_in_experience_lorota_security_updates_enabled",
 "check_in_experience_travel_btsss_ssm_urls_enabled",
 "profile_show_mhv_notification_settings",
 "facility_locator_lighthouse_covid_vaccine_query",
 "decision_review_save_encrypted_attachments",
 "profile_use_info_card",
 "pension_claim_submission_to_lighthouse",
 "representatives_portal_api",
 "show_new_view_test_lab_results_page",
 "ezcg_use_facility_api",
 "check_in_experience_patient_cell_phone",
 "profile_contacts",
 "profile_lighthouse_direct_deposit",
 "representatives_portal_frontend",
 "form526_backup_submission_force_offramp",
 "show_meb_ch33_self_form",
 "disability_526_ep_merge_new_claims",
 "cst_use_new_claim_cards",
 "yellow_ribbon_mvp_enhancement",
 "mhv_secure_messaging_blocked_triage_group_1_0",
 "va_profile_information_v3_transactions",
 "va_online_scheduling_GA4_migration",
 "cerner_override_663",
 "disability_526_ep_merge_multi_contention"]

[rmtolmach]

✅ 81 features removed from Staging

irb(main):086> staging_from_dd.count
=> 81
irb(main):087* staging_from_dd.each do |feature_name|
irb(main):088*   Flipper.remove(feature_name.to_sym)
irb(main):089> end
=> 
["combined_financial_status_report_enhancements",
 "decision_review_use_appeal_submitted_job",
 "mobile_precache_appointments",
 "cst_use_claim_details_v2",
 "decision_review_sc_pii_expunge_enabled",
 "hca_household_v2_enabled",
 "drupal_footer",
 "mhv_secure_messaging_unique_user_logging_enabled",
 "financial_status_report_streamlined_waiver",
 "vye_load_scrypt_config",
 "hca_vaprofile_military_info",
 "mobile_v1_lighthouse_facilities",
 "pension_module_enabled",
 "medical_copays_html_medical_statements_view_enabled",
 "check_in_experience_unified_landing_page",
 "financial_status_report_streamlined_waiver_assets",
 "cst_use_lighthouse",
 "dependents_central_submission_lighthouse",
 "omni_channel_link",
 "va_online_scheduling_med_review_instructions",
 "cerner_transition_556_t30",
 "va_online_scheduling_print_list",
 "dependents_central_submission",
 "flag_a_representative",
 "show_medical_copays",
 "my_va_enable_new_sip_config",
 "debt_letters_show_letters",
 "fsr_confirmation_email",
 "decision_review_hlr_pii_expunge_enabled",
 "show_authenticated_menu_enhancements",
 "disability_526_classifier",
 "disability_526_maximum_rating_api",
 "reactivation_experiment_rate_limit",
 "show_meb_unverified_user_alert",
 "simple_forms_lighthouse_benefits_intake_service",
 "decision_review_form_store_saved_claims",
 "disability_526_reduced_contention_list",
 "va_view_dependents_access",
 "check_in_experience_phone_appointments_enabled",
 "decision_review_sc_pact_act_boolean",
 "dependents_submit_674_independently",
 "find_a_rep",
 "decision_review_nod_pii_expunge_enabled",
 "supplemental_claim",
 "decision_review_use_hexapdf_for_encrypted_attachments",
 "check_in_experience_pre_check_in_handoff_enabled",
 "form10182_nod",
 "find_a_representative",
 "decision_review_updated_pii_rules",
 "sign_in_page_and_modal_experiment_lga",
 "reactivation_experiment_initial_gate",
 "va_online_scheduling_unenrolled_vaccine",
 "hca_tera_enabled",
 "check_in_experience_travel_api_v2_cutover",
 "toggle_vye_adress_direct_deposit_forms",
 "cerner_transition_556_t5",
 "decision_review_supplemental_claim_pdf_v3",
 "check_in_experience_45_minute_reminder",
 "decision_review_higher_level_review_pdf_v3",
 "post_911_gibill_statuses_new_error_handling",
 "check_in_experience_lorota_security_updates_enabled",
 "check_in_experience_travel_btsss_ssm_urls_enabled",
 "profile_show_mhv_notification_settings",
 "decision_review_save_encrypted_attachments",
 "pension_claim_submission_to_lighthouse",
 "representatives_portal_api",
 "hca_remove_id_fields_enabled",
 "ezcg_use_facility_api",
 "my_va_remove_cerner_message",
 "dependents_encrypt_jobs",
 "mhv_sm_session_policy",
 "representatives_portal_frontend",
 "form526_backup_submission_force_offramp",
 "show_meb_ch33_self_form",
 "disability_526_ep_merge_new_claims",
 "cst_use_new_claim_cards",
 "yellow_ribbon_mvp_enhancement",
 "mhv_secure_messaging_blocked_triage_group_1_0",
 "decision_review_updated_pii_ruiles",
 "disability_526_ep_merge_multi_contention",
 "toggle_vye_adress_direct_deposit_forms_in_profile"]
irb(main):090>

[rmtolmach]

I checked Datadog to verify that the log message, Consider removing features no longer in config/features.yml: ... didn't appear for dev in the past day but it was there 😕 😕. It listed 36 features, all features I removed yesterday. One interesting thing I realized was that these were all removed from features.yml within the last month and a half. I checked features.yml on Aug 5 and they were there.

This was the message in datadog in a dev pod:

Consider removing features no longer in config/features.yml: cst_use_claim_details_v2, injected_header_web_components, profile_show_email_notification_settings, mobile_v1_lighthouse_facilities, pension_module_enabled, medical_copays_html_medical_statements_view_enabled, check_in_experience_unified_landing_page, mhv_secure_messaging_to_phase_1, va_online_scheduling_med_review_instructions, pdf_warning_banner, show_medical_copays, my_va_enable_new_sip_config, debt_letters_show_letters, use_facilities_api_v2_for_cerner_cta, fsr_confirmation_email, ezr_tera_enabled, pw_ehr_cta_drupal_source_of_truth, ezr_async, simple_forms_lighthouse_benefits_intake_service, check_in_experience_phone_appointments_enabled, va_profile_information_v3_redis, va_online_scheduling_status_improvement_canceled, use_new_get_all_features, show_events_v2, va_profile_information_v3_service, va_online_scheduling_appointment_details_redesign, hca_submission_failure_email_va_notify, check_in_experience_pre_check_in_handoff_enabled, mhv_secure_messaging_to_va_gov_release, check_in_experience_45_minute_reminder, check_in_experience_lorota_security_updates_enabled, check_in_experience_travel_btsss_ssm_urls_enabled, profile_show_mhv_notification_settings, pension_claim_submission_to_lighthouse, disability_526_ep_merge_new_claims, va_profile_information_v3_transactions, disability_526_ep_merge_multi_contention

Seems like a caching issue but we do set config.flipper.memoize = false in our config, so idk... I ran Rails.cache.read("flipper/v1/features") in the dev console but it returns nil, so maybe it's not the cache 😒

For Staging, let the record show that after I deleted the 81 features earlier today, the logging was accurately showing no features to be removed. TBD what happens tomorrow 😃

What does this all mean??? Us asking people to delete their features from the console is useless 😆

[rmtolmach]

Update on Staging It's now listing one we should "consider removing" (search_use_v2_gsa), but that one was added in this PR 3 days ago. So it's like it doesn't have the updated features.yml in staging.

Update on Dev It's now listing 98 features needing removed 🤦‍♀️

Sandbox

✅ I removed 106 old features from Sandbox.

irb(main):222> old_features.count
=> 106
irb(main):223* old_features.each do |feature_name|
irb(main):224*   Flipper.remove(feature_name.to_sym)
irb(main):225> end
=> 
["combined_financial_status_report_enhancements",
 "cerner_allow_partial_facilities",
 "decision_review_use_appeal_submitted_job",
 "mobile_precache_appointments",
 "cst_use_claim_details_v2",
 "decision_review_sc_pii_expunge_enabled",
 "hca_household_v2_enabled",
 "drupal_footer",
 "sharable_link",
 "mhv_secure_messaging_unique_user_logging_enabled",
 "show_new_secure_messaging_page",
 "financial_status_report_streamlined_waiver",
 "vye_load_scrypt_config",
 "hca_vaprofile_military_info",
 "profile_use_field_editing_page",
 "check_in_experience_travel_logic",
 "profile_use_hub_page",
 "profile_show_direct_deposit_single_form_edu_downtime",
 "profile_show_email_notification_settings",
 "mobile_v1_lighthouse_facilities",
 "medical_copays_html_medical_statements_view_enabled",
 "check_in_experience_unified_landing_page",
 "financial_status_report_streamlined_waiver_assets",
 "cst_use_lighthouse",
 "check_in_experience_pre_check_in_action_link_top_placement",
 "dependents_central_submission_lighthouse",
 "omni_channel_link",
 "cerner_transition_556_t30",
 "va_online_scheduling_print_list",
 "cerner_override_463",
 "dependents_central_submission",
 "flag_a_representative",
 "show_medical_copays",
 "my_va_enable_new_sip_config",
 "debt_letters_show_letters",
 "decision_review_hlr_pii_expunge_enabled",
 "fsr_confirmation_email",
 "profile_show_direct_deposit_single_form_alert",
 "show_authenticated_menu_enhancements",
 "disability_526_classifier",
 "disability_526_maximum_rating_api",
 "reactivation_experiment_rate_limit",
 "show_meb_unverified_user_alert",
 "simple_forms_lighthouse_benefits_intake_service",
 "profile_hide_direct_deposit_comp_and_pen",
 "decision_review_form_store_saved_claims",
 "disability_526_reduced_contention_list",
 "va_view_dependents_access",
 "check_in_experience_phone_appointments_enabled",
 "va_online_scheduling_status_improvement_canceled",
 "show_new_get_medical_records_page",
 "decision_review_sc_pact_act_boolean",
 "dependents_submit_674_independently",
 "find_a_rep",
 "profile_show_direct_deposit_single_form",
 "decision_review_nod_pii_expunge_enabled",
 "supplemental_claim",
 "decision_review_use_hexapdf_for_encrypted_attachments",
 "hca_submission_failure_email_va_notify",
 "check_in_experience_pre_check_in_handoff_enabled",
 "form10182_nod",
 "find_a_representative",
 "show_header_v2",
 "cerner_override_531",
 "decision_review_updated_pii_rules",
 "va_online_scheduling_descriptive_back_link",
 "sign_in_page_and_modal_experiment_lga",
 "reactivation_experiment_initial_gate",
 "profile_use_notification_settings_checkboxes",
 "cerner_override_648",
 "virtual_agent_upgrade_webchat_14_15_8",
 "my_va_show_header_link",
 "va_online_scheduling_unenrolled_vaccine",
 "check_in_experience_travel_api_v2_cutover",
 "hca_tera_enabled",
 "toggle_vye_adress_direct_deposit_forms",
 "cerner_transition_556_t5",
 "va_online_scheduling_acheron_service",
 "decision_review_supplemental_claim_pdf_v3",
 "check_in_experience_45_minute_reminder",
 "decision_review_higher_level_review_pdf_v3",
 "post_911_gibill_statuses_new_error_handling",
 "check_in_experience_lorota_security_updates_enabled",
 "check_in_experience_travel_btsss_ssm_urls_enabled",
 "profile_show_mhv_notification_settings",
 "decision_review_save_encrypted_attachments",
 "pension_claim_submission_to_lighthouse",
 "representatives_portal_api",
 "hca_remove_id_fields_enabled",
 "show_new_view_test_lab_results_page",
 "ezcg_use_facility_api",
 "my_va_remove_cerner_message",
 "dependents_encrypt_jobs",
 "mhv_sm_session_policy",
 "profile_lighthouse_direct_deposit",
 "representatives_portal_frontend",
 "form526_backup_submission_force_offramp",
 "show_meb_ch33_self_form",
 "disability_526_ep_merge_new_claims",
 "cst_use_new_claim_cards",
 "yellow_ribbon_mvp_enhancement",
 "mhv_secure_messaging_blocked_triage_group_1_0",
 "cerner_override_663",
 "disability_526_ep_merge_multi_contention",
 "toggle_vye_adress_direct_deposit_forms_in_profile",
 "profile_show_direct_deposit_single_form_uat"]

Prod

✅ I removed 25 from prod.

irb(main):028> old_prod_features.count
=> 25
irb(main):029* old_prod_features.each do |feature_name|
irb(main):030*   Flipper.remove(feature_name.to_sym)
irb(main):031> end
=> 
["cst_use_claim_details_v2",
 "profile_show_direct_deposit_single_form_edu_downtime",
 "mobile_v1_lighthouse_facilities",
 "medical_copays_html_medical_statements_view_enabled",
 "check_in_experience_unified_landing_page",
 "show_medical_copays",
 "my_va_enable_new_sip_config",
 "debt_letters_show_letters",
 "fsr_confirmation_email",
 "profile_show_direct_deposit_single_form_alert",
 "simple_forms_lighthouse_benefits_intake_service",
 "decision_review_form_store_saved_claims",
 "check_in_experience_phone_appointments_enabled",
 "profile_show_direct_deposit_single_form",
 "check_in_experience_pre_check_in_handoff_enabled",
 "check_in_experience_travel_api_v2_cutover",
 "mhv_secure_messaging_to_va_gov_release",
 "check_in_experience_45_minute_reminder",
 "check_in_experience_lorota_security_updates_enabled",
 "check_in_experience_travel_btsss_ssm_urls_enabled",
 "profile_show_mhv_notification_settings",
 "pension_claim_submission_to_lighthouse",
 "disability_526_ep_merge_new_claims",
 "disability_526_ep_merge_multi_contention",
 "profile_show_direct_deposit_single_form_uat"]

[rmtolmach]

PR to add and modify flipper logging: https://github.com/department-of-veterans-affairs/vets-api/pull/18575

[rmtolmach]

There were 99 in dev today, so I deleted them for a second time.

irb(main):101* dev_deleted.each do |feature_name|
irb(main):102*   Flipper.remove(feature_name.to_sym)
irb(main):103> end
=> 
["combined_financial_status_report_enhancements",
 "cerner_allow_partial_facilities",
 "mobile_precache_appointments",
 "cst_use_claim_details_v2",
 "decision_review_sc_pii_expunge_enabled",
 "drupal_footer",
 "hca_household_v2_enabled",
 "sharable_link",
 "show_new_secure_messaging_page",
 "financial_status_report_streamlined_waiver",
 "injected_header_web_components",
 "disability_compensation_lighthouse_submit_migration",
 "hca_vaprofile_military_info",
 "profile_use_field_editing_page",
 "profile_show_direct_deposit_single_form_edu_downtime",
 "my_va_lighthouse_claims",
 "profile_show_email_notification_settings",
 "mobile_v1_lighthouse_facilities",
 "pension_module_enabled",
 "medical_copays_html_medical_statements_view_enabled",
 "va_online_scheduling_request_flow_update",
 "check_in_experience_unified_landing_page",
 "mhv_secure_messaging_to_phase_1",
 "cst_use_lighthouse",
 "check_in_experience_pre_check_in_action_link_top_placement",
 "omni_channel_link",
 "va_online_scheduling_med_review_instructions",
 "va_online_scheduling_print_list",
 "cerner_override_463",
 "pdf_warning_banner",
 "show_medical_copays",
 "income_limits_new",
 "my_va_enable_new_sip_config",
 "va_online_scheduling_clinic_location",
 "debt_letters_show_letters",
 "use_facilities_api_v2_for_cerner_cta",
 "fsr_confirmation_email",
 "decision_review_hlr_pii_expunge_enabled",
 "ezr_tera_enabled",
 "profile_show_direct_deposit_single_form_alert",
 "disability_526_classifier",
 "reactivation_experiment_rate_limit",
 "pw_ehr_cta_drupal_source_of_truth",
 "show_meb_unverified_user_alert",
 "ezr_async",
 "simple_forms_lighthouse_benefits_intake_service",
 "profile_hide_direct_deposit_comp_and_pen",
 "va_view_dependents_access",
 "check_in_experience_phone_appointments_enabled",
 "va_profile_information_v3_redis",
 "va_online_scheduling_status_improvement_canceled",
 "show_new_get_medical_records_page",
 "decision_review_sc_pact_act_boolean",
 "va_online_scheduling_appointment_list",
 "show_myva_dashboard_2_0",
 "use_new_get_all_features",
 "va_online_scheduling_status_improvement",
 "show_events_v2",
 "profile_show_direct_deposit_single_form",
 "decision_review_nod_pii_expunge_enabled",
 "supplemental_claim",
 "caregiver_sigi_enabled",
 "va_profile_information_v3_service",
 "va_online_scheduling_appointment_details_redesign",
 "hca_submission_failure_email_va_notify",
 "check_in_experience_pre_check_in_handoff_enabled",
 "form10182_nod",
 "show_header_v2",
 "cerner_override_531",
 "va_online_scheduling_descriptive_back_link",
 "disability_compensation_lighthouse_rated_disabilities_provider",
 "reactivation_experiment_initial_gate",
 "profile_use_notification_settings_checkboxes",
 "cerner_override_648",
 "mhv_landing_page_enabled",
 "va_online_scheduling_required_schedulable_param",
 "va_online_scheduling_unenrolled_vaccine",
 "mhv_secure_messaging_to_va_gov_release",
 "va_online_scheduling_acheron_service",
 "decision_review_supplemental_claim_pdf_v3",
 "check_in_experience_45_minute_reminder",
 "decision_review_higher_level_review_pdf_v3",
 "check_in_experience_lorota_security_updates_enabled",
 "check_in_experience_travel_btsss_ssm_urls_enabled",
 "profile_show_mhv_notification_settings",
 "facility_locator_lighthouse_covid_vaccine_query",
 "profile_use_info_card",
 "pension_claim_submission_to_lighthouse",
 "show_new_view_test_lab_results_page",
 "ezcg_use_facility_api",
 "profile_lighthouse_direct_deposit",
 "form526_backup_submission_force_offramp",
 "show_meb_ch33_self_form",
 "disability_526_ep_merge_new_claims",
 "yellow_ribbon_mvp_enhancement",
 "va_profile_information_v3_transactions",
 "cerner_override_663",
 "disability_526_ep_merge_multi_contention",
 "profile_show_direct_deposit_single_form_uat"]

[rmtolmach]

Datadog search for "Error processing Flipper features"

Datadog search for "The following feature flippers were added"

Compile documentation of this process and its success

✅ I created a doc in the Backend CoP space: https://vfs.atlassian.net/wiki/spaces/BCP/pages/3481370733/Removing+feature+flippers

Strategize on additional automation to remind developers on BE Support to delete the flag from the DB. Need to include DEV, STG, Sandbox, and PROD in the review.

We do have comprehensive documentation for developers on how to remove their features HERE. I frequently remind the BE CoP during our meetings about this. One automation idea is to post a comment to a PR if a feature in features.yml is being updated or deleted.

Update any feature toggle documentation to cover the process for maintaining alignment between the yaml files and the database to avoid this from reoccurring.

See above. Plus, because there are some pods with outdated features.yml, features get added to the db and there's no way to avoid this at the moment 😭 . That being said, it's not a huge deal if a feature gets added. It's just wrong and annoying and confusing 😑

✅ I created a new ticket, https://github.com/department-of-veterans-affairs/va.gov-team/issues/93511, to address the root cause.

[rmtolmach]

Validated by Lindsey. Slack thread.