search

department-of-veterans-affairs / va.gov-cms

Editor-centered management for Veteran-centered content.
https://prod.cms.va.gov
GNU General Public License v2.0
97 stars 69 forks source link

Migrate Facility services for VISN 7 and 8 #5273

Closed kevwalsh closed 3 years ago

kevwalsh commented 3 years ago

Dependencies

Service data is prepared in this format by VAMC upgrade team

ACs

swirtSJW commented 3 years ago

Discussion of system health service data is here https://dsva.slack.com/archives/CPQFCQKEH/p1621354635009000

swirtSJW commented 3 years ago

To satisfy creating a default paragraph with a sub paragraph I had to create a new migrate process plugin and contributed it to migration_tools. https://www.drupal.org/project/migration_tools/issues/3216353

swirtSJW commented 3 years ago

This updates us to the version of migration tools. https://github.com/department-of-veterans-affairs/va.gov-cms/pull/5486

swirtSJW commented 3 years ago

We have some data issues around West Palm beach.

  1. The facility health service CSV references West Palm Beach Medical Center but in the CMS the facility name is West Palm Beach **VA** Medical Center
  2. the system health service CSV does not have any system services for VA West Palm Beach health care

related discussion on slack https://dsva.slack.com/archives/CPQFCQKEH/p1622575005166800

swirtSJW commented 3 years ago

Found these issues with the system health service

VA Atlanta health care, HIV/hepatitis                                                                                      Could not find match for national service term.
VA Atlanta health care, Nephrology, renal and kidney                                                                       Could not find match for national service term.
VA Atlanta health care, Nutrition, food, and dietary                                                                       Could not find match for national service term.
VA Augusta health care, Low vision and blind rehabilitation                                                                Could not find match for national service term.
VA Augusta health care, Diabetes                                                                                           Could not find match for national service term.
VA Birmingham health care, PTSD treatment                                                                                  Could not find match for national service term.
VA Birmingham health care, Addiction and substance abuse treatment                                                         Could not find match for national service term.
VA Birmingham health care, Low vision and blind rehabilitation                                                             Could not find match for national service term.
VA Central Alabama health care, Nephrology, renal and kidney                                                               Could not find match for national service term.
VA Central Alabama health care, PTSD treatment                                                                             Could not find match for national service term.
VA Charlston South Carolina health care, Anesthesia                                                                        Could not find match for field_administration.              
VA Charlston South Carolina health care, Audiology and speech                                                              Could not find match for field_administration.              
VA Charlston South Carolina health care, Bariatric surgery                                                                 Could not find match for field_administration.              
VA Charlston South Carolina health care, Caregiver support                                                                 Could not find match for field_administration.              
VA Charlston South Carolina health care, Extended care and rehabilitation                                                  Could not find match for field_administration.              
VA Charlston South Carolina health care, Geriatrics                                                                        Could not find match for field_administration.              
VA Charlston South Carolina health care, Primary care                                                                      Could not find match for field_administration.              
VA Charlston South Carolina health care, Hematology/oncology                                                               Could not find match for field_administration.              
VA Charlston South Carolina health care, Homeless Veteran care                                                             Could not find match for field_administration.              
VA Charlston South Carolina health care, Palliative and hospice care                                                       Could not find match for field_administration.              
VA Charlston South Carolina health care, LGBT Veteran care                                                                 Could not find match for field_administration.              
VA Charlston South Carolina health care, Mental Health Care                                                                Could not find match for national service term.
VA Charlston South Carolina health care, Minority Veteran care                                                             Could not find match for field_administration.              
VA Charlston South Carolina health care, MOVE! weight management                                                           Could not find match for field_administration.              
VA Charlston South Carolina health care, My HealtheVet coordinator                                                         Could not find match for field_administration.              
VA Charlston South Carolina health care, Nutrition, food, and dietary                                                      Could not find match for national service term.
VA Charlston South Carolina health care, Cancer care                                                                       Could not find match for field_administration.              
VA Charlston South Carolina health care, Patient advocates                                                                 Could not find match for field_administration.              
VA Charlston South Carolina health care, Pharmacy                                                                          Could not find match for field_administration.              
VA Charlston South Carolina health care, Returning service member care                                                     Could not find match for field_administration.              
VA Charlston South Carolina health care, Sleep medicine                                                                    Could not find match for field_administration.              
VA Charlston South Carolina health care, Smoking and tobacco cessation                                                     Could not find match for field_administration.              
VA Charlston South Carolina health care, Social work                                                                       Could not find match for field_administration.              
VA Charlston South Carolina health care, Suicide prevention                                                                Could not find match for field_administration.              
VA Charlston South Carolina health care, Women Veteran care                                                                Could not find match for field_administration.              
VA Charlston South Carolina health care, Dermatology                                                                       Could not find match for field_administration.              
VA Charlston South Carolina health care, Laboratory and pathology                                                          Could not find match for field_administration.              
VA Charlston South Carolina health care, Radiology                                                                         Could not find match for field_administration.              
VA Charlston South Carolina health care, Physical therapy, occupational therapy and kinesiotherapy                         Could not find match for field_administration.              
VA Charlston South Carolina health care, Ophthalmology                                                                     Could not find match for field_administration.              
VA Charlston South Carolina health care, PTSD treatment                                                                    Could not find match for national service term.
VA Charlston South Carolina health care, Addiction and substance abuse treatment                                           Could not find match for national service term.
VA Charlston South Carolina health care, Telehealth                                                                        Could not find match for field_administration.              
VA Charlston South Carolina health care, Optometry                                                                         Could not find match for field_administration.              
VA Charlston South Carolina health care, Podiatry                                                                          Could not find match for field_administration.              
VA Charlston South Carolina health care, Gynecology                                                                        Could not find match for field_administration.              
VA Charlston South Carolina health care, Diabetes                                                                          Could not find match for national service term.
VA Charlston South Carolina health care, Neurology                                                                         Could not find match for field_administration.              
VA Charlston South Carolina health care, Psychiatry                                                                        Could not find match for field_administration.              
VA Columbia South Carolina health care, Addiction and substance abuse treatment                                            Could not find match for national service term.
VA Columbia South Carolina health care, Nutrition, food, and dietary                                                       Could not find match for national service term.
VA Columbia South Carolina health care, PTSD treatment                                                                     Could not find match for national service term.
VA Columbia South Carolina health care, Military sexual trauma                                                             Could not find match for national service term.
VA Dublin health care, Nutrition, food, and dietary                                                                        Could not find match for national service term.
VA Dublin health care, Diabetes                                                                                            Could not find match for national service term.
VA Dublin health care, HIV/hepatitis                                                                                       Could not find match for national service term.
VA Bay Pines health care, Diabetes                                                                                         Could not find match for national service term.
VA Bay Pines health care, Nutrition, food, and dietary                                                                     Could not find match for national service term.
VA Bay Pines health care, Low vision and blind rehabilitation                                                              Could not find match for national service term.
VA Bay Pines health care, PTSD treatment                                                                                   Could not find match for national service term.
VA Bay Pines health care, Military sexual trauma                                                                           Could not find match for national service term.
VA Bay Pines health care, Addiction and substance abuse treatment                                                          Could not find match for national service term.
VA Caribbean health care, Low vision and blind rehabilitation                                                              Could not find match for national service term.
VA Caribbean health care, Diabetes                                                                                         Could not find match for national service term.
VA Caribbean health care, Military sexual trauma                                                                           Could not find match for national service term.
VA Caribbean health care, Nephrology, renal and kidney                                                                     Could not find match for national service term.
VA Caribbean health care, Nutrition, food, and dietary                                                                     Could not find match for national service term.
VA Miami health care, Nutrition, food, and dietary                                                                         Could not find match for national service term.
VA Miami health care, Low vision and blind rehabilitation                                                                  Could not find match for national service term.
VA Miami health care, Addiction and substance abuse treatment                                                              Could not find match for national service term.
VA Miami health care, Nephrology, renal and kidney                                                                         Could not find match for national service term.
VA North Florida health care, Nutrition, food, and dietary                                                                 Could not find match for national service term.
VA North Florida health care, Low vision and blind rehabilitation                                                          Could not find match for national service term.
VA North Florida health care, Diabetes                                                                                     Could not find match for national service term.
VA North Florida health care, Addiction and substance abuse treatment                                                      Could not find match for national service term.
VA Orlando health care, Military sexual trauma                                                                             Could not find match for national service term.
VA Orlando health care, Nutrition, food, and dietary                                                                       Could not find match for national service term.
VA Orlando health care, Addiction and substance abuse treatment                                                            Could not find match for national service term.
VA Orlando health care, PTSD treatment                                                                                     Could not find match for national service term.
VA Orlando health care, Diabetes                                                                                           Could not find match for national service term.
VA Orlando health care, Low vision and blind rehabilitation                                                                Could not find match for national service term.
VA Tampa health care, Diabetes                                                                                             Could not find match for national service term.
VA Tampa health care, Military sexual trauma                                                                               Could not find match for national service term.
VA Tampa health care, Nutrition, food, and dietary                                                                         Could not find match for national service term.
VA Tampa health care, Low vision and blind rehabilitation                                                                  Entity lookup for field_service_name_and_descripti found nothing.

I passed the full audit to Stan and indicated that this is a blocker for auditing the facility health service data.

swirtSJW commented 3 years ago

Have gone back and forth a few more times of testing new system service data and identifying the issues with the data. Can not run the migration with the data we have.

https://dsva.slack.com/archives/CPQFCQKEH/p1622639038185700?thread_ts=1622575005.166800&cid=CPQFCQKEH

swirtSJW commented 3 years ago

I repaired the last 2 issues with the system health service data by hand. Was able to migrate system health services without incident. Processed 675 items (675 created, 0 updated, 0 failed, 0 ignored)

swirtSJW commented 3 years ago

237 errors on migrating facility service data Thread here https://dsva.slack.com/archives/CPQFCQKEH/p1623077620254300 visn-7-8-errors.txt

swirtSJW commented 3 years ago

The final csv provided hat 31 errors.

vha_534QC - Mental health care                                    , Entity lookup for field_facility_location found nothing.
vha_546BZ - Audiology and speech                                  , Entity lookup for field_facility_location found nothing.
vha_546BZ - Cardiology                                            , Entity lookup for field_facility_location found nothing.
vha_546BZ - Dental/oral surgery                                   , Entity lookup for field_facility_location found nothing.
vha_546BZ - Dermatology                                           , Entity lookup for field_facility_location found nothing.
vha_546BZ - Radiology                                             , Entity lookup for field_facility_location found nothing.
vha_546BZ - Gastroenterology                                      , Entity lookup for field_facility_location found nothing.
vha_546BZ - Internal medicine                                     , Entity lookup for field_facility_location found nothing.
vha_546BZ - Physical therapy, occupational therapy and kinesiotherapy, Entity lookup for field_facility_location found nothing.
vha_546BZ - Laboratory and pathology                              , Entity lookup for field_facility_location found nothing.
vha_546BZ - Nephrology                                            , Entity lookup for field_facility_location found nothing.
vha_546BZ - Neurology                                             , Entity lookup for field_facility_location found nothing.
vha_546BZ - Optometry                                             , Entity lookup for field_facility_location found nothing.
vha_546BZ - Orthopedics                                           , Entity lookup for field_facility_location found nothing.
vha_546BZ - Pharmacy                                              , Entity lookup for field_facility_location found nothing.
vha_546BZ - Plastic and reconstructive surgery                    , Entity lookup for field_facility_location found nothing.
vha_546BZ - Podiatry                                              , Entity lookup for field_facility_location found nothing.
vha_546BZ - Primary care                                          , Entity lookup for field_facility_location found nothing.
vha_546BZ - Rehabilitation and prosthetics                        , Entity lookup for field_facility_location found nothing.
vha_546BZ - Psychiatry                                            , Entity lookup for field_facility_location found nothing.
vha_546BZ - Psychology                                            , Entity lookup for field_facility_location found nothing.
vha_546BZ - Social work                                           , Entity lookup for field_facility_location found nothing.
vha_546BZ - Urology                                               , Entity lookup for field_facility_location found nothing.
vha_546BZ - Blind and low vision rehabilitation                   , Entity lookup for field_facility_location found nothing.
vha_546BZ - Women Veteran care                                    , Entity lookup for field_facility_location found nothing.
vha_573GF - Dental/oral surgery                                   , Entity lookup for field_facility_location found nothing.
vha_573GF - Primary care                                          , Entity lookup for field_facility_location found nothing.
vha_573GF - Advice nurse                                          , Entity lookup for field_facility_location found nothing.
vha_573GF - Pharmacy                                              , Entity lookup for field_facility_location found nothing.
vha_573GF - Psychiatry                                            , Entity lookup for field_facility_location found nothing.
vha_573GF - Surgery                                               , Entity lookup for field_facility_location found nothing.

The errors traced back to three facilities.

  1. Charleston was misspelled
  2. The failures for vha_546BZ Are due to having the encapuslation backward on the quotes. They are all supposed to be wrapped in single quotes. But for this facility you had "'William "Bill" Kling Department of Veterans Affairs Outpatient Clinic'" WHich needs to be 'William "Bill" Kling Department of Veterans Affairs Outpatient Clinic'
  3. The remaining failures were for vha_573GF which were due to bad encapsulation too. "'Sergeant Ernest I. "Boots" Thomas VA Clinic'" should be 'Sergeant Ernest I. "Boots" Thomas VA Clinic'

I repaired the errors and the migration ran fine locally. These errors were called out very early on (18 days ago) but crept back in https://github.com/department-of-veterans-affairs/va.gov-cms/pull/5424#issuecomment-846278151

swirtSJW commented 3 years ago

Stan QA'd and approved image

swirtSJW commented 3 years ago

Running this on production Baseline: image

System service migration: 675 created image

Facility Services 1491 migrated image