JoeTice commented 1 year ago

Description

Upgrade the PostgreSQL database to the latest version, which requires updating PostGIS to version 3.0 as a prerequisite. This update will improve performance, security, and maintainability by ensuring compatibility with the latest features and bug fixes. The team will need to backup the current database, update PostGIS, and then proceed with the PostgreSQL update.

Tasks

[ ] Review the release notes and documentation for PostgreSQL and PostGIS to understand the new features, bug fixes, and potential compatibility issues.
[ ] Update PostGIS to version 3.0:
- [ ] Install the new PostGIS package.
- [ ] Update the PostGIS extension in the database.
- [ ] Migrate any custom functions or objects that depend on the older PostGIS version.
- [ ] Test and verify that PostGIS 3.0 is functioning correctly.
[ ] Compile development notes in a Confluence page to support future developer and support docs as needed

Acceptance Criteria

[ ] The release notes and documentation for PostgreSQL and PostGIS have been reviewed and understood.
[ ] PostGIS has been updated to version 3.0 without any errors, and all custom functions or objects have been migrated.
[ ] Confluence page documenting development notes has been created.

holdenhinkle commented 1 year ago

I spent 4 hours yesterday trying to update Postgis to v3.0.8. I'm still in the process of trying to update it.

holdenhinkle commented 1 year ago

Using the Postgres.app

I can't upgrade Postgis 2.5 to 3 in Postgres 11. I've tried everything I can, and have exhausted ChatGPT4 (it's out of options too--it's repeating itself now).

I decided to bump Postgres from 11 to 12. Postgis 3 is included with Postgres 12 in the Postgres.app.

holdenhinkle commented 1 year ago

199 test failures:

Finished in 14 minutes 36 seconds (files took 41.35 seconds to load)
9334 examples, 199 failures, 10 pending

Failed examples:

rspec './spec/lib/hca/service_spec.rb[1:1:3:9]' # HCA::Service#submit_form conformance tests properly formats no-financials-spouse for transmission
rspec './spec/lib/hca/service_spec.rb[1:1:3:7]' # HCA::Service#submit_form conformance tests properly formats no-children for transmission
rspec './spec/lib/hca/service_spec.rb[1:1:3:4]' # HCA::Service#submit_form conformance tests properly formats child-financial for transmission
rspec './spec/lib/hca/service_spec.rb[1:1:3:2]' # HCA::Service#submit_form conformance tests properly formats australian-vet for transmission
rspec './spec/lib/hca/service_spec.rb[1:1:3:13]' # HCA::Service#submit_form conformance tests properly formats spouse-financial for transmission
rspec './spec/lib/hca/service_spec.rb[1:1:3:3]' # HCA::Service#submit_form conformance tests properly formats canadian-vet for transmission
rspec './spec/lib/hca/service_spec.rb[1:1:3:1]' # HCA::Service#submit_form conformance tests properly formats all-fields for transmission
rspec './spec/lib/hca/service_spec.rb[1:1:3:6]' # HCA::Service#submit_form conformance tests properly formats missing-booleans for transmission
rspec './spec/lib/hca/service_spec.rb[1:1:3:5]' # HCA::Service#submit_form conformance tests properly formats married-no-disclosure for transmission
rspec './spec/lib/hca/service_spec.rb[1:1:3:12]' # HCA::Service#submit_form conformance tests properly formats single-no-disclosure for transmission
rspec './spec/lib/hca/service_spec.rb[1:1:3:8]' # HCA::Service#submit_form conformance tests properly formats no-financials-children for transmission
rspec './spec/lib/hca/service_spec.rb[1:1:3:11]' # HCA::Service#submit_form conformance tests properly formats only-vet-authenticated-icn for transmission
rspec './spec/lib/hca/service_spec.rb[1:1:3:10]' # HCA::Service#submit_form conformance tests properly formats no-spouse for transmission
rspec ./spec/lib/hca/service_spec.rb:111 # HCA::Service#submit_form submitting short form works
rspec ./spec/lib/hca/service_spec.rb:99 # HCA::Service#submit_form submitting sigi field works
rspec ./spec/lib/hca/service_spec.rb:84 # HCA::Service#submit_form with a medicare claim number submits successfully to hca
rspec ./spec/lib/hca/service_spec.rb:135 # HCA::Service#submit_form submitting with attachment works
rspec ./spec/lib/hca/service_spec.rb:146 # HCA::Service#submit_form submitting with attachment with a non-pdf attachment works
rspec ./spec/models/facilities/vba_facility_spec.rb:15 # Facilities::VBAFacility is a Facilities::VBAFacility object
rspec ./spec/models/facilities/vba_facility_spec.rb:27 # Facilities::VBAFacility pull_source_data returns an array of Facilities::VBAFacility objects
rspec ./spec/models/facilities/vba_facility_spec.rb:20 # Facilities::VBAFacility pull_source_data pulls data from ArcGIS endpoint
rspec ./spec/models/facilities/vba_facility_spec.rb:100 # Facilities::VBAFacility pull_source_data with single facility gets the correct classification name
rspec ./spec/models/facilities/vba_facility_spec.rb:39 # Facilities::VBAFacility pull_source_data with single facility parses hours correctly
rspec ./spec/models/facilities/vba_facility_spec.rb:82 # Facilities::VBAFacility pull_source_data with single facility parses services
rspec ./spec/models/facilities/vba_facility_spec.rb:56 # Facilities::VBAFacility pull_source_data with single facility parses phone correctly
rspec ./spec/models/facilities/vba_facility_spec.rb:67 # Facilities::VBAFacility pull_source_data with single facility parses mailing address correctly
rspec ./spec/models/facilities/vba_facility_spec.rb:48 # Facilities::VBAFacility pull_source_data with single facility parses hours correctly 2
rspec ./spec/models/facilities/vba_facility_spec.rb:73 # Facilities::VBAFacility pull_source_data with single facility parses physical address correctly
rspec ./spec/models/facilities/vba_facility_spec.rb:94 # Facilities::VBAFacility pull_source_data with single facility parses benefits values
rspec ./spec/models/facilities/vba_facility_spec.rb:88 # Facilities::VBAFacility pull_source_data with single facility parses benefits keys
rspec './spec/lib/hca/enrollment_system_spec.rb[1:33:1:1]' # HCA::EnrollmentSystem#veteran_to_demographics_info with an input of {"veteranAddress"=>{"street"=>"123 NW 5th St", "street2"=>"", "street3"=>"", "city"=>"Ontario", "country"=>"CAN", "state"=>"ON", "provinceCode"=>"ProvinceName", "postalCode"=>"21231"}, "wantsInitialVaContact"=>true, "email"=>"foo@example.com", "homePhone"=>"1231241234", "isSpanishHispanicLatino"=>true, "isWhite"=>true, "maritalStatus"=>"Married", "vaMedicalFacility"=>"608", "isEssentialAcaCoverage"=>true, "sigiIsAmericanIndian"=>true} returns {"appointmentRequestResponse"=>true, "contactInfo"=>{"addresses"=>{"address"=>{"city"=>"Ontario", "country"=>"CAN", "line1"=>"123 NW 5th St", "provinceCode"=>"ON", "postalCode"=>"21231", "addressTypeCode"=>"P"}}, "emails"=>[{"email"=>{"address"=>"foo@example.com", "type"=>"1"}}], "phones"=>{"phone"=>[{"phoneNumber"=>"1231241234", "type"=>"1"}]}}, "ethnicity"=>"2135-2", "maritalStatus"=>"M", "preferredFacility"=>"608", "races"=>{"race"=>["2106-3"]}, "acaIndicator"=>true, "indianIndicator"=>true}
rspec './spec/lib/hca/enrollment_system_spec.rb[1:33:2:1]' # HCA::EnrollmentSystem#veteran_to_demographics_info with an input of {"veteranAddress"=>{"street"=>"123 NW 5th St", "street2"=>"", "street3"=>"", "city"=>"Ontario", "country"=>"CAN", "state"=>"ON", "provinceCode"=>"ProvinceName", "postalCode"=>"21231"}, "veteranHomeAddress"=>{"street"=>"567 SW 9th Ave.", "street2"=>"#102", "street3"=>"", "city"=>"Ontario", "country"=>"CAN", "state"=>"ON", "provinceCode"=>"ProvinceName", "postalCode"=>"21231"}, "wantsInitialVaContact"=>true, "email"=>"foo@example.com", "homePhone"=>"1231241234", "isSpanishHispanicLatino"=>true, "isWhite"=>true, "maritalStatus"=>"Married", "vaMedicalFacility"=>"608", "isEssentialAcaCoverage"=>true, "sigiIsAmericanIndian"=>true} returns {"appointmentRequestResponse"=>true, "contactInfo"=>{"addresses"=>{"address"=>[{"city"=>"Ontario", "country"=>"CAN", "line1"=>"123 NW 5th St", "provinceCode"=>"ON", "postalCode"=>"21231", "addressTypeCode"=>"P"}, {"city"=>"Ontario", "country"=>"CAN", "line1"=>"567 SW 9th Ave.", "line2"=>"#102", "provinceCode"=>"ON", "postalCode"=>"21231", "addressTypeCode"=>"R"}]}, "emails"=>[{"email"=>{"address"=>"foo@example.com", "type"=>"1"}}], "phones"=>{"phone"=>[{"phoneNumber"=>"1231241234", "type"=>"1"}]}}, "ethnicity"=>"2135-2", "maritalStatus"=>"M", "preferredFacility"=>"608", "races"=>{"race"=>["2106-3"]}, "acaIndicator"=>true, "indianIndicator"=>true}
rspec ./spec/lib/hca/enrollment_system_spec.rb:1403 # HCA::EnrollmentSystem#veteran_to_save_submit_form returns the right result
rspec ./spec/lib/hca/enrollment_system_spec.rb:1416 # HCA::EnrollmentSystem#veteran_to_save_submit_form with attachments creates the right result
rspec ./spec/models/saved_claim/veteran_readiness_employment_claim_spec.rb:69 # SavedClaim::VeteranReadinessEmploymentClaim#send_to_vre when VBMS response is VBMSDownForMaintenance calls #send_to_central_mail!
rspec ./spec/models/saved_claim/veteran_readiness_employment_claim_spec.rb:76 # SavedClaim::VeteranReadinessEmploymentClaim#send_to_vre when VBMS response is VBMSDownForMaintenance does not raise an error
rspec ./spec/controllers/v0/debt_letters_controller_spec.rb:16 # V0::DebtLettersController#index lists document id and letter details for debt letters
rspec ./spec/controllers/v0/debt_letters_controller_spec.rb:25 # V0::DebtLettersController#show sends the letter pdf
rspec ./spec/models/facilities_query/radial_query_spec.rb:8 # FacilitiesQuery::RadialQuery can increase simple_cov coverage
rspec ./spec/models/facilities/vc_facility_spec.rb:10 # Facilities::VCFacility is a Facilities::VCFacility object
rspec ./spec/models/facilities/vc_facility_spec.rb:22 # Facilities::VCFacility pull_source_data returns an array of Facilities::VCFacility objects
rspec ./spec/models/facilities/vc_facility_spec.rb:15 # Facilities::VCFacility pull_source_data pulls data from ArcGIS endpoint
rspec ./spec/controllers/v0/users_controller_spec.rb:86 # V0::UsersController when logged in as a vet360 user when profile claims enabled returns a JSON user profile with claims
rspec ./spec/controllers/v0/medical_copays_controller_spec.rb:70 # V0::MedicalCopaysController#show returns 404 when record is not found
rspec ./spec/controllers/v0/medical_copays_controller_spec.rb:54 # V0::MedicalCopaysController#show returns success when record is found
rspec ./spec/controllers/v0/medical_copays_controller_spec.rb:33 # V0::MedicalCopaysController user is enrolled in VA healthcare with copays returns success
rspec ./spec/models/saved_claim/education_benefits/va1990e_spec.rb:16 # SavedClaim::EducationBenefits::VA1990e#after_submit sends confirmation email for the 1990e
rspec ./spec/requests/health_care_applications_request_spec.rb:272 # Health Care Application Integration POST create with valid params anonymously with no email set renders success
rspec ./spec/requests/health_care_applications_request_spec.rb:295 # Health Care Application Integration POST create with valid params while authenticated renders success and delete the saved form
rspec ./spec/models/facilities_query_spec.rb:7 # FacilitiesQuery#query finds facility in the bbox
rspec ./spec/models/facilities_query_spec.rb:58 # FacilitiesQuery#query  with pdx setup finds facility by state code and type
rspec ./spec/models/facilities_query_spec.rb:80 # FacilitiesQuery#query  with pdx setup finds by zip code and services
rspec ./spec/models/facilities_query_spec.rb:76 # FacilitiesQuery#query  with pdx setup finds facility by zip code and type
rspec ./spec/models/facilities_query_spec.rb:62 # FacilitiesQuery#query  with pdx setup finds by services and state code
rspec ./spec/models/facilities_query_spec.rb:72 # FacilitiesQuery#query  with pdx setup finds facilities by zip code
rspec ./spec/models/facilities_query_spec.rb:26 # FacilitiesQuery#query  with pdx setup finds facility by type
rspec ./spec/models/facilities_query_spec.rb:43 # FacilitiesQuery#query  with pdx setup finds benefit facilities by services
rspec ./spec/models/facilities_query_spec.rb:35 # FacilitiesQuery#query  with pdx setup finds health facilities by services
rspec ./spec/models/facilities_query_spec.rb:53 # FacilitiesQuery#query  with pdx setup finds facility by state code, regardless of case
rspec ./spec/lib/sidekiq/form526_job_status_tracker/job_tracker_spec.rb:48 # Sidekiq::Form526JobStatusTracker::JobTracker with an exhausted callback message submits a backup submission to Central Mail via Lighthouse Benefits Intake API, if flipper enabled
rspec ./spec/requests/swagger_spec.rb:582 # the API documentation has valid paths debts tests debt letters index validates the route
rspec ./spec/requests/swagger_spec.rb:595 # the API documentation has valid paths debts tests debt letters show validates the route
rspec ./spec/requests/swagger_spec.rb:632 # the API documentation has valid paths medical copays tests medical copays index validates the route
rspec ./spec/requests/swagger_spec.rb:645 # the API documentation has valid paths medical copays tests medical copays show validates the route
rspec ./spec/requests/swagger_spec.rb:660 # the API documentation has valid paths medical copays tests medical copays get_pdf_statement_by_id validates the route
rspec ./spec/requests/swagger_spec.rb:847 # the API documentation has valid paths HCA tests supports submitting a health care application
rspec ./spec/requests/swagger_spec.rb:2243 # the API documentation has valid paths Direct Deposit Disability Compensation GET returns a 200
rspec ./spec/requests/swagger_spec.rb:2250 # the API documentation has valid paths Direct Deposit Disability Compensation GET returns a 400
rspec ./spec/requests/swagger_spec.rb:2257 # the API documentation has valid paths Direct Deposit Disability Compensation GET returns a 401
rspec ./spec/requests/swagger_spec.rb:2264 # the API documentation has valid paths Direct Deposit Disability Compensation GET returns a 403
rspec ./spec/requests/swagger_spec.rb:2271 # the API documentation has valid paths Direct Deposit Disability Compensation GET returns a 404
rspec ./spec/requests/swagger_spec.rb:2278 # the API documentation has valid paths Direct Deposit Disability Compensation GET returns a 502
rspec ./spec/requests/swagger_spec.rb:2287 # the API documentation has valid paths Direct Deposit Disability Compensation PUT returns a 200
rspec ./spec/requests/swagger_spec.rb:2298 # the API documentation has valid paths Direct Deposit Disability Compensation PUT returns a 400
rspec ./spec/requests/swagger_spec.rb:2685 # the API documentation has valid paths profiles communication preferences supports the communication preferences update response
rspec ./spec/requests/swagger_spec.rb:2702 # the API documentation has valid paths profiles communication preferences supports the communication preferences create response
rspec ./spec/requests/swagger_spec.rb:2719 # the API documentation has valid paths profiles communication preferences supports the communication preferences index response
rspec ./spec/requests/swagger_spec.rb:3584 # the API documentation and tests all documented routes
rspec ./spec/controllers/v0/profile/direct_deposits/disability_compensations_controller_spec.rb:137 # V0::Profile::DirectDeposits::DisabilityCompensationsController#show when bad gateway returns a status of 502
rspec ./spec/controllers/v0/profile/direct_deposits/disability_compensations_controller_spec.rb:111 # V0::Profile::DirectDeposits::DisabilityCompensationsController#show when user deceased returns a status of 403
rspec ./spec/controllers/v0/profile/direct_deposits/disability_compensations_controller_spec.rb:16 # V0::Profile::DirectDeposits::DisabilityCompensationsController#show when successful returns a status of 200
rspec ./spec/controllers/v0/profile/direct_deposits/disability_compensations_controller_spec.rb:24 # V0::Profile::DirectDeposits::DisabilityCompensationsController#show when successful returns a payment account
rspec ./spec/controllers/v0/profile/direct_deposits/disability_compensations_controller_spec.rb:60 # V0::Profile::DirectDeposits::DisabilityCompensationsController#show when successful does not return errors
rspec ./spec/controllers/v0/profile/direct_deposits/disability_compensations_controller_spec.rb:39 # V0::Profile::DirectDeposits::DisabilityCompensationsController#show when successful returns control information
rspec ./spec/controllers/v0/profile/direct_deposits/disability_compensations_controller_spec.rb:101 # V0::Profile::DirectDeposits::DisabilityCompensationsController#show when not authorized returns a status of 401
rspec ./spec/controllers/v0/profile/direct_deposits/disability_compensations_controller_spec.rb:73 # V0::Profile::DirectDeposits::DisabilityCompensationsController#show when bad request returns a status of 400
rspec ./spec/controllers/v0/profile/direct_deposits/disability_compensations_controller_spec.rb:208 # V0::Profile::DirectDeposits::DisabilityCompensationsController#update when missing account type returns a validation error
rspec ./spec/controllers/v0/profile/direct_deposits/disability_compensations_controller_spec.rb:292 # V0::Profile::DirectDeposits::DisabilityCompensationsController#update when invalid scopes are provided returns a 400
rspec ./spec/controllers/v0/profile/direct_deposits/disability_compensations_controller_spec.rb:164 # V0::Profile::DirectDeposits::DisabilityCompensationsController#update when successful returns a status of 200
rspec ./spec/controllers/v0/profile/direct_deposits/disability_compensations_controller_spec.rb:272 # V0::Profile::DirectDeposits::DisabilityCompensationsController#update when unprocessable entity returns a validation error
rspec ./spec/controllers/v0/profile/direct_deposits/disability_compensations_controller_spec.rb:229 # V0::Profile::DirectDeposits::DisabilityCompensationsController#update when missing account number returns a validation error
rspec ./spec/controllers/v0/profile/direct_deposits/disability_compensations_controller_spec.rb:250 # V0::Profile::DirectDeposits::DisabilityCompensationsController#update when missing routing number returns a validation error
rspec ./spec/requests/in_progress_forms_request_spec.rb:330 # V0::InProgressFormsController with a user #update with a new form can't have non-hash formData
rspec ./spec/jobs/facilities/state_cemetery_reload_job_spec.rb:53 # Facilities::StateCemeteryReloadJob loads state cemetery data from our data file
rspec ./spec/jobs/facilities/state_cemetery_reload_job_spec.rb:39 # Facilities::StateCemeteryReloadJob purges existing state cemetery data
rspec ./spec/jobs/facilities/state_cemetery_reload_job_spec.rb:69 # Facilities::StateCemeteryReloadJob makes addresses an empty hash if address data is blank
rspec ./spec/jobs/facilities/state_cemetery_reload_job_spec.rb:46 # Facilities::StateCemeteryReloadJob does not change other cemetery data
rspec ./spec/models/facilities/vha_facility_spec.rb:10 # Facilities::VHAFacility is a Facilities::VHAFacility object
rspec ./spec/models/facilities/vha_facility_spec.rb:14 # Facilities::VHAFacility is able to have multiple DrivetimeBands
rspec ./spec/models/facilities/vha_facility_spec.rb:25 # Facilities::VHAFacility pull_source_data pulls data from a GIS endpoint
rspec ./spec/models/facilities/vha_facility_spec.rb:114 # Facilities::VHAFacility pull_source_data with single facility gets the correct classification name
rspec ./spec/models/facilities/vha_facility_spec.rb:100 # Facilities::VHAFacility pull_source_data with single facility indicates if a facility is mobile
rspec ./spec/models/facilities/vha_facility_spec.rb:67 # Facilities::VHAFacility pull_source_data with single facility parses physical address correctly
rspec ./spec/models/facilities/vha_facility_spec.rb:107 # Facilities::VHAFacility pull_source_data with single facility includes visn for vha facilities
rspec ./spec/models/facilities/vha_facility_spec.rb:38 # Facilities::VHAFacility pull_source_data with single facility parses hours correctly
rspec ./spec/models/facilities/vha_facility_spec.rb:46 # Facilities::VHAFacility pull_source_data with single facility parses phone correctly
rspec ./spec/models/facilities/vha_facility_spec.rb:94 # Facilities::VHAFacility pull_source_data with single facility includes active status for facilities
rspec ./spec/models/facilities/vha_facility_spec.rb:88 # Facilities::VHAFacility pull_source_data with single facility includes websites for facilities
rspec ./spec/models/facilities/vha_facility_spec.rb:76 # Facilities::VHAFacility pull_source_data with single facility includes just be 5 digit if zip +4 is empty
rspec ./spec/models/facilities/vha_facility_spec.rb:61 # Facilities::VHAFacility pull_source_data with single facility parses mailing address correctly
rspec ./spec/models/facilities/vha_facility_spec.rb:82 # Facilities::VHAFacility pull_source_data with single facility includes zip +4 when available
rspec ./spec/models/facilities/vha_facility_spec.rb:143 # Facilities::VHAFacility pull_source_data with single facility with mental health data adds mental health info for facilities
rspec ./spec/models/facilities/vha_facility_spec.rb:178 # Facilities::VHAFacility pull_source_data with single facility with services parses services
rspec ./spec/models/facilities/vha_facility_spec.rb:200 # Facilities::VHAFacility with_services returns a list of facilities that provide the selected services
rspec ./spec/services/medical_copays/vbs/response_data_spec.rb:163 # MedicalCopays::VBS::ResponseData#transformed_body excludes the outdated statement
rspec ./spec/services/medical_copays/vbs/response_data_spec.rb:159 # MedicalCopays::VBS::ResponseData#transformed_body transforms all the keys in an array of hashes
rspec ./spec/controllers/v0/profile/communication_preferences_controller_spec.rb:74 # V0::Profile::CommunicationPreferencesController#update updates a communication_permission
rspec './spec/controllers/v0/profile/communication_preferences_controller_spec.rb[1:2:1:1]' # V0::Profile::CommunicationPreferencesController#update with invalid params returns validation error
rspec ./spec/controllers/v0/profile/communication_preferences_controller_spec.rb:142 # V0::Profile::CommunicationPreferencesController#index returns the right data
rspec ./spec/controllers/v0/profile/communication_preferences_controller_spec.rb:113 # V0::Profile::CommunicationPreferencesController#create creates a communication permission
rspec './spec/controllers/v0/profile/communication_preferences_controller_spec.rb[1:3:1:1]' # V0::Profile::CommunicationPreferencesController#create with invalid params returns validation error
rspec ./spec/lib/debt_management_center/financial_status_report_service_spec.rb:103 # DebtManagementCenter::FinancialStatusReportService#submit_vba_fsr with valid form data sends a confirmation email
rspec ./spec/lib/bgs/form674_spec.rb:30 # BGS::Form674 calls all methods in flow
rspec ./spec/lib/bgs/form686c_spec.rb:34 # BGS::Form686c#submit form_686c_674_kitchen_sink calls all methods in flow
rspec ./spec/models/drivetime_band_spec.rb:15 # DrivetimeBand belongs to a facility
rspec ./spec/models/drivetime_band_spec.rb:26 # DrivetimeBand find_within_max_distance returns bands that intersect a point <= a max time
rspec ./spec/models/drivetime_band_spec.rb:35 # DrivetimeBand find_within_max_distance only finds for a subset facilities if a list of ids is provided
rspec ./spec/services/medical_copays/vbs/service_spec.rb:69 # MedicalCopays::VBS::Service#get_copays includes zero balance statements if available
rspec ./spec/models/saved_claim/education_benefits/va5490_spec.rb:17 # SavedClaim::EducationBenefits::VA5490#after_submit sends confirmation email for the 5490 chapter 33
rspec ./spec/models/saved_claim/education_benefits/va5490_spec.rb:38 # SavedClaim::EducationBenefits::VA5490#after_submit sends confirmation email for the 5490 chapter 35
rspec ./spec/jobs/facilities/pssg_download_spec.rb:130 # Facilities::PSSGDownload when encountering an error logs pssg download error to sentry
rspec ./spec/jobs/facilities/pssg_download_spec.rb:144 # Facilities::PSSGDownload when encountering an error continues to process bands
rspec ./spec/jobs/facilities/pssg_download_spec.rb:93 # Facilities::PSSGDownload matching facility does not populate facility with drive time data when there are no rings
rspec ./spec/jobs/facilities/pssg_download_spec.rb:52 # Facilities::PSSGDownload matching facility populates facility with drive time data overriding existing band
rspec ./spec/jobs/facilities/pssg_download_spec.rb:80 # Facilities::PSSGDownload matching facility rounds bands with float bounds
rspec ./spec/jobs/facilities/pssg_download_spec.rb:101 # Facilities::PSSGDownload matching facility leaves facility with original drive time band
rspec ./spec/jobs/facilities/pssg_download_spec.rb:68 # Facilities::PSSGDownload matching facility populates facility with drive time data
rspec ./spec/requests/v0/user_transition_availabilities_controller_spec.rb:40 # V0::UserTransitionAvailabilitiesController /v0/user_transition_availabilities when Flipper organic_conversion_experiment is enabled When the user does not have associated AVC or IVC returns true for organic adoption modal
rspec ./spec/models/saved_claim/education_benefits/va1990_spec.rb:39 # SavedClaim::EducationBenefits::VA1990#after_submit sends confirmation email for the 1990 with relinquished
rspec ./spec/models/saved_claim/education_benefits/va1990_spec.rb:17 # SavedClaim::EducationBenefits::VA1990#after_submit sends confirmation email for the 1990 chapter 33
rspec ./spec/models/facilities_query/ids_query_spec.rb:8 # FacilitiesQuery::IdsQuery can increase simple_cov coverage
rspec ./spec/requests/medical_copays_request_spec.rb:15 # medical_copays GET medical_copays#index returns a formatted hash response
rspec ./spec/controllers/v0/profile/payment_history_controller_spec.rb:27 # V0::Profile::PaymentHistoryController#index with mixed payments and return payments returns both
rspec ./spec/controllers/v0/profile/payment_history_controller_spec.rb:10 # V0::Profile::PaymentHistoryController#index with only regular payments returns only payments and no return payments
rspec ./spec/lib/debt_management_center/debt_letter_downloader_spec.rb:109 # DebtManagementCenter::DebtLetterDownloader VBMS is down #list_letters notifies Sentry upon downstream service error
rspec ./spec/lib/debt_management_center/debt_letter_downloader_spec.rb:87 # DebtManagementCenter::DebtLetterDownloader VBMS is available #file_name with a proper document id returns a filename
rspec ./spec/lib/debt_management_center/debt_letter_downloader_spec.rb:95 # DebtManagementCenter::DebtLetterDownloader VBMS is available #file_name without a proper document id raises an unauthorized error
rspec ./spec/lib/debt_management_center/debt_letter_downloader_spec.rb:71 # DebtManagementCenter::DebtLetterDownloader VBMS is available #get_letter with a document not in the users folder raises an unauthorized error
rspec ./spec/lib/debt_management_center/debt_letter_downloader_spec.rb:65 # DebtManagementCenter::DebtLetterDownloader VBMS is available #get_letter with a document in the users folder downloads a debt letter
rspec ./spec/lib/debt_management_center/debt_letter_downloader_spec.rb:78 # DebtManagementCenter::DebtLetterDownloader VBMS is available #list_letters gets letter ids and descriptions
rspec ./spec/jobs/evss/disability_compensation_form/submit_form526_all_claim_spec.rb:240 # EVSS::DisabilityCompensationForm::SubmitForm526AllClaim.perform_async with a client error sets the job_status to "non_retryable_error"
rspec ./spec/middleware/rack/attack_spec.rb:91 # Rack::Attack check_in/ip when more than 10 requests when POST endpoint throttles with status 429
rspec ./spec/middleware/rack/attack_spec.rb:69 # Rack::Attack check_in/ip when more than 10 requests when GET endpoint throttles with status 429
rspec ./spec/jobs/facilities/facility_location_download_job_spec.rb:180 # Facilities::FacilityLocationDownloadJob VHA Facilities adds data that does not exist in the db
rspec ./spec/jobs/facilities/facility_location_download_job_spec.rb:172 # Facilities::FacilityLocationDownloadJob VHA Facilities retrieves and persists facilities data
rspec ./spec/jobs/facilities/facility_location_download_job_spec.rb:231 # Facilities::FacilityLocationDownloadJob with wait time data has the wait time indicated services
rspec ./spec/jobs/facilities/facility_location_download_job_spec.rb:242 # Facilities::FacilityLocationDownloadJob with wait time data standardizes closed days to "Closed"
rspec ./spec/jobs/facilities/facility_location_download_job_spec.rb:116 # Facilities::FacilityLocationDownloadJob VBA Facilities adds data that does not exist in the db
rspec ./spec/jobs/facilities/facility_location_download_job_spec.rb:127 # Facilities::FacilityLocationDownloadJob VBA Facilities standardizes closed days to "Closed"
rspec ./spec/jobs/facilities/facility_location_download_job_spec.rb:105 # Facilities::FacilityLocationDownloadJob VBA Facilities indicates Pensions for appropriate facilities
rspec ./spec/jobs/facilities/facility_location_download_job_spec.rb:97 # Facilities::FacilityLocationDownloadJob VBA Facilities retrieves and persists facilities data
rspec ./spec/jobs/facilities/facility_location_download_job_spec.rb:158 # Facilities::FacilityLocationDownloadJob VC Facilities standardizes closed days to "Closed"
rspec ./spec/jobs/facilities/facility_location_download_job_spec.rb:147 # Facilities::FacilityLocationDownloadJob VC Facilities adds data that does not exist in the db
rspec ./spec/jobs/facilities/facility_location_download_job_spec.rb:139 # Facilities::FacilityLocationDownloadJob VC Facilities retrieves and persists facilities data
rspec ./spec/jobs/facilities/facility_location_download_job_spec.rb:85 # Facilities::FacilityLocationDownloadJob NCA Facilities standardizes closed days to "Closed"
rspec ./spec/jobs/facilities/facility_location_download_job_spec.rb:56 # Facilities::FacilityLocationDownloadJob NCA Facilities removes data from the db that does not exist in the source
rspec ./spec/jobs/facilities/facility_location_download_job_spec.rb:45 # Facilities::FacilityLocationDownloadJob NCA Facilities adds data that does not exist in the db
rspec ./spec/jobs/facilities/facility_location_download_job_spec.rb:23 # Facilities::FacilityLocationDownloadJob NCA Facilities does not update data with the same fingerprint
rspec ./spec/jobs/facilities/facility_location_download_job_spec.rb:34 # Facilities::FacilityLocationDownloadJob NCA Facilities does update data with a changed fingerprint
rspec ./spec/jobs/facilities/facility_location_download_job_spec.rb:15 # Facilities::FacilityLocationDownloadJob NCA Facilities retrieves and persists facilities data
rspec ./spec/jobs/facilities/facility_location_download_job_spec.rb:73 # Facilities::FacilityLocationDownloadJob NCA Facilities source data returns empty does not delete cached data
rspec ./spec/models/facilities/nca_facility_spec.rb:10 # Facilities::NCAFacility is an Facilities::NCAFacility object
rspec ./spec/models/facilities/nca_facility_spec.rb:22 # Facilities::NCAFacility pull_source_data returns an array of Facilities::NCAFacility objects
rspec ./spec/models/facilities/nca_facility_spec.rb:15 # Facilities::NCAFacility pull_source_data pulls data from ArcGIS endpoint
rspec ./spec/models/facilities/nca_facility_spec.rb:34 # Facilities::NCAFacility pull_source_data with single facility gets the correct classification name
rspec ./spec/models/facilities/nca_facility_spec.rb:40 # Facilities::NCAFacility pull_source_data with single facility includes websites for facilities
rspec ./spec/services/medical_copays/request_spec.rb:45 # MedicalCopays::Request#headers has request headers
rspec ./spec/lib/efolder/service_spec.rb:78 # Efolder::Service#list_documents lists document ids and descriptions
rspec ./spec/lib/efolder/service_spec.rb:55 # Efolder::Service#get_document with a document in the users folder downloads a document
rspec ./spec/lib/efolder/service_spec.rb:67 # Efolder::Service#get_document with a document not in the users folder raises an unauthorized error
rspec ./spec/policies/medical_copays_policy_spec.rb:19 # MedicalCopaysPolicy access? with the mcp feature enabled with a user who has the required mcp attributes increments statsD success
rspec ./spec/lib/va_profile/military_personnel/service_spec.rb:59 # VAProfile::MilitaryPersonnel::Service#get_service_history when successful sorts service history episodes
rspec ./spec/lib/va_profile/military_personnel/service_spec.rb:45 # VAProfile::MilitaryPersonnel::Service#get_service_history when successful returns multiple service history episodes
rspec ./spec/services/acceptable_verified_credential_adoption_service_spec.rb:50 # AcceptableVerifiedCredentialAdoptionService.perform when Flipper organic_conversion_experiment is enabled User is dslogon authenticated When user has no avc/ivc logs attempt
rspec ./spec/services/acceptable_verified_credential_adoption_service_spec.rb:44 # AcceptableVerifiedCredentialAdoptionService.perform when Flipper organic_conversion_experiment is enabled User is dslogon authenticated When user has no avc/ivc hash returns true
rspec ./spec/services/acceptable_verified_credential_adoption_service_spec.rb:108 # AcceptableVerifiedCredentialAdoptionService.perform when Flipper organic_conversion_experiment is enabled When user has no avc/ivc logs attempt
rspec ./spec/services/acceptable_verified_credential_adoption_service_spec.rb:102 # AcceptableVerifiedCredentialAdoptionService.perform when Flipper organic_conversion_experiment is enabled When user has no avc/ivc hash returns true
rspec ./spec/requests/user_request_spec.rb:189 # Fetching user data GET /v0/user - when an LOA 1 user is logged in returns proper json
rspec ./spec/requests/user_request_spec.rb:219 # Fetching user data GET /v0/user - when an LOA 1 user is logged in with camel inflection returns proper json
rspec ./spec/requests/user_request_spec.rb:22 # Fetching user data GET /v0/user - when an LOA 3 user is logged in dont stub mpi GET /v0/user - returns proper json
rspec ./spec/requests/user_request_spec.rb:77 # Fetching user data GET /v0/user - when an LOA 3 user is logged in with camel header inflection GET /v0/user - returns proper json
rspec ./spec/models/base_facility_spec.rb:42 # BaseFacility VCFacility saves and retrieve all attributes and they should match the original object
rspec ./spec/models/base_facility_spec.rb:201 # BaseFacility#find_facility_by_id has hours that are sorted by day
rspec ./spec/models/base_facility_spec.rb:197 # BaseFacility#find_facility_by_id finds facility by id
rspec ./spec/models/base_facility_spec.rb:152 # BaseFacility VBAFacility saves and retrieve all attributes and they should match the original object
rspec ./spec/models/base_facility_spec.rb:186 # BaseFacility NCAFacility saves and retrieve all attributes and they should match the original object
rspec ./spec/models/base_facility_spec.rb:105 # BaseFacility VHAFacility saves and retrieve all attributes and they should match the original object
rspec ./spec/services/rapid_ready_for_decision/sidekiq_job_selector_spec.rb:21 # RapidReadyForDecision::SidekiqJobSelector#sidekiq_job when given single-issue hypertension claim for increase submission returns RRD processor
rspec ./spec/services/rapid_ready_for_decision/sidekiq_job_selector_spec.rb:31 # RapidReadyForDecision::SidekiqJobSelector#sidekiq_job when given single-issue asthma claim for increase submission returns Form526AsthmaJob

Randomized with seed 48785

holdenhinkle commented 1 year ago

postgis/postgis:12-3.3 vs postgres:12-alpine images:

The postgis/postgis:12-3.3 image includes PostGIS extensions for working with geospatial data, while the postgres:12-alpine image does not include it by default.
The postgis/postgis:12-3.3 image has a larger size than the postgres:12-alpine image, because it includes PostGIS and its dependencies.
The postgres:12-alpine image is a official PostgreSQL image, maintained by the PostgreSQL community, while the postgis/postgis:12-3.3 image is a third-party image that includes PostGIS and is also based on the official PostgreSQL image.

Upgrade to Postgres 12 and Postgis 3

PR - https://github.com/department-of-veterans-affairs/vets-api/pull/12506

When upgrading the image, I got the following error:

vets-api-postgres-1 | 2023-04-27 20:17:59.458 UTC [1] DETAIL: The data directory was initialized by PostgreSQL version 11, which is not compatible with this version 12.14. vets-api-postgres-1 exited with code 1

To resolve this issue, you can create a new data directory for PostgreSQL 12.14, or you can try upgrading the existing data directory.

I WENT WITH OPTION 2:

Option 1: Create a new data directory

Stop the running containers using docker-compose down.
Remove or rename the existing data directory (./data) to create a new one, e.g., mv ./data ./data_old.
Start the containers again with docker-compose up. This will create a new data directory with the correct PostgreSQL version.
Keep in mind that this option will result in a new, empty database. If you need to preserve the data from the old database, consider Option 2.

Option 2: Upgrade the existing data directory

Upgrading the existing data directory requires the use of pg_upgrade. This can be done using Docker containers to avoid installing PostgreSQL locally. Here's how:

Stop the running containers using docker-compose down.
Start a temporary container with the old PostgreSQL version (11) and mount the data directory: docker run --rm --name pg_old -v $(pwd)/data:/var/lib/postgresql/data -p 54320:5432 -e POSTGRES_PASSWORD=password -d postgres:11
Start a temporary container with the new PostgreSQL version (12) and create a new, empty data directory: docker run --rm --name pg_new -v $(pwd)/data_new:/var/lib/postgresql/data -p 54321:5432 -e POSTGRES_PASSWORD=password -d postgres:12
Install the pg_upgrade tool in the old PostgreSQL container: docker exec -it pg_old bash -c "apt-get update && apt-get install -y postgresql-11"
Install the pg_upgrade tool in the new PostgreSQL container: docker exec -it pg_new bash -c "apt-get update && apt-get install -y postgresql-12"
Run pg_upgrade:

First, create a temporary directory inside the pg_new container: docker exec -it -u postgres pg_new mkdir /tmp/pg_upgrade_logs Run the pg_upgrade command with the output redirected to the new log file: docker exec -it -u postgres pg_new bash -c "pg_upgrade -b /usr/lib/postgresql/11/bin -B /usr/lib/postgresql/12/bin -d /var/lib/postgresql/data_old -D /var/lib/postgresql/data_new -o '-c config_file=/etc/postgresql/11/main/postgresql.conf' -O '-c config_file=/etc/postgresql/12/main/postgresql.conf' > /tmp/pg_upgrade_logs/pg_upgrade.log 2>&1"

After running the command, you can check the content of the log file using: docker exec -it -u postgres pg_new cat /tmp/pg_upgrade_logs/pg_upgrade.log

This will help you keep track of the upgrade process and check for any errors or issues.

Replace the old data directory with the upgraded one:
```
rm -rf ./data
mv ./data_new ./data
```

I'm going to go through this process again because somehow db/schema.rb was updated which we don't want:

holdenhinkle commented 1 year ago

I recreated this branch, this time without the following migration which removed fields from several tables (which we don't want):

class DisableThenEnablePostgisExtension < ActiveRecord::Migration[6.1]
  def change
    disable_extension "postgis"
    enable_extension "postgis"
  end
end

Draft PR - https://github.com/department-of-veterans-affairs/vets-api/pull/12608

I deleted the data directory and it was recreated when I ran make up.

vets-api is successfully running via docker:

More updates coming...

holdenhinkle commented 1 year ago

I'm able to run rspec tests natively. Confirming the correct version of Postgres and Postgis:

√ vets-api % psql -d vets-api      
psql (14.7 (Homebrew), server 12.14)
Type "help" for help.

vets-api=# SELECT version();
                                                   version                                                    
--------------------------------------------------------------------------------------------------------------
 PostgreSQL 12.14 on x86_64-apple-darwin16.7.0, compiled by Apple LLVM version 8.1.0 (clang-802.0.42), 64-bit
(1 row)

vets-api=# SELECT PostGIS_full_version();
                                                                        postgis_full_version                                                                         
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
 POSTGIS="3.0.8 9b4c2b9" [EXTENSION] PGSQL="120" GEOS="3.8.3-CAPI-1.13.4" PROJ="6.2.1" LIBXML="2.9.14" LIBJSON="0.13.1" LIBPROTOBUF="1.3.3" WAGYU="0.4.3 (Internal)"
(1 row)

However, I can't run them in Docker:

√ vets-api % make spec
WARN[0000] Found orphan containers ([vets-api-web-1 vets-api-worker-1 vets-api-clamav-1]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up. 
[+] Running 2/0
 ⠿ Container vets-api-postgres-1  Running                                                                                                    0.0s
 ⠿ Container vets-api-redis-1     Running                                                                                                    0.0s
The Gemfile's dependencies are satisfied
/usr/local/bundle/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32: warning: ⛔️ WARNING: Sidekiq testing API enabled, but this is not the test environment.  Your jobs will not go to Redis.

An error occurred while loading ./spec/jobs/webhooks/callback_url_job_spec.rb.
Failure/Error: @supported_events << event

FrozenError:
  can't modify frozen Array: ["gov.va.developer.benefits-intake.status_change"]
# ./lib/webhooks/utilities.rb:48:in `register_event'
# ./lib/webhooks/utilities.rb:65:in `block in register_events'
# ./lib/webhooks/utilities.rb:64:in `each'
# ./lib/webhooks/utilities.rb:64:in `register_events'
# ./spec/jobs/webhooks/registrations.rb:8:in `<module:Registrations>'
# ./spec/jobs/webhooks/registrations.rb:3:in `<top (required)>'
# ./spec/jobs/webhooks/callback_url_job_spec.rb:7:in `require_relative'
# ./spec/jobs/webhooks/callback_url_job_spec.rb:7:in `<top (required)>'
/srv/vets-api/src/spec/jobs/webhooks/registrations.rb:5: warning: already initialized constant Registrations::TEST_EVENT
/srv/vets-api/src/spec/jobs/webhooks/registrations.rb:5: warning: previous definition of TEST_EVENT was here
/srv/vets-api/src/spec/jobs/webhooks/registrations.rb:6: warning: already initialized constant Registrations::API_NAME
/srv/vets-api/src/spec/jobs/webhooks/registrations.rb:6: warning: previous definition of API_NAME was here
/srv/vets-api/src/spec/jobs/webhooks/registrations.rb:7: warning: already initialized constant Registrations::MAX_RETRIES
/srv/vets-api/src/spec/jobs/webhooks/registrations.rb:7: warning: previous definition of MAX_RETRIES was here

An error occurred while loading ./spec/jobs/webhooks/notifications_job_spec.rb.
Failure/Error: @supported_events << event

FrozenError:
  can't modify frozen Array: ["gov.va.developer.benefits-intake.status_change"]
# ./lib/webhooks/utilities.rb:48:in `register_event'
# ./lib/webhooks/utilities.rb:65:in `block in register_events'
# ./lib/webhooks/utilities.rb:64:in `each'
# ./lib/webhooks/utilities.rb:64:in `register_events'
# ./spec/jobs/webhooks/registrations.rb:8:in `<module:Registrations>'
# ./spec/jobs/webhooks/registrations.rb:3:in `<top (required)>'
# ./spec/jobs/webhooks/notifications_job_spec.rb:7:in `require_relative'
# ./spec/jobs/webhooks/notifications_job_spec.rb:7:in `<top (required)>'
WARNING: Shared example group 'V1 Facility Locator' has been previously defined at:
  /srv/vets-api/src/spec/requests/v1/apidoc/shared_examples/facility_locator_spec.rb:5
...and you are now defining it at:
  /srv/vets-api/src/spec/requests/v1/apidoc/shared_examples/facility_locator_spec.rb:5
The new definition will overwrite the original one.
WARNING: Shared example group 'paginated request from params with expected IDs' has been previously defined at:
  /srv/vets-api/src/spec/requests/v1/facilities/va_request_spec.rb:5
...and you are now defining it at:
  /srv/vets-api/src/modules/facilities_api/spec/requests/facilities_api/v1/va_request_spec.rb:5
The new definition will overwrite the original one.
Run options:
  include {:focus=>true}
  exclude {:wip_doc_sc_v2_potential_pact_act=>true}

All examples were filtered out; ignoring {:focus=>true}

Finished in 0.00009 seconds (files took 54.92 seconds to load)
0 examples, 0 failures, 2 errors occurred outside of examples

Coverage report generated for RSpec to /srv/vets-api/src/coverage. 21245 / 77170 LOC (27.53%) covered.
Stopped processing SimpleCov as a previous error not related to SimpleCov has been detected
make: *** [spec] Error 1
?2 vets-api %

holdenhinkle commented 1 year ago

The problem with running tests in Docker is not related to the postgres image change. I reverted the change locally and got the same error running the tests using the docker-compose.yml in main. So the problem must be my environment?

holdenhinkle commented 1 year ago

I had to uncomment this line to run the tests - https://github.com/department-of-veterans-affairs/vets-api/blob/master/lib/webhooks/utilities.rb#L137

And they're taking hours to run. 3 hours in: iScreen Shoter - 20230511143459125.jpg

holdenhinkle commented 1 year ago

All the postgres upgrades should be branched off of the k8s branch.

There are a subset of tests that consistently fail in the k8s branch, so we created a ticket to fix them - https://app.zenhub.com/workspaces/platform-cop-backlog-6359535e384ff5a473132130/issues/gh/department-of-veterans-affairs/va.gov-team/58410

PR to simply swap out the mdallion postgres image with the equivalient postgis image:

Branched off k8s - https://github.com/department-of-veterans-affairs/vets-api/pull/12639

Incidentally, the branch to upgrade Postgres 11 to 12 and Post 2.5 to 3 against master has all green checks :-)

Branched off Master - https://github.com/department-of-veterans-affairs/vets-api/pull/12664

pjhill commented 1 year ago

@holdenhinkle -- Is this going to be considered part of the Postgres upgrade project?

holdenhinkle commented 1 year ago

For details regarding everything I learned from working on this ticket, please see vets-api Postgres Upgrade, From v11 To v15

Closing this as OBE.

holdenhinkle commented 1 year ago

One last thing: I struggled with my local env a lot when trying to upgrade Postgres. The failing tests noted in this ticket failed locally because my env was messed up but they pass in CI.

Please ignore the comments in this ticket, and see that confluence doc linked in that previous comment - https://vfs.atlassian.net/wiki/spaces/TT4/pages/2636054965/vets-api+Postgres+Upgrade+From+v11+To+v15

department-of-veterans-affairs / va.gov-team

Update PostGIS to 3.0 #56819

Description

Tasks

Acceptance Criteria