camerojo commented 5 months ago

As the tech team we need to capture somewhere special manual tasks that have to be carried out for a release so that we don't forget

Typically we will carry out these tasks progressively over time on staging, but all at once in production.

We should add the tasks as a task list as we carry them out on staging, then when we release into production, we just copy the task list, and mark them off as we complete them on production.

This type of issue starts in Review/QA so that we can always easily find it. It is only moved to Done when the production release is completed.

Follow the release runbook

Staging task list

Post-release:

New Translations

[x] chat.mark-as-read - Mark as Read
[x] chat.add-link - Add link
[x] chat.link-placeholder - Display text:
[x] chat.link-url - Link to:
[x] chat.url-prefix - URL including http(s):// or mailto: prefix
[x] chat.url-prefix-error - Please enter a valid URL including http(s):// or mailto: prefix
[x] chat.link-edit-btn - Edit
[x] chat.link-remove-btn - Remove
[x] chat.link-save-btn - Save
[x] chat.link-cancel-btn - Cancel
[x] chat_info.label.participants - Participants
[x] chat_info.label.purpose - Purpose
[x] chat_info.participants.candidate_prospect - Source partner and candidate in source location who is either a prospect for a job OR who has been reached out to by a partner prior to becoming a prospect for a job.
[x] chat_info.participants.all_job_candidates - All source partners, the job destination partner (recruiter), and all candidates who have received and accepted the job offer.
[x] chat_info.participants.job_creator_source_partner - Source partner associated with prospects for the job in a particular source location and destination partner (the recruiter).
[x] chat_info.participants.candidate-recruiting - Source partner, destination partner (the recruiter) and candidate in source location whom the employer has expressed interest in (ie past the CV Review stage)
[x] chat_info.participants.job_creator_all_source_partners - The job destination partner (the recruiter) plus all source partners.
[x] chat_info.purpose.candidate_prospect - Private communication between source partner and a single prospective candidate.
[x] chat_info.purpose.all_job_candidates - Communication with successful candidates, and all partners who have worked on the job.
[x] chat_info.purpose.job_creator_source_partner - Communication related to the job between the destination partner (the recruiter) and the source partner in a particular location.
[x] chat_info.purpose.candidate-recruiting - This brings the destination partner (the recruiter) into direct communication with the prospective candidate (and their source partner) for the first time.
[x] chat_info.purpose.job_creator_all_source_partners - This is how the destination partner (the recruiter) can provide all source partners with general information about the job.
[x] destinations.label.interest - Are you interested in {{country}}?
[x] destinations.label.notes - Notes
[x] destinations.title - Destinations
[x] registration.header.title.exam - Tell us about your exams
[x] registration.exam.label.exam - Exam Type
[x] registration.exam.label.score - Score
[x] registration.exam.label.year - Year
[x] registration.exam.label.notes - Notes
[x] registration.exam.label.otherexam - Other Exam
[x] registration.exam.button.add - Add
[x] registration.exam.delete.title - Delete Exam
[x] registration.exam.delete.confirmation - Are you sure you want to delete this exam?
[x] registration.exam.delete.yes - Yes
[x] registration.exam.delete.no - No
[x] profile.exams.title - Exams
[x] edit.title.exams - Exams
[x] form.placeholder.selectorexam - Select
[x] REGISTRATION.PERSONAL.NOTE.STATELESS Select Stateless if you do not have any nationality. Select Palestinian Territories if you are Palestinian.
[x] REGISTRATION.PERSONAL.LABEL.OTHER_NATIONALITY Do you have any other nationality?
[x] REGISTRATION.PERSONAL.LABEL.OTHER_NATIONALITIES Other Nationalities? (you can select more than one)
[x] joi.benefits.label Associated benefits
[x] joi.benefits.tooltip Describe any benefits that come with the job
[x] joi.cost_commitment.label Employer cost commitments
[x] joi.cost_commitment.tooltip Describe what the employer has agreed to pay for - typically visa cost and travel at a minimum.
[x] joi.EDUCATION_REQUIREMENTS.label Employer-required education or certifications (please indicate if there is flexibility in these requirements)
[x] joi.EDUCATION_REQUIREMENTS.tooltip Education or certifications that are required or desirable. Please specify which is required and which is just desirable.
[x] joi.EMPLOYMENT_EXPERIENCE.label Employer-required work experience (please indicate if there is flexibility in these requirements)*
[x] joi.EMPLOYMENT_EXPERIENCE.tooltip Work experience that is required or desirable. Please specify which is required and which is just desirable.
[x] joi.LANGUAGE_REQUIREMENTS.label Employer-required language proficiency for role (please indicate if there is flexibility in these requirements)
[x] joi.LANGUAGE_REQUIREMENTS.tooltip Minimum language proficiency that is required. Please specify written and spoken separately.
[x] joi.LOCATION.label Location of the position?
[x] joi.LOCATION.tooltip Where the job is located
[x] joi.LOCATION_DETAILS.label Details of job location (ie rural vs city, cost of living, population etc? Any links to further info)
[x] joi.LOCATION_DETAILS.tooltip More details about the location of the job.
[x] joi.MIN_SALARY.label Minimum salary (Do not enter salary UNLESS it is confirmed with the employer.)
[x] joi.MIN_SALARY.tooltip The minimum salary for the job. This is a firm commitment.
[x] joi.OCCUPATION_CODE.label Occupation Code
[x] joi.OCCUPATION_CODE.tooltip If known and if one exists enter the code of the occupation associated with the job as assigned by immigration authorities. For example, the Australian immigration authorities use "ANZSCO" codes and the Canadian immigration authorities use "NOC" codes.
[x] joi.RECRUITMENT_PROCESS.label Outline of recruitment process
[x] joi.RECRUITMENT_PROCESS.tooltip Described the employers proposed recruitment process. For example, how many interviews, and the nature of each interview - eg technical and soft skills. Is there any testing involved? etc
[x] joi.SALARY_RANGE.label Salary range (if known)
[x] joi.SALARY_RANGE.tooltip Minimum and maximum expected salaries. Please indicate whether per month or per year.
[x] joi.SKILL_REQUIREMENTS.label Specific skills to screen for (please indicate priority, or break down into essential & desired skills)
[x] joi.SKILL_REQUIREMENTS.tooltip Skills that are required or desirable. Please specify which is required and which is just desirable. You may also indicate roughly how many years of experience is desirable for each skill.
[x] joi.VISA_PATHWAYS.label List the immigration pathways that meet job & employer requirements. Please elaborate on unique requirements of each visa if applicable.
[x] joi.VISA_PATHWAYS.tooltip If known please indicate which visa pathway(s) you expect will be relevant to this particular job.
[x] STAFF_INSTRUCTION.CANDIDATE_NO_INTEREST - If candidate says that they are not interested, just change their status to "Withdrawn" and finish the intake. \</p> When you change their status you can optionally add extra information into the "Admin Note" which will be added to the automatically generated candidate note.
[x] STAFF_INSTRUCTION.CANDIDATE_UNSURE_INTEREST - If a candidate is unsure, refer them to information that could help them make their decision and suspend the intake. Ask them to contact you once they have made their decision so that you can continue the intake. \<p/> In the meantime, add a candidate note recording that the intake has been suspended. You can also add any other comments that you think are appropriate - for example, explaining why the candidate is unsure.

Cloud Cache

[x] setup AWS elasticache instance - tc-test-cache | 3 default nodes | v7.1.0 | cluster mode disabled | multi-az disabled | cache.t2.micro | single shard | auro-failover enabled | encrypt in transit disabled | auto upgrade minor versions enabled | auto backups disabled | tags: tc-test-cache
[x] configure to use same vpc as tc service (subnet name: tc-talent-test-cache)
[x] create a new security group - tc-test-cache | allow access on redis cache port | inbound custom tcp 6379 ::/0 + 0.0.0.0/0 | outbound all 0.0.0.0/0
[x] enable cloudwatch logs - slow logs + engine logs - new log group: tc-talent-test-cache
[x] create task-role with AmazonElastiCacheFullAccess permissions
[x] add redis environment variables to task definition - primary endpoint host (REDIS_HOST) + primary endpoint port (REDIS_PORT)
[x] add task-role to task definition - taskRoleArn - tctalent-test-fargate-task-role

Build Elasticsearch Index with Infra scripts

[x] run /infra/elasticsearch index and alias migration scripts

Run SystemAdminAPI fix_null_case_sfids

[x] Fixes null SF references in cases that resulted from bug issue #1244

Salesforce tidy up

See #1335

Instructions: Production task list

To be copied from staging task list above, then worked through during the production release. Translation text can be copied from Staging.

Production task list

Post-release:

[x] Test TC-SF candidate sync before Sunday evening post-release (when the scheduled method is set to run):
This System Admin Api call will run the sync for 10,000 candidates: sf-update-candidates/200-0-50
Check the logs and metrics to ensure that we're happy for the full sync to run Sunday evening BST — if any serious issues, put a hot fix in to disable the scheduled trigger.
(It's running nicely in TC Test for 4,000 candidates, I'm not actually expecting any issues!)
[x] Confirm candidate sync has successfully run on the Sunday after release - see screenshots below
[ ] Clean up general-purpose SF Contact records (see #1335) — after the candidate sync has successfully run on the Sunday after release.
[x] Set up Iress partners - Au and Uk
[x] Update Map India, IRC and Iress logos

Cloud Cache

[x] setup AWS elasticache instance - tc-prod-cache | 3 default nodes | v7.1.0 | cluster mode disabled | multi-az disabled | cache.t2.micro | single shard | auro-failover enabled | encrypt in transit disabled | auto upgrade minor versions enabled | auto backups disabled | tags: tc-prod-cache
[x] configure to use same vpc as tc service (subnet name: tc-talent-prod-cache)
[x] create a new security group - tc-prod-cache | allow access on redis cache port | inbound custom tcp 6379 ::/0 + 0.0.0.0/0 | outbound all 0.0.0.0/0
[x] enable cloudwatch logs - slow logs + engine logs - new log group: tc-talent-prod-cache
[x] create task-role with AmazonElastiCacheFullAccess permissions
[x] add redis environment variables to task definition - primary endpoint host (REDIS_HOST) + primary endpoint port (REDIS_PORT)
[x] add task-role to task definition - taskRoleArn - ecsTaskRole

camerojo commented 1 week ago

Release problems

Github Jib build

Jib failed to build. Tried again and worked!

Flyway failed

Flyway failure - prod would not start. SQL returning multiple values when just one was expected. It was failing on Flyway 317 (which copies the relocating dependents fields to the candidate opportunity). It was failing because the select was returning more than one result.

Fix 1 involved deleting visa job check records which were causing the SQL to return multiple values when just one was expected. The deleted records were stored in this TSV file - changed to txt because GitHub does not support uploads of tsv files. It has the candidate_visa_job_checks that we deleted to get prod to start.

tctalent_public_candidate_visa_job_check_6.txt

That helped - but the Flyway was still failing.

Fix 2. There was other bad data which caused the update to fail: multiple visa job checks for the same job. We executed queries to identify the bad data, then manually deleted the bad entries using the TC.

Timeout problem on the Elasticsearch scripts

Fixed by starting run and not waiting for completion. Checked for completion manually before running next scripts.

camerojo commented 1 week ago

I reimported the deleted visa job checks from the above dump file to see if I could fix the data.

There was just one multiple value being returned from update select.

It turns out that this last problem was not due to bad data but a problem in the update SQL.

The problem came from the fact that a candidate can have more than one visa check (NOT visa JOB check) carried out based on country. For example a candidate may have been checked for basic eligibility to both Australia and Canada. For example see https://tctalent.org/admin-portal/candidate/76864

In that the case the Update SQL from Flyway will fail because multiple values will be returned in the select - one for the visa check for Australia and one for the visa check for Canada. The correct update SQL needs to add an extra restriction to the select's where clause specifying that the visa check country should match the job country.

update candidate_opportunity co set relocating_dependant_ids =
    (select relocating_dependant_ids from candidate_visa_job_check cvjc
          join salesforce_job_opp jo on cvjc.job_opp_id = jo.id
          join candidate_visa_check cvc on cvjc.candidate_visa_check_id = cvc.id
          join candidate c on cvc.candidate_id = c.id
     where co.candidate_id = c.id and co.job_opp_id = cvjc.job_opp_id 
-->     and jo.country_object_id = cvc.country_id);    <--

I manually ran the above update against the production database (with the previously deleted data reimported) and it ran without error.

@cazcam34 @sadatmalik

In summary

There were actually two reasons that the Flyway failed:

Bad data in the production database left over from a previous bug. This was duplicate visa job checks for the same job. Removing the duplicates fixed the problem.
An error in the Flyway SQL which did not account for the fact that there can be multiple candidate visa checks - for different countries. Manually running corrected SQL on the production database fixed that problem.

camerojo commented 1 week ago

Forgot to add password for new email noreply@talentcatalog.net to prod Task deployment. Should have been in these release planning notes but I forgot to add it.

Updated now in prod task definition (40) - and restarted prod.

camerojo commented 1 week ago

Two showstopper bugs surfaced today which have been fixed and pushed into production.

1422 Email was broken - due to incompatible libraries - backed out last email library changes
1423 Show/hide closed opps in a submission list - which is much used and important functionality was not working. It now is.