Test TransformAgency ELT process in dev/staging

[chouinar]

Summary

Test process, fix any issues, create tickets as follow-ups.

Acceptance criteria

• [ ] Note down observations

[chouinar]

Created several follow-up tickets after doing one round of testing yesterday. Two issues identified:

• Need to handle top-level agency names
• Need to handle agencies without certain fields unconfigured that we don't intend to use anyways

https://github.com/HHS/simpler-grants-gov/issues/2490 - add top-level agency name to the API response https://github.com/HHS/simpler-grants-gov/issues/2491 - add top-level agency info to the DB model https://github.com/HHS/simpler-grants-gov/issues/2492 - populate the top-level agency info in the transform process https://github.com/HHS/simpler-grants-gov/issues/2493 - ease up validation on some agency transformations

[chouinar]

Here are the agencies that failed to process when running in dev:

1 | GDIT-PDCM-LEV2 | Cannot create agency GDIT-PDCM-LEV2 as required fields are missing: AgencyContactCity
-- | -- | --
2 | GDIT-AG | Cannot create agency GDIT-AG as required fields are missing: AgencyContactCity
3 | GDIT-311 | Cannot create agency GDIT-311 as required fields are missing: AgencyContactCity
4 | GDIT-21 | Cannot create agency GDIT-21 as required fields are missing: AgencyContactCity
5 | GDIT-31 | Cannot create agency GDIT-31 as required fields are missing: AgencyContactCity
6 | GDIT-2 | Cannot create agency GDIT-2 as required fields are missing: AgencyContactCity
7 | GDIT-06282011 | Cannot create agency GDIT-06282011 as required fields are missing: AgencyContactCity
8 | GDIT-070120111 | Cannot create agency GDIT-070120111 as required fields are missing: AgencyContactCity
9 | GDIT-07012011 | Cannot create agency GDIT-07012011 as required fields are missing: AgencyContactCity
10 | dxprobeyd0000BA | Cannot create agency dxprobeyd0000BA as required fields are missing: ldapGp
11 | dxprobeyd00000F | Cannot create agency dxprobeyd00000F as required fields are missing: ldapGp
12 | dxprobeyd00006C | Cannot create agency dxprobeyd00006C as required fields are missing: ldapGp
13 |   | Cannot create agency as required fields are missing: ldapGp

Only GDIT-PDCM-LEV2 has any opportunities associated with it, and just 2 old ones. The rest of these are clearly test agencies. Odd that several of them have no contact city, that wasn't an issue in the prod data. While I'm fine adjusting to not have an ldapGp since we don't use it, I would prefer the contact info be intact. I could work around these old agencies if I manually created them + set the transformed flag. Just to get the errors to go away. I'll first fix the ldapGp issue though.