GSA-TTS / FAC

GSA's Federal Audit Clearinghouse
Other
18 stars 5 forks source link

3980 address ace reports migration #3986

Closed sambodeme closed 2 weeks ago

sambodeme commented 2 weeks ago

Description

The FAC app currently lacks the logic to handle ACE audit reports, making it infeasible to migrate historical ACE reports using the same algorithm as for other reports. Consequently, the team has decided to make ACE reports available for download in CSV format. Additionally, users will have the ability to download PDF reports. However, these reports are stored on an S3 bucket and require a temporary token for access, which is currently only obtainable through the FAC app. To address this, users will need to interact with the FAC app to obtain the appropriate links to download the PDF reports. To facilitate access, we will create a generic SAC object for each ACE report with basic information on the report. When users attempt to access ACE reports from the static web and wish to download a PDF, they will be directed to the corresponding disseminated SAC object, which will provide them with the correct download link. This PR involves updating the logic in census_historical_migration module to identify ACE reports and generate the required SAC objects, along with the dissemination processes needed to make these available to the public.

How to test

  1. Launch the stack from the main branch.
  2. Attempt to migrate the historical report with audit_year=2022 and dbkey=254036 and notice how it fails due to "ACE report type"
  3. Switch to this PR branch and re-run the migration for the same report and observe that it processes properly.

PR checklist: submitters

PR checklist: reviewers

The larger the PR, the stricter we should be about these points.

github-actions[bot] commented 2 weeks ago

Terraform plan for dev

Plan: 4 to add, 2 to change, 2 to destroy. ```diff Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create !~ update in-place -/+ destroy and then create replacement <= read (data resources) Terraform will perform the following actions: # module.dev.cloudfoundry_network_policy.logshipper-network-policy will be updated in-place !~ resource "cloudfoundry_network_policy" "logshipper-network-policy" { id = "f188b467-82e6-e5d6-bae1-138086a288c5" + policy { + destination_app = "a6eb3870-2769-4aa9-a8f2-f1a31939f563" + port = "61443" + protocol = "tcp" + source_app = (known after apply) } } # module.dev.module.cg-logshipper.cloudfoundry_app.cg_logshipper_app will be created + resource "cloudfoundry_app" "cg_logshipper_app" { + buildpack = (known after apply) + buildpacks = [ + "https://github.com/cloudfoundry/apt-buildpack", + "nginx_buildpack", ] + disk_quota = 256 + enable_ssh = (known after apply) + environment = (sensitive value) + health_check_http_endpoint = (known after apply) + health_check_invocation_timeout = (known after apply) + health_check_timeout = (known after apply) + health_check_type = "process" + id = (known after apply) + id_bg = (known after apply) + instances = 1 + memory = 256 + name = "logshipper" + path = "../shared/modules/cg-logshipper/logshipper.zip" + ports = (known after apply) + space = "06525ba3-19c2-451b-96e9-ea4a9134e8b9" + stack = (known after apply) + stopped = false + strategy = "rolling" + timeout = 180 + routes { + port = (known after apply) + route = (known after apply) } + service_binding { + service_instance = "03df74b7-065a-46df-9a85-0bac201bf36c" } + service_binding { + service_instance = "29c42cfd-0722-4152-858e-d16521f1a694" } + service_binding { + service_instance = "d79b8e4c-c1f5-4711-8172-5da3f9298cc4" } } # module.dev.module.cg-logshipper.cloudfoundry_route.logshipper will be created + resource "cloudfoundry_route" "logshipper" { + domain = "50ba3f69-cd54-4963-9172-14f3334b479e" + endpoint = (known after apply) + hostname = "fac-dev-logshipper" + id = (known after apply) + port = (known after apply) + space = "06525ba3-19c2-451b-96e9-ea4a9134e8b9" } # module.dev.module.file_scanner_clamav.data.cloudfoundry_app.app will be read during apply # (config refers to values not yet known) <= data "cloudfoundry_app" "app" { + buildpack = (known after apply) + command = (known after apply) + disk_quota = (known after apply) + enable_ssh = (known after apply) + environment = (sensitive value) + health_check_http_endpoint = (known after apply) + health_check_timeout = (known after apply) + health_check_type = (known after apply) + id = (known after apply) + instances = (known after apply) + memory = (known after apply) + name = (known after apply) + name_or_id = "****************" + space = (known after apply) + stack = (known after apply) + state = (known after apply) } # module.dev.module.file_scanner_clamav.data.cloudfoundry_domain.internal will be read during apply # (depends on a resource or a module with changes pending) <= data "cloudfoundry_domain" "internal" { + domain = (known after apply) + id = (known after apply) + internal = (known after apply) + name = "apps.internal" + org = (known after apply) } # module.dev.module.file_scanner_clamav.data.cloudfoundry_space.space will be read during apply # (depends on a resource or a module with changes pending) <= data "cloudfoundry_space" "space" { + id = (known after apply) + name = "dev" + org = (known after apply) + org_name = "gsa-tts-oros-fac" + quota = (known after apply) } # module.dev.module.file_scanner_clamav.cloudfoundry_app.clamav_api must be replaced -/+ resource "cloudfoundry_app" "clamav_api" { + buildpack = (known after apply) !~ enable_ssh = true -> (known after apply) + health_check_http_endpoint = (known after apply) !~ health_check_invocation_timeout = 0 -> (known after apply) !~ health_check_timeout = 0 -> (known after apply) !~ id = "************************************" -> (known after apply) !~ id_bg = "************************************" -> (known after apply) - labels = {} -> null name = "fac-av-dev-fs" !~ ports = [] -> (known after apply) !~ space = "06525ba3-19c2-451b-96e9-ea4a9134e8b9" # forces replacement -> (known after apply) # forces replacement + stack = (known after apply) # (9 unchanged attributes hidden) - routes { - port = 0 -> null - route = "d471b94d-6031-4589-85f7-64a50f3f9dd2" -> null } + routes { + port = (known after apply) + route = (known after apply) } } # module.dev.module.file_scanner_clamav.cloudfoundry_network_policy.clamav_routing will be updated in-place !~ resource "cloudfoundry_network_policy" "clamav_routing" { id = "a12e15b9-a189-47d2-cd69-95a38056d3b5" - policy { - destination_app = "2a10693f-1166-465a-93c9-8291585eeac5" -> null - port = "61443" -> null - protocol = "tcp" -> null - source_app = "bfcc67a1-8ace-45fe-b472-0594da8ab39b" -> null } + policy { + destination_app = (known after apply) + port = "61443" + protocol = "tcp" + source_app = (known after apply) } } # module.dev.module.file_scanner_clamav.cloudfoundry_route.clamav_route must be replaced -/+ resource "cloudfoundry_route" "clamav_route" { !~ domain = "26df58ef-0c0d-4997-b68b-8defb7b3998b" # forces replacement -> (known after apply) # forces replacement !~ endpoint = "fac-av-dev-fs.apps.internal" -> (known after apply) !~ id = "************************************" -> (known after apply) + port = (known after apply) !~ space = "06525ba3-19c2-451b-96e9-ea4a9134e8b9" # forces replacement -> (known after apply) # forces replacement # (2 unchanged attributes hidden) } Plan: 4 to add, 2 to change, 2 to destroy. Warning: Argument is deprecated with module.dev.module.cg-logshipper.module.s3-logshipper-storage.cloudfoundry_service_instance.bucket, on /tmp/terraform-data-dir/modules/dev.cg-logshipper.s3-logshipper-storage/s3/main.tf line 14, in resource "cloudfoundry_service_instance" "bucket": 14: recursive_delete = var.recursive_delete Since CF API v3, recursive delete is always done on the cloudcontroller side. This will be removed in future releases (and 4 more similar warnings elsewhere) ```

:white_check_mark: Plan applied in Deploy to Development and Management Environment #719

github-actions[bot] commented 2 weeks ago

Terraform plan for meta

No changes. Your infrastructure matches the configuration. ``` No changes. Your infrastructure matches the configuration. Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed. Warning: Argument is deprecated with module.s3-backups.cloudfoundry_service_instance.bucket, on /tmp/terraform-data-dir/modules/s3-backups/s3/main.tf line 14, in resource "cloudfoundry_service_instance" "bucket": 14: recursive_delete = var.recursive_delete Since CF API v3, recursive delete is always done on the cloudcontroller side. This will be removed in future releases ```

:white_check_mark: Plan applied in Deploy to Development and Management Environment #719

github-actions[bot] commented 2 weeks ago

File Coverage Missing
All files 87% :white_check_mark:
api/serializers.py 88% :white_check_mark: 177-178 183 188
api/test_views.py 95% :white_check_mark: 103
api/uei.py 88% :white_check_mark: 87 118-119 163 167-168
api/views.py 98% :white_check_mark: 198-199 337-338
audit/forms.py 60% :x: 31-38 109-116
audit/intake_to_dissemination.py 88% :white_check_mark: 57-62 264 308-316
audit/mixins.py 96% :white_check_mark: 28
audit/test_commands.py 91% :white_check_mark:
audit/test_intakelib.py 88% :white_check_mark: 154-158
audit/test_manage_submission_access_view.py 98% :white_check_mark: 15 19
audit/test_mixins.py 90% :white_check_mark: 159-160 164-166 254-255 259-261
audit/test_validators.py 95% :white_check_mark: 439 443 611-612 851 858 865 872 1117-1118 1149-1150 1175-1180
audit/test_views.py 98% :white_check_mark: 132
audit/test_workbooks_should_fail.py 88% :white_check_mark: 58 87-88 92
audit/test_workbooks_should_pass.py 87% :white_check_mark: 59 74-76
audit/utils.py 86% :white_check_mark: 9 19 60-62 65
audit/validators.py 93% :white_check_mark: 138 190 279 419-420 435-436 519-520 622-626 631-635 651-660
audit/cross_validation/additional_ueis.py 93% :white_check_mark: 33
audit/cross_validation/check_award_ref_declaration.py 90% :white_check_mark:
audit/cross_validation/check_award_reference_uniqueness.py 93% :white_check_mark:
audit/cross_validation/check_certifying_contacts.py 87% :white_check_mark:
audit/cross_validation/check_findings_count_consistency.py 87% :white_check_mark: 35
audit/cross_validation/check_ref_number_in_cap.py 91% :white_check_mark:
audit/cross_validation/check_ref_number_in_findings_text.py 91% :white_check_mark:
audit/cross_validation/errors.py 78% :x: 30 77
audit/cross_validation/naming.py 93% :white_check_mark: 229
audit/cross_validation/submission_progress_check.py 91% :white_check_mark: 83 126 174 182-183
audit/cross_validation/tribal_data_sharing_consent.py 81% :x: 33 36 40
audit/cross_validation/validate_general_information.py 65% :x: 77 81-84 96 99
audit/fixtures/dissemination.py 71% :x: 38
audit/fixtures/single_audit_checklist.py 55% :x: 160-197 245-254
audit/intakelib/exceptions.py 71% :x: 7-9 12
audit/intakelib/intermediate_representation.py 91% :white_check_mark: 27-28 73 91 129 200-203 212-213 283-284
audit/intakelib/mapping_audit_findings.py 97% :white_check_mark: 55
audit/intakelib/mapping_audit_findings_text.py 97% :white_check_mark: 54
audit/intakelib/mapping_federal_awards.py 93% :white_check_mark: 92
audit/intakelib/mapping_util.py 79% :x: 21 25 29 63 99 104-105 114-120 130 145 150
audit/intakelib/checks/check_all_unique_award_numbers.py 79% :x: 24
audit/intakelib/checks/check_cluster_names.py 75% :x: 20-25
audit/intakelib/checks/check_cluster_total.py 95% :white_check_mark: 99
audit/intakelib/checks/check_finding_reference_pattern.py 74% :x: 34 44-45
audit/intakelib/checks/check_findings_grid_validation.py 89% :white_check_mark: 59
audit/intakelib/checks/check_has_all_the_named_ranges.py 95% :white_check_mark: 66
audit/intakelib/checks/check_is_a_workbook.py 69% :x: 20
audit/intakelib/checks/check_loan_balance_entries.py 83% :x: 28
audit/intakelib/checks/check_look_for_empty_rows.py 91% :white_check_mark: 18
audit/intakelib/checks/check_no_major_program_no_type.py 76% :x: 18 27
audit/intakelib/checks/check_no_repeat_findings.py 88% :white_check_mark: 21
audit/intakelib/checks/check_other_cluster_names.py 81% :x: 23 33
audit/intakelib/checks/check_passthrough_name_when_no_direct.py 83% :x: 11 49 58
audit/intakelib/checks/check_sequential_award_numbers.py 82% :x: 25 35
audit/intakelib/checks/check_start_and_end_rows_of_all_columns_are_same.py 89% :white_check_mark: 14
audit/intakelib/checks/check_state_cluster_names.py 81% :x: 23 33
audit/intakelib/checks/check_version_number.py 73% :x: 30 40-41
audit/intakelib/checks/runners.py 95% :white_check_mark: 187 217
audit/intakelib/common/util.py 90% :white_check_mark: 22 39
audit/intakelib/transforms/xform_rename_additional_notes_sheet.py 81% :x: 14
audit/management/commands/load_fixtures.py 47% :x: 40-46
audit/models/models.py 85% :white_check_mark: 58 60 65 67 209 215 227 239-242 260 437 455-456 464 486 584-585 589 597 606 612
audit/views/audit_info_form_view.py 27% :x: 25-74 77-117 120-137
audit/views/manage_submission.py 86% :white_check_mark: 73-80
audit/views/manage_submission_access.py 98% :white_check_mark: 113-114
audit/views/pre_dissemination_download_view.py 78% :x: 15-16 21-22 29-39
audit/views/submission_progress_view.py 96% :white_check_mark: 182-183
audit/views/tribal_data_consent.py 34% :x: 23-41 44-79
audit/views/unlock_after_certification.py 57% :x: 28-51 73-87
audit/views/upload_report_view.py 26% :x: 32-35 44 91-117 120-170 178-209
audit/views/views.py 53% :x: 74 81-100 123-124 198-199 220-230 257 268-269 280-281 283-287 329-342 345-359 364-377 394-400 405-425 452-456 461-490 533-537 542-562 589-593 598-627 670-674 679-691 694-704 709-721 754-768
census_historical_migration/change_record.py 98% :white_check_mark: 30
census_historical_migration/end_to_end_core.py 26% :x: 57-89 93-111 116-155 161-187 246-258 263 273-307
census_historical_migration/invalid_record.py 94% :white_check_mark: 50 54 58 62 66
census_historical_migration/migration_result.py 75% :x: 17 21 25 29 33-42 46
census_historical_migration/report_type_flag.py 96% :white_check_mark: 19
census_historical_migration/test_federal_awards_xforms.py 99% :white_check_mark: 219-220
census_historical_migration/sac_general_lib/audit_information.py 91% :white_check_mark: 28 82-87 336
census_historical_migration/sac_general_lib/cognizant_oversight.py 68% :x: 11
census_historical_migration/sac_general_lib/general_information.py 86% :white_check_mark: 166-167 177-178 186-187 195-200 233-255 354-355
census_historical_migration/sac_general_lib/sac_creator.py 90% :white_check_mark: 34
census_historical_migration/sac_general_lib/utils.py 84% :x: 35 62-71
census_historical_migration/transforms/xform_remove_hyphen_and_pad_zip.py 92% :white_check_mark: 18
census_historical_migration/transforms/xform_retrieve_uei.py 67% :x: 10
census_historical_migration/transforms/xform_string_to_bool.py 87% :white_check_mark: 17
census_historical_migration/workbooklib/additional_eins.py 84% :x: 58-60 67-77
census_historical_migration/workbooklib/additional_ueis.py 77% :x: 27-29 36-46
census_historical_migration/workbooklib/corrective_action_plan.py 46% :x: 49-51 65 93-125 134-153
census_historical_migration/workbooklib/excel_creation_utils.py 69% :x: 110 119-124 129-136 140-158 171-175 189-192
census_historical_migration/workbooklib/federal_awards.py 77% :x: 181-184 262-301 487 554-562 572-597 621-622 918-1022
census_historical_migration/workbooklib/findings.py 69% :x: 81-86 154-174 179-199 216-218 330-354
census_historical_migration/workbooklib/findings_text.py 46% :x: 50-51 67 97-129 138-160
census_historical_migration/workbooklib/notes_to_sefa.py 66% :x: 34-38 101-102 104-105 107-108 136-143 152-160 168-170 279-326
census_historical_migration/workbooklib/post_upload_utils.py 21% :x: 22-35 66-83 89-111
census_historical_migration/workbooklib/secondary_auditors.py 88% :white_check_mark: 128-130 186-205
census_historical_migration/workbooklib/workbook_builder.py 38% :x: 14-17 26-41
census_historical_migration/workbooklib/workbook_builder_loader.py 41% :x: 18-30
config/error_handlers.py 94% :white_check_mark: 22
config/test_settings.py 92% :white_check_mark: 33-34 49-50
config/urls.py 72% :x: 88
dissemination/file_downloads.py 81% :x: 43-61 91-93
dissemination/forms.py 87% :white_check_mark: 135 144 255
dissemination/search.py 88% :white_check_mark: 113 115 119 127-128
dissemination/summary_reports.py 78% :x: 274 300-302 306-310 421 438 459 511-575 603 638-640 664-672
dissemination/test_search.py 93% :white_check_mark: 51-66 473-474 579-596 608-632 644-669 677-693
dissemination/test_summary_reports.py 98% :white_check_mark:
dissemination/views.py 75% :x: 134 140-142 159-225 268 298 300 336 387 389 391 469-474
dissemination/migrations/0002_general_fac_accepted_date.py 47% :x: 10-12
dissemination/searchlib/search_alns.py 37% :x: 44-58 78-110 115-177 184-187
dissemination/searchlib/search_direct_funding.py 86% :white_check_mark: 21-22
dissemination/searchlib/search_findings.py 76% :x: 18-24 34 36 38
dissemination/searchlib/search_general.py 96% :white_check_mark: 138
dissemination/searchlib/search_passthrough_name.py 35% :x: 21-31
djangooidc/backends.py 78% :x: 32 57-63
djangooidc/exceptions.py 66% :x: 19 21 23 28
djangooidc/oidc.py 16% :x: 32-35 45-51 64-70 92-149 153-199 203-226 230-275 280-281 286
djangooidc/views.py 81% :x: 22 43 109-110 117
djangooidc/tests/common.py 97% :white_check_mark:
report_submission/forms.py 92% :white_check_mark: 35
report_submission/test_views.py 98% :white_check_mark: 796
report_submission/views.py 77% :x: 90 235 259-260 265-266 306-476 479-489 548 584-586 594-595 598-600
report_submission/templatetags/get_attr.py 76% :x: 8 11-14 18
support/admin.py 88% :white_check_mark: 76 79 84 91-97 100-102
support/cog_over.py 91% :white_check_mark: 29-32 92 116-120 156
support/test_admin_api.py 81% :x: 23 147-148 237-238 317-318
support/test_cog_over.py 98% :white_check_mark: 174-175 264
support/management/commands/seed_cog_baseline.py 98% :white_check_mark: 20-21
support/models/cog_over.py 89% :white_check_mark: 103-104
tools/update_program_data.py 89% :white_check_mark: 96
users/admin.py 99% :white_check_mark: 27
users/auth.py 96% :white_check_mark: 58-59
users/models.py 96% :white_check_mark: 18 74-75
users/fixtures/user_fixtures.py 91% :white_check_mark:

Minimum allowed coverage is 85%

Generated by :monkey: cobertura-action against c0a289e9bd57394bf78247d28303f2d4750b1f74