Clean up the submission/delivery code

[mkalish]

User Story

As a RS developer, I would like the delivery submission endpoints to work effectively and to be well tested

Description/Use Case

The code for these endpoints is brittle and does not support easily iterating on the pipeline. Since the report lineage is agnostic of the steps the code should be able to be refactored to not need to specifically pull data out of the specifically named steps.

Risks/Impacts/Considerations

• The endpoints no longer working as expected; however the understanding is that they are rarely used so the risk is low

Dev Notes

• The recursive query can likely be replaced in favor the new report graph code

The problem code is below. The issue is that the route steps are hardcoded rather than being flexible to support arbitrary steps

descendants.filter { it.actionName == TaskAction.translate }.forEach { descendant ->
                enrichWithTranslateAction(descendant)
            }
            descendants.filter { it.actionName == TaskAction.route }.forEach { descendant ->
                enrichWithRouteAction(descendant)
            }
            descendants.filter { it.actionName == TaskAction.convert }.forEach { descendant ->
                enrichWithConvertAction(descendant)
            }

Acceptance Criteria

• [ ] The delivery/submission endpoints are flexible enough to support any adjustments to the UP steps without having to be amended
• [ ] Database tests are written to cover the functionality

[Andrey-Glazkv]

Hey team! Please add your planning poker estimate with Zenhub @adegolier @arnejduranovic @brick-green @jack-h-wang @jalbinson @mkalish @thetaurean

[Andrey-Glazkv]

Please add your planning poker estimate with Zenhub @JFisk42

[Andrey-Glazkv]

Please add your planning poker estimate with Zenhub @david-navapbc

[david-navapbc]

talked with @mkalish and got some clarity as to the scope of the ticket

• the ask fundamentally is to replace the enrichment methods that are scoped to a particular route step and make all the enrichment happen in a single method call that is agnostic to report steps.
• in the description text for this ticket there is a typo. The word "date" should be "data"
• the things we need resultant from the refactor are the SubmissionHistory objs to contain the following
  • action log warnings and errors - both count and description
  • names of the receivers
  • status as to whether or not the report was or was not sent. Not just a boolean - some indication of what happened

probably going to leave the recursive db call alone for now and do that in a different, specifically scoped ticket. The SubmissionHistory obj is stateful and in theory can be refactored to present to the rest of the system in the same way.

Now that I have a better understanding of what the ask is I'm going to come up with an action plan and check in again with @mkalish to ensure we're in alignment as to the definition of done.

[david-navapbc]

going to try something like this

    private fun actionAgnosticEnrich(descendants: List<DetailedSubmissionHistory>) {
        descendants.forEach {
            destinations += it.destinations
            errors += it.errors
            warnings += it.warnings
            overallStatus = it.overallStatus
        }
    }

seems like the simplest way to hit the ACs

[david-navapbc]

unified method is failing on four tests.

SubmissionHistoryTests test DetailedSubmissionHistory overallStatus (delivered) test DetailedSubmissionHistory overallStatus (partially delivered) test UP enrichWithDescendants stopped at route test UP enrichWithDescendants filterLogs populate for the corresponding receiver

[david-navapbc]

@mkalish @arnejduranovic

I've investigated the cause of the remaining four test failures and in essence what's happening is that, depending on the action reported to have been taken, elements may be added to SubmissionHistory.destinations. The quantity and disposition of those destination elements matters with respect to computing the summary - including the overall status value.

Unfortunately what this means is that creating a common method to handle this isn't going to work without more surgery than a 3pt ticket allows. There are too many tests and other bits of logic that work only if the action-specific logic is in play.

Going to investigate a bit and see if I can't come up with a different solution. This is an FYI that the work of this is more involved than we originally planned for.

[david-navapbc]

talked with @mkalish

• @arnejduranovic will need to weigh in on what challenges there are with respect to changing the API contract should that be necessary
• @mkalish suggested refactoring the DetailSubmissionHistory obj to capture information by walking the entire graph from each leaf to root might yield enough information to not have to change the api contract where the status field is concerned
• transport_status is a column in one of the database tables that might prove useful
• this is definitely under-pointed given the volume of work.

[david-navapbc]

it looks like if we change the DetailedSubmissionHistory class we're changing the payload callers receive upon submission.

https://github.com/CDCgov/prime-reportstream/blob/f473828604a8602d5df7ab91f0692292ab85d014/prime-router/src/main/kotlin/azure/ReportFunction.kt#L216

[david-navapbc]

This is a screenshot of what comes back when a caller submits a report. Note the "Location" header value providing the URL of of the submission history for this submission.

What's fun about that is it's wrong. The actual URL for that record is http://localhost:7071/api/waters/report/1221/history

This is why you get a 404 from the app when you try to use the provided history endpoint. I've been operating under the assumption this whole time there was a bug in either the serialization or the fetch logic when in fact the path itself is wrong.

I will fix that in this ticket. Additionally I'm going to look at why the RECEIVE action is not being recorded to the action_log table.

Lastly - I think for lots of reasons it makes sense to change the status enum.

here's what's currently there

• VALID - successfully validated, but not sent
• ERROR - error on initial submission
• RECEIVED - passed the received step in the pipeline and awaits processing/routing
• NOT_DELIVERING - processed but has no intended receivers
• WAITING_TO_DELIVER - processed but yet to be sent to/downloaded by any receivers
• PARTIALLY_DELIVERED - processed, successfully sent to/downloaded by at least one receiver
• DELIVERED - processed, successfully sent to/downloaded by all receivers

I want to change it to

• RECEIVED
• PROCESSING
• PROCESSING_COMPLETE_OK
• PROCESSING_COMPLETE_ERROR

The reasons are many. Making this change makes the it a lot easier to meet the ACs of this ticket. It also makes this possible to make this area of code less wonky. Last but not least ask yourself why in general the sender cares about whether or a message has intended receivers or whether or not the message has been downloaded by intended receivers? From their perspective why do they care about anything other than the submission being valid and whether or not its been processed? Senders aren't using us to send reports to particular receivers, they are using us to have us worry about who is supposed to get what.

Meaning why are we telling them about delivery details when the app is here to abstract that away from them? Our value is we worry about the sausage making details of delivery. They just need to know if their submission was processed and whether or not the result of that processing produced some kind of error.

[david-navapbc]

@mkalish I added the updated url to ReportFunction.

[david-navapbc]

@mkalish cc: @arnejduranovic

It appears as though you've created a new git branch to work this ticket and in the course of doing so have undone the bugfix described in the comment above. Friendly reminder to put the fix back before putting the PR up for review.

ty!