When this project goes live, we will need to ensure all PII is scrubbed from any run logs and messages (which we will purge after processing).
When HTTP post and put requests fail, we're noticing that the run is logging the sensitive data sent in the request body as part of the failed run log.
The specific request, in as few words as possible
In recent run failures, we're seeing data being printed in the run logs - see L216 in this example run.
As a default, can we scrub data from these error logs like we do for url? See L214: url: '***',
I'm not sure if this is a job or adaptor change?
state.json
To replicate, you can send this payload to the DTP API endpoint - see LP DTP - ReceiveIncomingReferral (PROD)
{"primero_user":"progresv4_primero_intake@cpims-gambella.primero.org","position":"Case Worker","email":"caseworker@primero.org","phone":"0000000000","full_name":"Primero CP","request_type":"ReceiveIncomingReferral","service_implementing_agency":"ProGresTesting","service_response_day_time":"2022-04-27T00:34:47.000Z","service_type":"Alternative Care","service_type_other":null,"service_referral_notes":"Test Reason for Referral May 4","owned_by_agency_id":"UNICEF","unhcr_individual_no":"989-00000007","unhcr_id_no":"989-21-00003","name_first":"Gertrude","name_last":"Stuart","name_middle":null,"name_nickname":"Gertie","date_of_birth":"2014-10-10","sex":"female","address_current":" ","telephone_current":"+27217890018","protection_concerns":["DS-DF","DS-PM"],"protection_concerns_other":null,"language":"Other","id":"1f96ebaa-19e1-41aa-a66e-cd5748f8e42d#fc6db17c7433"}'
adaptor
http v4.0.0
expression.js
To replicate, send the custom payload via a new test job that only sends a post. We know sending this payload should fail and the request should receive a response with error code.
Background, context, and business value
When this project goes live, we will need to ensure all PII is scrubbed from any run logs and messages (which we will purge after processing).
When HTTP post and put requests fail, we're noticing that the run is logging the sensitive data sent in the request body as part of the failed run log.
The specific request, in as few words as possible
In recent run failures, we're seeing data being printed in the run logs - see L216 in this example run.
As a default, can we scrub data from these error logs like we do for url? See L214:
url: '***',I'm not sure if this is a job or adaptor change?state.json
To replicate, you can send this payload to the DTP API endpoint - see LP
DTP - ReceiveIncomingReferral (PROD)adaptor
http
v4.0.0expression.js
To replicate, send the custom payload via a new test job that only sends a post. We know sending this payload should fail and the request should receive a response with error code.
Your test http post should look like this: https://github.com/OpenFn/primero-progres/blob/60e440994f9334ab979c53fe0e2fc21f6d8a84c1/jobs/PROD/f2-2.sendReferralToDTP.js#L263-L275
output.json
The run log should look the same as this run, except with data scrubbed on L216 - e.g.,
data: '***',