search

cal-itp / data-infra

Cal-ITP data infrastructure
https://docs.calitp.org/data-infra
GNU Affero General Public License v3.0
48 stars 13 forks source link

Blanks in ‘card scheme’ within ‘FCT Payments Rides V2’ #3429

Open erikamov opened 2 months ago

erikamov commented 2 months ago

Email tread with images and files

From: Georgia Jones (Rebel Group):

i. I’m getting blanks in ‘card scheme’ within ‘FCT Payments Rides V2’– see below. Is this test data? There isn’t a specific time for the blanks – which are scattered throughout the data. ii. I can see that an issuer with a distinctly Polish name is showing as ‘United States’ – is this right? Or should I be checking with LittlePay?

Akos and I have done some digging here on the data. It shows that the staging environment for the base ‘customer funding source’ has no blank scheme slots (see below) – but in the ‘FCT Rides v2’ table (attached), there are a fair few blanks across the time period.

Would you be able to double check this please before I reach out to LittlePay?

erikamov commented 1 month ago

I found 3360 records on payments_rides_v2 with Blank Cards. 428 records are not displaying the card because the participant_id does not match. I sent the list to Georgia (Rebel) to take a look.

For the rest of the 2932 records, that contains 153 different funding_source_vault_id, there is no information about those cards on cal-itp-data-infra.external_littlepay.customer_funding_source. The dates of those transactions are from 2023-01-03 to 2024-09-03. (searches made on 09/04/2024)

I will be checking if there is any problem with the update process for this information.

erikamov commented 1 month ago

I sent a list with all the missing funding source ids to Georgia and Akos (Rebel group), they are meeting with Littlepay tomorrow (Wednesday) to talk about the problem. I am waiting for them to get back to us.

Same ticket as #3413

erikamov commented 1 month ago

Littlepay is checking the missing funding source ids.

erikamov commented 2 weeks ago

We got some news on this issue from Akos and Georgia (Rebel group).

Noel from LittlePay has suggested that he can backfill the data as an extra file (see his below email). I need to talk to @charlie-costanzo to know if it is something we can do and the size of the work.

From: Noel Clarkson
Sent: Tuesday, October 8, 2024 7:31 AM
Subject: Re: LittlePay <> Rebel: Cal-ITP database queries

Hi Georgia,

thanks for this, I've had a look and the customer funding sources that are missing (or are missing the principal customer id) are all quite old so I think it would be best for us to just create a "catch up" file of these and post it to your datafeed bucket.

That may take me a little time, but before I do that, is it going to be fine to just have an extra file put in the bucket (we always advise that there might be more than one but don't want to cause problems while solving a problem.

I'd expect it'll be one additional file with the missing or incomplete records (per merchant of course), it will be timestamped the date it is produced and should just be read in in order as is normal.

Let me know and I'll get that sorted out at this end.

Regards,

Noel.