Complete SacRT Production Validation

thekaveman commented 1 year ago

Related to #1193.

[x] Blocked by #1239.
[x] Blocked by SacRT need to update their merchant portal to create a new Fare Capping product
[x] Blocked by LP ticket about apparent capping issue

Acceptance Criteria

[ ] @indexing to confirm that this ticket is done

What is the definition of done?

Product validation is completed (2-3 times) with:

[ ] Payments successfully processed (ideally multiple payment types)
[ ] Product team is able to view/report on payments info visible in Metabase
[ ] Payments comply based on the intended configuration / Ex - capping rule was correctly applied to payments.

Resources

1194

MarikoLochridge commented 1 year ago

@jarrettkrumrei @johnatstate completed the SacRT production validation testing

Feedback from @johnatstate Discount Product was applied correctly but the Time Cap was not.

Waiting on feedback from @jarrettkrumrei

Resources: Cal-ITP Benefits: SacRT production validation testing [Living doc] SacRT Gantt chart - Cal-ITP Benefits

Slack Guidance on SacRT Production Validation Testing Updates to SacRT Gantt chart - Cal-ITP Benefits

MarikoLochridge commented 1 year ago

@o-ram will demo SacRT Production Validation testing + feedback at 2/23 Weekly Demo Meeting - moved the meeting back 30 minutes to accomadate her schedule

MarikoLochridge commented 1 year ago

Feedback Slack thread from @johnatstate regarding SacRT production validation

Screenshots of Littlepay customer portal in this Google Doc

Major Points:

Confusion on transaction amount. First charged $1 and later $1.25. Mastercard does a $1 preauthorization hold sometimes but it’s temporary and the amount updates/officially goes through after all transactions are aggregated at the end of the day and sent off for settlement
Validators did not work. Validators would scan the card but not communicate with LittlePay.
No proof of payment. According to a fare ambassador she has seen people scan their cards or phones to receive a green check, then she would get no evidence when she scanned their card afterward. This could trouble with the fare inspection app whether it’s the app itself or the training of the ambassadors.
Senior discount is being applied successfully, but not the cap.
No charges for tap made with Apple card via Apple pay

MarikoLochridge commented 1 year ago

Here is a link to the demo that that @o-ram did about production validation.

You can see in the demo how payments are reflected inside of Metabase. She also explains the fare capping issues that came up during the production validation process.

MarikoLochridge commented 1 year ago

Following the February Retro our goal moving forward is to continue improving this process.

List of options + timelines for PV
Clearly identify ownership of AT and PV and handling the transfer between AT to PV (likely @MarikoLochridge with support of @o-ram Olivia + agency staffing)

@MarikoLochridge to follow-up with @o-ram on to document this in the production validation document that has already been started.

indexing commented 1 year ago

@MarikoLochridge Now that SacRT is moving again, let's add this to the agenda of items to discuss. How do we unblock and begin resolving this issue?

MarikoLochridge commented 1 year ago

@indexing Testing is happening now, please refer to this convo on Slack as well as this ticket SacRT Go Live Checklist #1194

MarikoLochridge commented 1 year ago

@johnatstate @jarrettkrumrei completed production validation on Friday, June 2nd. Here is a link to the Slack thread with documentation of the experience but major points include:

@o-ram Summary based on John/Jarret feedback + Metabase data:

Good: I think the cap worked in most cases, i.e. the card was only charged once for multiple taps made within 90 min Bad: The cap appears to have not kicked in for John’s Visa card for some unknown reason Confusing: I can’t see all taps made within a 90 min period - only one. And while the customer portal shows “capped”, Metabase does not give any indication that multiple taps were made/anything was capped- the tap that shows up just has the regular “senior discount single ride” label. This is odd because usually the other taps still show up as a $0 charge and are labeled as “capping”.

Also, noted that John experienced issues with terminals being out of order or screens being out of order, according to a conversation with a local fare ambassador "She commented that she has seen people tap twice within five minutes and be double-charged"

Next Steps: @o-ram to speak with LittlePay

Why was John’s visa card not capped?
Why is it not evident that multiple taps were made besides the “capping” label within the customer portal?

o-ram commented 1 year ago

I just sent an email off to Littlepay. I expect to see them early tomorrow morning on a SacRT call, and their support team is in UK time, so they might see my email before the call. Therefore, I wanted to get it off ASAP. Will keep everyone informed and/or loop others in to the email thread based on Littlepay's response.

thekaveman commented 1 year ago

Notes doc with more info from Metabase about the questions for Littlepay: https://docs.google.com/document/d/1LfcQxH-RJTHxolORCpr_F-dQGaL-thVYwTEFlcbMxGI/edit#

thekaveman commented 1 year ago

@johnatstate @jarrettkrumrei are performing another round of testing this week, after SacRT made the necessary product updates last week.

indexing commented 1 year ago

A Compiler+Rebel team completed another round of on-the-ground validation testing on Tuesday, Aug 22, 2023 and documented results here:

- SacRT Test Log_Aug 2023

Rebel team is in touch with SC Soft to get support on what appears to be connectivity issues on validators that prevent transactions from reaching LIttlepay.

indexing commented 10 months ago

Throughout testing at SacRT the Benefits app has performed as expected despite issues with the contactless payment service as a whole. Given this success, we've been asking SacRT if we can "soft launch" Benefits to production over the past couple months.

After numerous attempts from various angles to get approval from SacRT to move their Benefits instance from testing to production, we've encountered silence. It seems SacRT has paused all efforts related to the contactless payment demonstration project for rail and is focusing instead on the more comprehensive solution that includes buses.

Thus, we are leaving this issue as paused until we hear from SacRT and removing it from sprints. It may be that we simply close this issue as unfinished.

cc// @o-ram @thekaveman

indexing commented 3 months ago

In a meeting with the payments team last week we learned that SacRT is removing all hardware validators on light rail that they implemented during the pilot. Thus, we will soon no longer be able to test the pilot implementation.

Going forward, any production validation will need to be done using hardware implemented as part of the new contracts.

Given that the pilot is effectively over, I am closing this issue.

cal-itp / benefits