Support SagePay Integration (RESTful API, "Pi")

[judgej]

This one is going to be big, I feel. SagePay have been working on a JS API over the last year, based on the experience of Stripe and other similar gateways, and have finally launched their own JS API.

This is something to get incorporated into this driver, because it could take off fairly quickly. There are still far too many sites using SagePay Direct without adequate PCI compliance, and the jump to this API will make a lot of sense to them.

The API is in beta, and the docs are here:

https://test.sagepay.com/documentation/

[judgej]

Just a cursory glance through the docs, it looks like it works very much like Stripe. The payment form is on your site, and the card details are fields with no name (so they never POST). A JS library reads the form content, POSTs it to SagePay, gets a response, puts it into the form as a hidden data, then the browser posts the complete form (sans CC number, since those fields have no name) back to your site. Your site then goes direct to SagePay with those details to pick up the complete transaction result.

Like SagePay Server, the server initiates the transaction by getting a session token directly from SagePay (not sure if Stripe does that). The sequence diagram shows this sequence very clearly: the server communicates with SagePay twice, and the browser just once (using JS).

[judgej]

Server-to-server calls to this API use HTTP basic authentication. That's not something I'm come across in a payment gateway before, but the HTTP package OmniPay uses would need to support it. I'd be surprised if they didn't switch to Digest Auth before the beta trial is over, as handling the auth sessions is a lot more flexible.

Response codes all come through the HTTP response, and the detailed messages come in through the JSON body. It's all proper REST. Alternative error codes will also be supplied through the body. It supports multiple errors (yippee) so you can potentially receive a whole bunch of validation errors for multiple form fields.

[greydnls]

This is very similar to the way that Braintree's Js library does things. So, in this case, Omnipay wouldn't handle the JS part, but the handling of the card identifier that's returned from SagePay to the JS portion and then posted to the server, correct?

In the sequence diagram you sent we would set in at "Use Card Identifier". Am I reading that correctly?

[judgej]

It would make sense to include the "Request Merchant Session Key" stage too. That is a REST call server-to-server to get a temporary (400 second lifetime) access token.

The "Use Card Identifier" stage involves the user posting all their personal details, plus the card details transformed into a temporary (200 second) token. So yes, all those details would be collected together, passed to OmniPay to send to the SagePay API in order to be authorised. The response may be a "success" or may be a redirect for further off-site 3DSecure authentication from the end user. How the 3DSecure stuff comes back to the merchant site, I have no idea - maybe Braintree offers some clues. There are definitely some rough edges on this spec at this time, but it is only in beta.

[judgej]

This API implements token-based security though HTTP header fields in all calls, plus HTTP Basic auth on most calls. I'm not sure how many other gateways mess around with custom header fields, but it's something to bear in mind.

I'm putting together, and will maintain, an independent package for this API. I'm starting with value objects for all the messages, with a certain amount of validation built in. It is just going to be the business logic, which can then be wrapped with something that handles the communications. The idea is that this separate package can be wrapped as an OmniPay driver (whether it is "official" or not) but can also be used in other applications that aren't able to run OmniPay for various reasons. Hopefully doing this will help me understand how the API works a bit better.

[judgej]

I have realised while testing this API that the sagepay.js script is not necessary to use it. The whole API can be used just like SagePay Direct without grabbing any card tokens at the front end using the JavaScript it provides (with additional PCI burden, so done at your own risk). So, I've asked whet this API should be called officially, suggesting "SagePay REST", which is probably a little too techie for their marketing, but you never know.

[judgej]

Another note is this API will be rolling out a feature where the payment form can be hosted by SagePay. This, I guess, will be a RESTful version of SagePay Server.

[judgej]

SagePay Integration is its name for now. There are good reasons why it should be a completely separate driver to the current SagePay driver - largely because it is going to be able to do everything ALL the other APIs can do, and there is going to be very little shared between them.

Also because it is in beta, there will be a number of major changes coming up. TBH it is less beta than it is a lean approach, releasing a bit at a time. ATM it only supports payments. It supports them in production, but still only payments. It does not support 3DSecure, nor separate authorise/capture, nor taking card tokens for making future payments against any account. So it is a bit experimental. I've also raised a number of issues (I'm in the beta programme) but there is no way to track what is being done about them. I'm tracking those issues myself here to keep up-to-date.

[judgej]

I'm aiming for OmniPay 3.0 to support this gateway. It can be used as an equivalent to SagePay Direct (with the additional PCI burden) as well as Sage Pay Server (with no credit card details hitting your [uncompromised] application).

[benjam-es]

Is there a proposed release date for 3.0? (and consequently the above mentioned addition)

[judgej]

Support for Omnipay 3.0 was released earlier this week for the Server and Direct access methods.

I've been using the REST API in production for a few years, but it has not made it to an Omnipay plugin yet. If there is interested in that, I will wrap that message API into an Omnipay 3.0 driver.

The problem I have with the SagePay REST API is the way the tokens work. They provide JavaScript to tokenise a credit card at the front end. You need to generate an access token to do that, and that access token lasts just three attempts before it must be renewed (which can be done with AJAX and your own custom JavaScript). Then once you have that tokenised card, it can be used up to three times to create a transaction, with any validation errors (e.g. invalid character in an address) "using up" one attempt. After that, the card needs to be tokenised again. Om top of that, the JavaScript SagePay provide cannot be used with any other validation JavaScript that you may want to use (since it takes control of the form suvmission) so the credit card form and the personal details form must be two separate forms. This means lots of states need to be taken into account, with two temporary tokens in two forms, that could be "used up" or expire independently, and you still want to give the end user a great experience by not having to re-enter details over and over (some sites that require CC details to be re-entered every time a validation error in an address is found, for example, are an affront to UX IMO). I know what's wrong, but am not a JavaScript expert, so not in a position to fix it. Suffice to say, an Omnipay driver has two choices:

a) A potentially cumbersome UX, with a fairly simple implementation; or b) A great UX, but so different to most drivers, that it would hardly be an Omnipay driver; or perhaps c) A front-end expert could provide a decent wrapper that handles the states in the front end, bypassing the JavaScript SagePay provides. The new Authorize.Net do this perfectly IMO - CC and address details can be mixed in one form, and you can run your own front-end validation without it interfering with the CC fields validation. This would be the ideal.

Sorry this turned into a bit of a rant, but I'm stalled while deciding what the best way forward would be. In my production site I need to take the user through two steps, two forms, two pages. I really don't like having to do that though, so making an Omnipay driver that implements that would not be recommended.

[judgej]

I've had a standalone library for supporting the Pi interface for ages, and it would be good to pull it in here. Also Sage Pay Form support would be good.

I'm refactoring the whole package now - mainly just extending test coverage, adding more consistency and supporting a few additional features. After that, putting some namespace structure in (so we can drop the Direct/Server/Shared class prefixes and moved them to the namespace. Then that should make it a lot easier to add in other gateway modes alongside these, without the Messages namespace gettign too crowded.

[benjam-es]

@judgej Just checking if there is support for SagePay PI yet, before looking into creating a custom integration?

[judgej]

No, not yet. I created a PSR-7 message implementation for the Pi AP some time ago, which I use in production. Feel free to use that directly as a dependency, or rip whatever out of it.

https://github.com/academe/SagePay-Integration

I'm also trying to get my hands on the source of the OpenAPI description for Pi. I've got all the files that are a couple of years old, and are full of validation errors, but hopefully will get that fixed soon and can look at auto-generating a PSR-18 client that can be very easily wrapped into an Omnipay driver. You can find those files here:

https://github.com/academe/sage-pay-pi-open-api/tree/master/openapi

TBH I really don't like their Java Script implementation. Ideally you would want to put the CC form and the customer details form together, and have validation/retries etc. intelligently handled (so a failed validation in either part of the form does not require you to enter details in the other part of the form again). I couldn't work it out, so the only practical solution IMO is a two-stage checkout: first enter the CC details, then once you have a valid token (valid for a five minutes or so OR three attempted uses) then the customer details can be entered. The customer details can be validated locally, but then still need to be validated by Sage Pay, and the token gives you three attempts at that, before the user has to be sent back to re-enter their CC details. It's a bit of a messy dance that I have not been able to do justice to to turn it into an Omnipay driver.

Anyway - have fun, and I'm here to help if you want any test scripts or insights :-)

Just some notes on my package. It constructs models, then sends them as PSR-7 messages. The PSR-7 response that comes back is converted back into models using calculated guessing (you don't have to tell it what models they are, it just looks at the payload and works it out). Several years on, and I'm generating clients and models for other APIs and they work differently. Instead every request is driven by the OpenAPI operation, and given an operationId, the application knows what models to send, and what models it should receive (which could vary by HTTP code), which is a much better way to handle it. We just need more gateways to release OpenAPI descriptions and creating Omnipay drivers will become a lot easier with a less laborious code to write.

[benjam-es]

@judgej mind if I get in touch via email/slack/other? just wanted to get your view on my implementation for PI before making a pull request

[judgej]

Hi Ben - no problem, happy to help. The weekend would be easier if that is okay with you?

These days I generate API code almost exclusively from OpenAPI descriptions. I tried to extract the Pi OpenAPI details some time ago - https://github.com/academe/sage-pay-pi-open-api/tree/master/openapi - in the hope of doing that for Pi. Kind of stalled, and I am wondering if SagePay have officially published their description now, or are still hiding it behind the documentation?

Anyway, code generation takes a whole bunch of crudwork out of the process, and can generate code to make the requests, validate models and responses etc. leaving just a tiny layer to implement the Omnipay stuff.

That's just my current go-to approach, and I'm not trying to push you down that route in the slightest, just want everything people code to be meaningful :-)

Drop me an email and we can swap credentials.

[benjam-es]

Thanks, where do I get your email from? speak at the weekend

[judgej]

Should be on my profile - jason.judge@consil.co.uk

[jamieburchell]

I tried to extract the Pi OpenAPI details some time ago - https://github.com/academe/sage-pay-pi-open-api/tree/master/openapi - in the hope of doing that for Pi. Kind of stalled, and I am wondering if SagePay have officially published their description now, or are still hiding it behind the documentation?

I'm late to this party and you probably already know this (I didn't even know about this "new" API) - the OpenAPI spec is at the top of this page now. https://developer.elavon.com/products/opayo/v1/api-reference.