Local validation for address parts

[judgej]

This is just experience from SagePay Server.

Each address field has valid permitted lengths (min and max) and also sets of characters that are accepted. Being able to access those validation rules from the application to ensure the address and other details are valid, would be useful.

For example, postal codes for SagePay have a maximum length of 10 characters and only accept letters (case-insensitive), digits, spaces and a hyphen. It is also a mandatory field. Anything outside of that will result in a failure when submitted. Catching that before it is sent to SagePay helps in error reporting - the user can be told what is wrong and where, while the error messages that come back from SagePay are not so helpful.

So, are there any thoughts on how this would be implemented? We have solved this in our implementations using a model for the addresses, and then putting validation rules into those. Quite often we find that users need to amend their address, just for the transaction, in order to meet the SagePay validation rules. A good example that comes up a lot in one project is Irish clients who put "N/A" into the postcode field, since the Republic of Ireland does not have postal codes yet. SagePay raises a fatal "invalid character" which means nothing to anyone. In this case it is the "/" that is causing that errror. For other payment gateways that "N/A" may be perfectly acceptable.

I can provide some further examples of what we do practically, if that helps understand where I am coming from. I realise validation cannot all be done before sending to the payment gateway, but if valid acceptable data from these gateways varies a lot, then some of the more obvious validation issues can be caught early.

[amacneil]

I think it's too big an issue to solve in this library. I've tried to err on the side of less validation, and leave it up to the remote gateway to check things out. However, the expectation is that you will write your own form validation using your framework's methods, and not catch the Omnipay exceptions (it should never get to that point).

One problem with this is that if for example Sage Pay change their validation rules, this library would get out of date quickly. If you are only using one gateway, it's easy to just code these rules into your app. If you support multiple gateways, it's easier to just take an extra half second and let the remote gateway do the validation.

Perhaps we should see if we can improve the error message returned from sage pay? I'll see if there is any more useful data in the response we could output.

On Saturday, March 9, 2013, Jason Judge wrote:

This is just experience from SagePay Server.

Each address field has valid permitted lengths (min and max) and also sets of characters that are accepted. Being able to access those validation rules from the application to ensure the address and other details are valid, would be useful.

For example, postal codes for SagePay have a maximum length of 10 characters and only accept letters (case-insensitive), digits, spaces and a hyphen. It is also a mandatory field. Anything outside of that will result in a failure when submitted. Catching that before it is sent to SagePay helps in error reporting - the user can be told what is wrong and where, while the error messages that come back from SagePay are not so helpful.

So, are there any thoughts on how this would be implemented? We have solved this in our implementations using a model for the addresses, and then putting validation rules into those. Quite often we find that users need to amend their address, just for the transaction, in order to meet the SagePay validation rules. A good example that comes up a lot in one project is Irish clients who put "N/A" into the postcode field, since the Republic of Ireland does not have postal codes yet. SagePay raises a fatal "invalid character" which means nothing to anyone. In this case it is the "/" that is causing that errror. For other payment gateways that "N/A" may be perfectly acceptable.

I can provide some further examples of what we do practically, if that helps understand where I am coming from. I realise validation cannot all be done before sending to the payment gateway, but if valid acceptable data from these gateways varies a lot, then some of the more obvious validation issues can be caught early.

— Reply to this email directly or view it on GitHubhttps://github.com/adrianmacneil/omnipay/issues/28 .

[judgej]

My thought was less about actually doing the validation in this package, and more about providing validation metadata in a consistent form, that may vary wildly between different gateways (and yet, may not). I'm assuming also the validation rules would be versioned at the gateway end, so if you know the gateway version you are using, then you would be guaranteed to know the validation rules that would apply.

An example of where this has impacted me, is in a project for a membership organisation. FuelPHP is used to provide an interface for users to pay their subs via SagePay. When they wish to pay, we fatch their address and personal details from the CRM and then present those details to the user in a form. These are the details the user then submits in order to make the payment. However, many members are international, and have different formats set up for their addresses, and many of those addresses are invalid for submission to SagePay (part of which involves SagePay only accepting ISO8859-1 characters and not UTF-8 - doh).

So the form the user submits has a model behind it with various validation rules, mainly mandatory flags, max/min lengths and regex patterns for the fields. We are then able to use the standard CRUD-type functionality to handle the address/email/name submission and present any errors inline for the user to amend. Once they have done this, it is guaranteed to pass validation at the SagePay end.

If we let anything through, then we get general errors like "invalid character found", which is of no use to anyone.

Countries also have to be submitted as ISO3166-2 codes, so even that is not free-format (we do a translation from the CRM internal codes to the ISO codes when presenting the form).

[judgej]

So, given that, depending on which card payment system the site visitor selects, we may (and I mean may, since I have never seen a tabulated comparison of the rules that all these gateways enforce) have to switch validation rules in the form.

And when I say "have to", even that may be more important than is immediately obvious. For example, different payment gateways may be available to different countries, and one of the reasons for that is down to mandatory requirements on the address validation rules (e.g. postcodes being mandatory for SagePay, but some countries such as Ireland not actually having implemented postcodes).

That also leaves a chicken-and-end situation - you cannot get the address validation rules without first knowing the country, which is a part of the address. We can at least detect this and inform the user that they must go back and select a different gateway when they try to submit their address to make a payment.

[amacneil]

I think you're giving the gateways too much credit :smile: While Sage Pay may have well defined validation rules, most gateways have no such specific requirements. I don't think it's a good idea using trial and error to determine the gateway validation rules, then codify them in this library (whether they are enforced or not). Many gateways don't even require an address at all. Also, like you said, how the gateway validates the address may depend on which country the customer is in.

That said, I think this library should be responsible for normalizing the address, so in most cases you shouldn't run into any errors. For example, we always expect ISO alpha 2 country codes, and will automatically translate these for the few gateways which use ISO numeric country codes.

I think in your case it's probably worth hiding the postcode field when your customers select Ireland as their country so you don't run into any issues there, and adding any validation rules where the Sage Pay errors are completely useless (like the one you mentioned above). But in most cases it's probably best to just pass what you have through to Sage Pay and display their response back to the customer.

[judgej]

Cool. I can't say I have the experience of a wide range of payment gateways, but have noticed how awful the documentation is in many cases. I suppose what I am hoping for, is the ability to create a generic front end for a platform that uses that library, and does not require a lot of custom functionality for each gateway. That really is probably hoping for too much while each gateway provider does their own thing.

I'll also do some more experimenting with SagePay errors to see if there is any additional context that I'm missing.

[judgej]

As an example of awful documentation (drupal site, but document provided by Sage):

http://drupal.org/files/issues/SagePaySERVERProtocolandIntegrationGuidelines.pdf

It states there that the validation errors are in a human-readable form, and only used when debugging your scripts. By "human readable" it means "makes sense to a developer" but not something you would put in front of an end user. It then goes on to explain that once you have debugged your scripts, you will never see these validation errors again, and if you do, they should be recorded for debugging, and the end user simply told, "sorry, tough luck mate, we can't process your order". It is so detached from reality.

What is means is that there really is nothing in the error messages that we can give to the users to help them complete their transaction (and completing the transaction is what this is all about). So we cannot rely on SagePay validation to catch errors for us - the way the system is designed, it is assumed that point when the payment is sent, is too late to give the user a chance to correct anything. So every conceivable bit of validation we can perform, must be done in the application before we try to send a transaction. In my experience, doing that is reliable and we get very few, if any, failed payments.

Anyway, I don't want to flog a dead horse, but just explaining where I am coming from, as I am not sure how it compares to the other gateways. They may all return nice error messages with attached context, so the user can make corrections and resubmit. Or they may all be like this. I just don't know, but I hope these examples help in deciding what to do about validation.

What to do may even involve spinning off separate packages to handle that kind of thing, perhaps more geared towards the requirements of a GUI front end.