Implement VAT validation for GB VAT numbers

[firstpromoter]

Since they are not working wtih VIES, UK gov has their own API https://developer.service.hmrc.gov.uk/api-documentation/docs/api/service/vat-registered-companies-api/1.0

I think everyone will appreciate if this check will be implemented.

Thanks

[yolk]

I am really hesitant adding this. There are similar APIs for Switzerland, Norway or the US. But our focus with valvat has always been validation of european VAT numbers for trade within the EU. As an EU based business there is no need to validate VAT-IDs from the UK.

I think I'll keep this issue open for further input by users.

[firstpromoter]

I understand the situation, the biggest issues is that there are tons of existing businesses that already trade with EU and they need their VAT id checked. If this was just for new businesses it would be easier, but vat check is still needed for existing companies before invoicing for example.

In our situation we're doing invoicing for EU companies and for each new invoice we need to verify the VAT id to make sure it didn't get invalid. Right now everything stopped for anything doing business with UK companies since we can't validate VAT.

[pastullo]

Hey @yolk , happy Valvat user here...until today 🤣

We are a European business based in DE and we sell to any EU country including the UK which is a considerable share of our business. Now that the UK is technically outside the EU, we are still selling products to them and need to validate their VAT.

The fact that we are arbitrarily drawing a line between the EU and the UK doesn't mean that there is a ton of business between those two countries happening every single day. If Valvat decides that the UK should not be included in the validation list, we will have to essentially copy paste this project and add UK support...which is less than ideal. 😅

Again i want to stress that your sentence "As an EU based business there is no need to validate VAT-IDs from the UK." is absolutely arbritary and makes Valvat worthless for several users. We highly appreciate this project and it's been super useful. Would be great to hear from you your thoughts and plans. Thank you!

PS: @firstpromoter same exact situation here!

[firstpromoter]

@pastullo exactly. I'm pretty sure that many users of this awesome gem already do businesses with UK and won't stop just because of brexit. Since business continues as usual with UK companies, this gem also should support GB validation as usual, to keep the pre-brexit business flow working.

[firstpromoter]

Any news on this? We want to know if there's any possibility to have this resolved or look ourselves for alternatives. thank you

[yolk]

@pastullo You are absolutely right about my earlier statement being somewhat arbitrary. There are undoubtedly business cases which would benefit from added support for VAT numbers from UK. But there are also other cases.

valvat was first build for our own inhouse needs and is shared and maintained for everyone to use. If you have the requirement to validate UK VAT numbers, you have all the possibilities to take the existing code and extend it. But don't require from us/me to put (free) work into your issue.

@firstpromoter I am very sorry, but I don't have the capacity at the moment to work on this. But please step forward, if you want to give it a try.

I took several looks at the api from HMRC and it seems to be still in beta. Till only one week ago they disallowed unauthenticated users to access the endpoint at all. So be prepared for a somewhat moving target.

[pastullo]

Hey Sebastian, totally understand and nobody expects you guys to do it for us. It's great to know that you would be potentially open to support UK VAT numbers if there was a PR coming in though.

[jarthod]

I understand the frustration of companies (like me) for which this stopped working and now UK VAT numbers are considered invalid, but keep in mind that as @yolk pointed out, it's the same for other countries which are close but not in the EU so it does not make much more sense to implement this one, than to implement every countries in the world.

So the question is more: should valvat support every country people want (which would require starting by allowing checksum validation without lookup, adding more options so people can limit to EU to preserve existing behavior, and gradually adding and maintaining hundreds of API integrations which will often break). I agree this would be handy and I would like that too, but I don't think any of you guys would like to maintain that and handle the support if you only a couple one of them personally.

Alternatively we could make an exception for UK as a "former EU country" and draw the line here instead, or maybe draw the line at "countries which does support reverse charge and thus require VAT number validation". But this needs to be clear so the gem doesn't become impossible to maintain. If @yolk is OK with this approach I am OK to spend some time contributing the lookup code for GB.

Edit: For the record I tested their API a bit (https://api.service.hmrc.gov.uk/organisations/vat/check-vat-number/lookup) and it has been working well on my client numbers (without authentication).

[jarthod]

Ok I just spend some time trying to integrate the HMRC API to validate GB numbers into Valvat, the result is currently in a branch on my fork and I opened a PR with the diff: #101.

@pastullo @firstpromoter if you want a quick solution you can already use my branch (I am already using it in production):

gem 'valvat', github: 'jarthod/valvat', branch: 'hmrc'

Though I don't consider this a good long-term solution of course, so @yolk I though about a couple options to discuss with you:

1. If you're ok with this, you could merge my PR (#101) to integrate this directly into Valvat (after you validate it of course and I can do some changes if you want). This is likely the simplest for users (just need to upgrade version) to get back GB validation but I totally understand if you don't want to do that.
2. I could release my fork under another name (like valvat_plus_hmrc, valvat_plus_uk or something like that) and we should just put a note in your readme that for people who needs GB number validation they can use this one instead (not great UX IMO but much less changes on your side)
3. We could merge some of the refactoring I made in valvat but keep only VIES in there, and then publish a "plugin" gem (like valvat_hmrc) which would plug into valvat to add GB number lookup. So this way people who need both will simply need to add both gems, and in your gem you only keep support for VIES. This is a bit more work to write/maintain and a bit more to do for the users, but it has the advantage of being pretty future-proof as it could support other services and people could add their own without bothering you.

Let me know what you think and if you have other suggestions, I hope we can find a solution which works well for everybody ;)

[crysper]

@jarthod you're a life saver! We just planned to work on this but this will save us so much time, thanks a ton.

[pastullo]

Hey guys, just wanted to let you know that we have been happily using the hmrc branch in production for several months now and it worked just fine! 🥳 Unfortunately a few days ago it stopped working since it's not updated with the v1.1.2 fix. 😢

Personally i think we should merge the PR from @jarthod back into the master project and support UK VAT validation and keep the scope as EU + UK since it was a former EU country and there is ton of business going on through them.

@yolk did you have a chance to think on what you would like to do with your gem? Let us know, i feel that there are a few of us a bit in the limbo with Valvat now.

[crysper]

We also use the hmrc branch in production for quite a while and we fixed the issue manually a few days ago. I can understand that there's work required and that's totally valid reason to not add GB code validation, but besides that it doesn't make any sense in a business and practical way, almost everyone from EU is able to do business with UK companies as before, nothing changes business wise so the gem's purpose shouldn't change as well.

[jarthod]

Oh thanks for the heads-up guys, I did not notice this problem yet. I just updated my branch with this fix (still in https://github.com/yolk/valvat/pull/101)

[yolk]

I started to (re)implement the solution by @jarthod in the nethttp branch. It should behave like the code in the hmrc branch; except it requires to pass in the :uk option set to true.

Maybe some of you could take a look at it? Would like to release it as valvat 1.2 soon.

[jarthod]

Thanks @yolk!

I just tried this branch but couldn't get it to work with the validator, maybe the :uk option is not supported here yet? I tried with:

validates :vat_number, valvat: {uk: true, lookup: true, allow_blank: true, checksum: true}, if: :vat_number_changed?

:bow: Edit: of course I noticed my error with the proper syntax in the readme:

validates :vat_number, valvat: {lookup: {uk: true}, allow_blank: true, checksum: true}, if: :vat_number_changed?

Seems to be working fine now, I'll run a couple more validations and deploy this in the next days.

I also had a quick look at the code diff and didn't saw anything work commenting :+1: The swtich to Net:HTTP instead of savon is great IMO :+1:

[yolk]

@jarthod Thx for looking into the new branch.

In ActiveModel you have to set the :uk option on the lookup key:

validates :vat_number, valvat: {lookup: {uk: true}, allow_blank: true, checksum: true}, if: :vat_number_changed?

Or as direct lookup:

Valvat::Lookup.validate("GB553557881", uk: true)

[yolk]

Ahh, you found it by yourself! Sorry, we commented in parallel ;-)

One question: I implemented it with the :uk option defaulting to false for backwards compatibility to don't break the API for an 1.2 release. But I am now unsure if it wouldn't make sense to release a 2.0 and maybe even default the option to true. Any options on that issue?

[jarthod]

Ahh, you found it by yourself! Sorry, we commented in parallel ;-)

Yep sorry, I commented too quickly and found the answer 2 minutes later ^^

One question: I implemented it with the :uk option defaulting to false for backwards compatibility to don't break the API for an 1.2 release. But I am now unsure if it wouldn't make sense to release a 2.0 and maybe even default the option to true. Any options on that issue?

Well in my opinion the "backward compatibility" was actually "broken" in 2021 when UK was removed from VIES :rofl: so supporting it again using HMRC is not "breaking" again, it's just restoring the previously working behavior. I think most older users from before brexit like me would prefer uk to be enabled by default (like I did in my branch).

But I also understand that users starting to use the gem later may need to validate only EU countries, and as doing business with the UK now requires registering a UK VAT numbers and filling separate HMRC return form, not every company will choose to do it (at first at least) because of the papework, especially new startups in EU.

It sounds more "dangerous" to realize after the fact that you've accidentally been accepting UK payments and collecting UK VAT without any way to declare it legally, than getting complaints from a UK client who can't validate his VAT number.

So in conclusion even though this option is useless for me and would be simpler if true by default, I think for other users it's a good idea to keep the option to false by default, so they only opt-in if they know what they are doing. And in that case 1.2 version bump is enough.

[yolk]

Thanks alot for your suggestion. Makes sense; I guess I'll go with 1.2

[yolk]

I released v1.2 with support for lookup of UK VAT numbers via HMRC.