Windscribe server seizure self disclosure

[yegors]

Hi folks,

This is in regards to the addition made here: https://github.com/privacytools/privacytools.io/pull/2395 ("Known Privacy Related Incidents & Gag Orders to Date" list)

This is an unfair inclusion in this list, as it's unlike all the others. You've highlighted our entirely voluntary self disclosure of the Ukrainian server seizure incident, where we have no good reason to believe that this was exploited in the short period that it took to rotate keys, as the vector for the attack is highly challenging (explained in our blog further below). 

We could have said nothing at all, like most other providers who had identical incidents (Express, PIA, TorGuard and Nord, latter 2 of were actually compromised and hid this fact until it leaked online). You would never know about the Windscribe incident if we were not honest with our users. Every single other occurrence confirms cooperation with law enforcement, or an arrest of some kind. This was a precautionary and voluntary disclosure on our part, and it's rather unfair to be included. People make mistakes, we took full responsibility, self disclosed the issue (to our determent), and fixed it.  

Additionally, almost every VPN out there is vulnerable to the exact same thing, right now. Including those who already had an incident before. This is empirically testable by anyone.  https://blog.windscribe.com/ukrainian-server-seizure-a-commentary-and-state-of-the-industry-e71e8d205b26/

If Windscribe is included, everything else should be included too. Although I'd argue, those who brushed this under a rug with their legions or PR people are far worse, but dishonesty clearly works, since they're not included in this list of "Known Privacy Related Incidents & Gag Orders to Date". 

Look forward to your thoughts on the matter. 

[dngray]

The reason it was included was because the private keys weren't encrypted on that server at that point in time nor were they kept in memory only, nor were they rotated regularly enough.

The concern was that it mentions in that blog article that they were encrypted in "some regions", this would imply that configuration changes weren't centrally managed through orchestration tools. This leads to mistakes, and differences in configuration where system administrators believe servers are configured in a particular way when they are in fact not.

However that blog article does indicate you've addressed these things and learned from the incident. It is good to see a detailed response addressing it, (which is more than I can say for a lot of companies) who just employ their marketing spin doctors, or never disclose details about a breach that might have occurred.

Although I'd argue, those who brushed this under a rug with their legions or PR people are far worse, but dishonesty clearly works, since they're not included in this list of "Known Privacy Related Incidents & Gag Orders to Date".

We aim to keep a history there of all VPN based breaches/events like this because they are incredibly hard to find in search engines thanks to VPN companies constantly plugging SEO.

Regarding previous events we aim to include all of them. That section really started out when people kept asking "why not NordVPN", and they had made a series of mistakes in a row. That's why it only really dates back to 2019.

With this particular case I'm thinking the best approach might be to replace the arstechnica article, with your blog article. The reason being is it does address the original issue, but also provides update on how it was rectified.

[ghost]

@dngray Does the Team look Forward to include Windscribe in the list of VPN Providers after necessary changes to be made were completed by them ?

[dngray]

@dngray Does the Team look Forward to include Windscribe in the list of VPN Providers after necessary changes to be made were completed by them ?

As with the current providers listed, we'd need to evaluate how Windscribe matches up to our criteria.

Should be noted though the Jurisdiction, we want to address with https://github.com/privacytools/privacytools.io/issues/1437

[yegors]

The reason it was included was because the private keys weren't encrypted on that server at that point in time nor were they kept in memory only, nor were they rotated regularly enough.

One or more of these things is true for almost every single VPN out there. The article I linked provides proof of this (long lived certs and lack of X509 verification at the client). The standards should apply equally to everyone, not singling out one service that came clean with their users, to their own determent. Although despite doing this, and announcing it to every single one of our users, we've saw an INCREASE in signups/conversions after this event. The general thoughts of the users were summarized as such: "It sucks that this happened, but I have more faith in Windscribe simply because they told me this happened, and they didn't have to".

Server seizures are a common thing for any VPN provider with a sizable server footprint. This was not our first one, all previous instances were of encrypted servers and were non-events, that's why you never heard of them. This seizure was different, which is why you do know and why drastic measures were required. I have no way to prove this, but I'd argue most VPN providers had seizures before. As most VPN providers don't encrypt their servers, don't operate in-memory servers, or have shared/long lived keys/certs (provable). Logic dictates that this happens all the time, and customers are kept in the dark. We're not the biggest provider, but even we get up to 18 subpoenas per month: https://windscribe.com/transparency

The concern was that it mentions in that blog article that they were encrypted in "some regions", this would imply that configuration changes weren't centrally managed through orchestration tools. This leads to mistakes, and differences in configuration where system administrators believe servers are configured in a particular way when they are in fact not.

This is a conjecture. This system totally exists, however we're not happy with the encryption system, which is why it wasn't used in all locations as it's an optional addon to the orchestration, that requires human action. This has been rectified by not storing secrets on the server which is in place now (as per the article), and is being further improved with the in-memory provisioning system we're building, that will be open sourced when it's ready. This is in stark contrast to everyone else who claims to have the same system in place, by virtue of audits that are more or less meaningless and serve as yet another PR move "Look! KPMG said we're all good." Yet you can empirically test that the server certificates are effectively interchangeable, and a compromise of a single one allows the attacker to impersonate any other server, despite the claims of otherwise. Can you prove that the in-memory system was implemented correctly or trust a narrow audit that has a very clear business goal attached to it? I'd argue no, since the audited systems have these flaws that you can verify yourself. Open source is the only option, which is why all our software is about to be open sourced, starting with the Windows/Mac/Linux applications in the coming weeks (to be followed by the provisioning system).

We aim to keep a history there of all VPN based breaches/events like this because they are incredibly hard to find in search engines thanks to VPN companies constantly plugging SEO.

Yes, this is a challenge, but I'd argue critical thinking is required here, especially in circles full of tech savvy individuals. If you don't verify that the presented server certificate is for a server you think you're connecting to, then logically, a compromise of a single server yields the same situation that occurred with Windscribe. We could have easily brushed this under the rug and claimed we're rotating the keys as a "preventive security update" (like PIA has done) or claim "No logs were on the server as confirmed by courts" (as Express has done), but in both situations the reality was the same.

As for the whole "5 eyes" thing, my opinion (which is based on legal research we've done before opening the company in Canada) suggests that this is a meaningless thing, as alluded to in the discussion thread linked above. For technically uneducated bloggers/marketers that usually promote VPNs, it's a simple to understand category. 5 eyes = bad. Offshore = good. Why? Highest paying affiliate programs are operated by companies based offshore, so it's a nice "fact" to throw around to make them seem superior. This is then parroted by everyone else. In reality, the offshore "non-5 eyes" based companies are not doing this for the benefit of the users, but rather the benefit of the owners, with a nice bonus of not needing to pay income tax. Why else would they be based in tax havens like Panama, BVI, Switzerland?

I'd argue the VPN industry is one of the most toxic/snake-oil selling industries out there, with zero oversight, regulation, and critical thinking on the part of the "review" industry. Everything is taken verbatim from press releases, and nothing is truly analyzed from the technical perspective. Things are slowly changing, as everyone is waking up to this mess where a malware vendor headed by an ex-con owns 50% of the VPN market, with Nord owning the majority of the remainder with every Youtuber on the planet hawking snakeoil for a quick buck in the name of "hackers stealing your bank data on public wifi".

[dngray]

One or more of these things is true for almost every single VPN out there. The article I linked provides proof of this (long lived certs and lack of X509 verification at the client).

I was talking about the original reason the arstechnica article was linked (and that it was a third party).

The standards should apply equally to everyone, not singling out one service that came clean with their users, to their own determent. Although despite doing this, and announcing it to every single one of our users, we've saw an INCREASE in signups/conversions after this event.

This is good, we very much like to encourage transparency.

The general thoughts of the users were summarized as such: "It sucks that this happened, but I have more faith in Windscribe simply because they told me this happened, and they didn't have to".

Exactly.

Server seizures are a common thing for any VPN provider with a sizable server footprint. This was not our first one, all previous instances were of encrypted servers and were non-events, that's why you never heard of them. This seizure was different, which is why you do know and why drastic measures were required. I have no way to prove this, but I'd argue most VPN providers had seizures before.

That wouldn't surprise me.

This is a conjecture. This system totally exists, however we're not happy with the encryption system, which is why it wasn't used in all locations as it's an optional addon to the orchestration, that requires human action.

I assume because such private keys are in some kind of vault?

This has been rectified by not storing secrets on the server which is in place now (as per the article), and is being further improved with the in-memory provisioning system we're building, that will be open sourced when it's ready.

This is really good to hear. I think this will be very useful to a variety of usecases such as people running their own personal VPN server. (That makes sense where the threat model is the local ISP and commercial providers are blocked.) Is there any likelihood client apps may also be open sourced for external auditing?

IVPN (one of your competitors) has undergone comprehensive network audits, in addition to having their apps audited by Cure53. I'm curious to know if this is something Windscribe would consider?

As for the whole "5 eyes" thing, my opinion (which is based on legal research we've done before opening the company in Canada) suggests that this is a meaningless thing, as alluded to in the discussion thread linked above. For technically uneducated bloggers/marketers that usually promote VPNs, it's a simple to understand category. 5 eyes = bad. Offshore = good. Why? Highest paying affiliate programs are operated by companies based offshore, so it's a nice "fact" to throw around to make them seem superior. This is then parroted by everyone else. In reality, the offshore "non-5 eyes" based companies are not doing this for the benefit of the users, but rather the benefit of the owners, with a nice bonus of not needing to pay income tax. Why else would they be based in tax havens like Panama, BVI, Switzerland?

This section of the of the site was quite old. We're thinking of replacing it with a section that gets a person to think about their own personal threat model. For example there are certainly cases where using a server in a five-eyes country would actually be a good thing over using one locally, if your locale is a country where the can do whatever they want anyway.

I'd argue the VPN industry is one of the most toxic/snake-oil selling industries out there, with zero oversight, regulation, and critical thinking on the part of the "review" industry. Everything is taken verbatim from press releases, and nothing is truly analyzed from the technical perspective. Things are slowly changing, as everyone is waking up to this mess where a malware vendor headed by an ex-con owns 50% of the VPN market, with Nord owning the majority of the remainder with every Youtuber on the planet hawking snakeoil for a quick buck in the name of "hackers stealing your bank data on public wifi".

Indeed. I have to admit it's refreshing to see someone from a VPN company, write in such a way that is honest.

[ghost]

Open source is the only option, which is why all our software is about to be open sourced, starting with the Windows/Mac/Linux applications in the coming weeks (to be followed by the provisioning system).

@dngray, They seemed to be looking Forward to Open Source the Apps too. Which is actually great including the Provisiong System is also to be Open-Source. So, It is great

@yegors You are making a Good Job Constantly at Windscribe

[yegors]

Is there any likelihood client apps may also be open sourced for external auditing? IVPN (one of your competitors) has undergone comprehensive network audits, in addition to having their apps audited by Cure53. I'm curious to know if this is something Windscribe would consider?

Our Windows/Mac apps have undergone an audit a few weeks ago, we're addressing the few raised concerns now ahead of the open source release (v2.3).

IVPN is my 2nd favorite VPN, I've personally used it for years before Windscribe existed. Those folks run a tight ship, and do things right. I mentioned this fact in several blog articles I've written.

As for the network/infra audits, this is on our roadmap as soon as the provisioning system is ready for deployment. This will be followed by a SOC2 certification and codebase audits of the mobile apps ahead of their open source release. We've recently did an iOXT blackbox audit, and passed with flying colors: https://compliance.ioxtalliance.org/product/326

If you enjoy a breath of fresh air as far as VPN company communications go, you should check out our blog. I write a lot of "edgy" stuff there:

https://blog.windscribe.com/were-not-paying-for-1-25b4e55ca10f/ https://blog.windscribe.com/consolidation-of-the-vpn-industry-spells-trouble-for-the-consumer-57e638634cf0/

[ghost]

@yegors Do you plan to make a No-Logs Audit ?

Though, You are a very Trusted and Transparent company in my Opinion. Though Still some folks consider the Fives Eyes Country as a criteria for Qualification. Despite, the fact that it is unnecessary to consider and being used to determine a Service rather than Deep Investigation of service.

Still this makes a major leap along with other Audits including Apps (This is crucial as Companies like Zerodium is looking into exploit many other Big Players, making an audit for Apps will be good a lot and makes Windscribe safer from those companies. I saw your Quote Tweet. Great work on intimating this to other People) and Server Infrastructure.

[dngray]

@dngray, They seemed to be looking Forward to Open Source the Apps too. Which is actually great including the Provisiong System is also to be Open-Source. So, It is great

I think at this point we'd actually add the to our VPN page, as a recommended provider.

[yegors]

@Loki-L1130 Yes, this is on our roadmap after the new infra is live.

My issue with the whole "5 eyes" thing and some people considering this part of the criteria is simply the fact that the inception has already been performed, by VPN marketing ("review) sites. People think this is a requirement simply because every blogger who wants to make a quick buck promoting VPNs will say this, to promote offshore based VPNs of the well known snakeoil salesmen you see sponsoring your favorite Youtube channels.

Every reviewer that I spoke to that was somewhat reasonable always said something like: "Yeah, it probably doesn't matter, but everyone says this, so we will too".

I personally think this is no different than doctors recommending smoking as a good thing, as they did in 1950s.

[dngray]

My issue with the whole "5 eyes" thing and some people considering this part of the criteria is simply the fact that the inception has already been performed, by VPN marketing ("review) sites. People think this is a requirement simply because every blogger who wants to make a quick buck promoting VPNs will say this, to promote offshore based VPNs of the well known snakeoil salesmen you see sponsoring your favorite Youtube channels.

We're looking at removing that from the criteria as a part of the PR addressing https://github.com/privacytools/privacytools.io/issues/1437

The reason is I'm certain there dozens of other intelligence gathering agreements that probably have no name, but are just as dangerous if you run a server in the countries that they exist.

Every reviewer that I spoke to that was somewhat reasonable always said something like: "Yeah, it probably doesn't matter, but everyone says this, so we will too".

It's something very from 2010-ish era when Snowden's made his releases. Prior to Snowen a lot of what he said was suspected especially with regard to Room 641A and the whistleblower from Mark Klein, however as there weren't documents marked as "secret" it never really got traction with the media.

I think in a way it was because people were concerned about current events at the time and that marketing companies wanted to capitalize off those events.