Solaris support

[sm-tradeboox]

Hello.

Any progress on Solaris support? I'm a paying customer of ZeroTier, haven't put it to much serious use yet, but are working on it. We are also running multiple VMs in SmartOS. Zerotier would be a great alternative to having to build one's own network topology across datacenters and for provisioning user access.

Thanks.

Sam

[adamierymenko]

I looked into this a bit. Solaris seems to lack a native tun/tap driver, which along with OpenBSD makes it challenging. Are you aware of any other mechanisms that could permit that kind of thing? If it has dummy net devices plus libpcap, those together can be used to fake a tap though at somewhat lower performance than a real one. If it has a tun, we have some ideas about how to plug a tun into a tap but it'll need some work and testing... But when I looked I couldn't find either type of virtual port driver. I did find some third party code but it seemed really old.

[sm-tradeboox]

Hello Adam,

Thank you for your prompt reply.

I am unfortunately only an end user, not too much of a technical person. Doing a search brings up some possible candidates for a TAP/TUN driver for Solaris, but I can’t comment on the quality of these. 

Solaris has a very sophisticated network virtualisation layer, perhaps Zerotier can use the facilities crossbow provides directly in Solaris? Setting up a switch, vlan, etc. is relatively straightforward. 

I also see there is a package in the repository for Freenet6. Would your requirements be along the same lines?

I just don’t know enough of this topic to contribute more.

Thanks.

Samir 

From: Adam Ierymenko notifications@github.com Reply: zerotier/ZeroTierOne reply@reply.github.com> Date: 30 May 2015 at 10:10:11 To: zerotier/ZeroTierOne zerotierone@noreply.github.com> Cc: sm-tradeboox sm@tradeboox.net> Subject:  Re: [ZeroTierOne] Solaris support (#169)

I looked into this a bit. Solaris seems to lack a native tun/tap driver, which along with OpenBSD makes it challenging. Are you aware of any other mechanisms that could permit that kind of thing? If it has dummy net devices plus libpcap, those together can be used to fake a tap though at somewhat lower performance than a real one. If it has a tun, we have some ideas about how to plug a tun into a tap but it'll need some work and testing... But when I looked I couldn't find either type of virtual port driver. I did find some third party code but it seemed really old.

— Reply to this email directly or view it on GitHub.

[sm-tradeboox]

Or see http://www.whiteboard.ne.jp/~admin2/tuntap/, which is what OpenConnect recommends for the TAP driver.

[adamierymenko]

That one looks a bit newer. Perhaps I'll give it a try. If it works, maybe ZeroTier could bundle a driver for Solaris the way it does for OS X and Windows.

I did a little bit of searching around SmartOS and found that it supports vnics (virtual nics). These are not taps in that they don't have built-in support for attaching to the 'other end,' but this can be accomplished via libpcap's packet injection/sniffing support. This is also possible on OpenBSD as far as I can tell, so in the short term both these platforms could be supported this way. But like I said this is a slightly slower solution than a native driver. You wouldn't notice it under ordinary loads, but if you were pushing large amounts of data it might become significant.

In the longer term, I see that Crossbow speaks VXLAN. We've discussed the possibility of enabling VXLAN interoperability somehow -- via a separate gateway service perhaps. That would be one way. Integrating with Crossbow would probably be significantly more time consuming.

[keesbos]

It looks to me that http://www.whiteboard.ne.jp/~admin2/tuntap/ might even work on Solaris8 ...

(which I now and then do encounter in legacy environments)

[adamierymenko]

Is there a place I can get Solaris x86 that I can install in a VM host like Parallels? Wanted to ask here to see what versions are recommended for widest possible support.

[sm-tradeboox]

Hello.

1st choice - SmartOS. Boots off a USB/ISO, no installation required. Best in a VM. Set up a SmartOS VM (or Zone) in the VM (actually much easier then you may think) to do the testing.

2nd - OmniOS. It’s very minimalistic though.

Solaris 11.2 is also available from Oracle for testing, but I don’t think the guys who are buying Solaris licenses from Oracle are going to be the target market for Zerotier. 

—  Reply: zerotier/ZeroTierOne reply@reply.github.com> Date: 27 June 2015 at 01:20:47 To: zerotier/ZeroTierOne zerotierone@noreply.github.com> Cc: sm-tradeboox sm@tradeboox.net> Subject:  Re: [ZeroTierOne] Solaris support (#169)

Is there a place I can get Solaris x86 that I can install in a VM host like Parallels? Wanted to ask here to see what versions are recommended for widest possible support.

— Reply to this email directly or view it on GitHub.

[adamierymenko]

Yeah I tried SmartOS but it blew chunks in Parallels Desktop for Mac. Might try again with some different hardware settings. To get FreeBSD and OpenBSD to run in there I had to use 'legacy mode' IDE disks.

[sm-tradeboox]

I’ve run SmartOS on VirtualPC on MAC without any issues. Make sure the HW VT-whatever flags are enabled.

I’m a Mac user myself, Parallels is great for running Windows as a VM, I’m not so happy with other OSs. Prefer VirtualPC.

— 

From: Adam Ierymenko notifications@github.com Reply: zerotier/ZeroTierOne reply@reply.github.com> Date: 27 June 2015 at 01:53:12 To: zerotier/ZeroTierOne zerotierone@noreply.github.com> Cc: sm-tradeboox sm@tradeboox.net> Subject:  Re: [ZeroTierOne] Solaris support (#169)

Yeah I tried SmartOS but it blew chunks in Parallels Desktop for Mac. Might try again with some different hardware settings. To get FreeBSD and OpenBSD to run in there I had to use 'legacy mode' IDE disks.

— Reply to this email directly or view it on GitHub.

[elcritch]

We're running on the SmartOS based Triton. It's overall a great system but the lack of ZeroTier support for SmartOS/Solaris branches is limiting the ability to roll out ZeroTier on our internal servers.

I started trying to port using the Linux / FreeBSD make file paths. However, there's a bit of code in the main driver files where I'm stymied by my lack of knowledge on SmartOS vs Linux vs Network programming. :/ I'll try and post some of the details I ran into.

Also, here's an issue with the tun's and zone support on SmartOS (e.g. figure it might need to use a scheme similar to zt on FreeBSD): https://github.com/joyent/smartos-live/issues/626

[elcritch]

Ok, so modifying the Make file and setting CFLAGS to emulate a combination of Linux/BSD: OSTYPE=FreeBSD CFLAGS="-std=c++11 -D__UNIX_LIKE__ -D__SMARTOS__ " make

I also tried a variant of: OSTYPE=FreeBSD CFLAGS="-D__UNIX_LIKE__ -D_XOPEN_SOURCE=1 -D_XOPEN_SOURCE_EXTENDED=1 -D__EXTENSIONS__=1 " make

Since SmartOS doesn't seem to have the same set of socket primitives as Linux/FreeBSD.

There's an error on the physical unix writable layer (osdep/Phy.hpp:979): _handler->phyOnUnixWritable((PhySocket *)&(*s),&(s->uptr),false);

Finally, if anyone is up for trying to get a port going, I'd be willing to setup a Joyent SmartOS container instance for build / test purposes.

P.S. I haven't made much progress, but I pushed the changes I did to elcritch/dev-smartos

[gdamore]

A far, far, better solution here would be to implement an in-kernel GLDv3 driver. I have lots of experience doing this (quite likely I've written or modified more GLDv3 drivers than anyone else on the planet), and would be quite happy to help with such an effort in the not-distant future.

[gdamore]

Basically, I imagine that we'd open in-kernel network ports for the wire packet functions (the physical network stuff), and then expose an L2 network API using GLDv3 primitives.

We would need to have support for the pluggable synchronization primitives though, and probably some effort needed to gut specific locking stuff. Honestly, if we just want on-host support, we could even store the state from in-kernel APIs, but it might be better to link to a daemon that stores the state.

This device should also be able to support zones properly, as it would be a network provider assignable to zones ala Crossbow.

[lennartkoopmann]

I'd also love to see this work. :)

[adamierymenko]

Triage. Does anyone use Solaris anymore?

[sm-tradeboox]

We are using Joyent/SmartOS with Docker containers in production. So we aren't using Solaris except indirectly, as SmartOS is based on Illuminos. Much better than Linux for virtualization.

On 29 Nov 2017 21:05, "Adam Ierymenko" notifications@github.com wrote:

Triage. Does anyone use Solaris anymore?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/zerotier/ZeroTierOne/issues/169#issuecomment-347897866, or mute the thread https://github.com/notifications/unsubscribe-auth/AHX8caDB4AqX78eDvm7HYJ-FegvMz9q7ks5s7XnAgaJpZM4EwB9G .

[terinjokes]

Likewise, SmartOS/illumos user here. Would love to see ZTO integrated into the virtualized network.

On Nov 29, 2017 7:43 AM, "sm-tradeboox" notifications@github.com wrote:

We are using Joyent/SmartOS with Docker containers in production. So we aren't using Solaris except indirectly, as SmartOS is based on Illuminos. Much better than Linux for virtualization.

On 29 Nov 2017 21:05, "Adam Ierymenko" notifications@github.com wrote:

Triage. Does anyone use Solaris anymore?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/zerotier/ZeroTierOne/issues/169#issuecomment-347897866

, or mute the thread https://github.com/notifications/unsubscribe-auth/AHX8caDB4AqX78eDvm7HYJ- FegvMz9q7ks5s7XnAgaJpZM4EwB9G .

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/zerotier/ZeroTierOne/issues/169#issuecomment-347900524, or mute the thread https://github.com/notifications/unsubscribe-auth/AAQsZTS7MTeKWK-aIMdZtBC9fGA5kOZHks5s7XuYgaJpZM4EwB9G .

[sm-tradeboox]

SmartOS now supports Linux containers (LXC) VMs. This supports running a complete Linux distribution in a Solaris zone.

We have also tried running the ZeroTier service in this LXC VM, unfortunately the service does not start. No luck yet. Someone else is working on this, I'll try and provide the logs if you want.

ZeroTier in VMs would be a huge help with our security requirements.

On 29 Nov 2017 21:15, "Terin Stock" notifications@github.com wrote:

Likewise, SmartOS/illumos user here. Would love to see ZTO integrated into the virtualized network.

On Nov 29, 2017 7:43 AM, "sm-tradeboox" notifications@github.com wrote:

We are using Joyent/SmartOS with Docker containers in production. So we aren't using Solaris except indirectly, as SmartOS is based on Illuminos. Much better than Linux for virtualization.

On 29 Nov 2017 21:05, "Adam Ierymenko" notifications@github.com wrote:

Triage. Does anyone use Solaris anymore?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/zerotier/ZeroTierOne/issues/169#issuecomment-347897866

, or mute the thread https://github.com/notifications/unsubscribe-auth/AHX8caDB4AqX78eDvm7HYJ- FegvMz9q7ks5s7XnAgaJpZM4EwB9G .

— You are receiving this because you are subscribed to this thread.

Reply to this email directly, view it on GitHub <https://github.com/zerotier/ZeroTierOne/issues/169#issuecomment-347900524 , or mute the thread https://github.com/notifications/unsubscribe-auth/AAQsZTS7MTeKWK- aIMdZtBC9fGA5kOZHks5s7XuYgaJpZM4EwB9G .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/zerotier/ZeroTierOne/issues/169#issuecomment-347901127, or mute the thread https://github.com/notifications/unsubscribe-auth/AHX8cbRK20T3NTRzXOPiTIzYCYUBJwWsks5s7XwPgaJpZM4EwB9G .

[elcritch]

@sm-tradeboox are you talking about the LXC containers (kernels level library calls) or the KVM based VMs? I had success running ZT on SmartOS KVM instances since they're fully virtualized but it means you're not taking advantage of the LXC bare metal containers.

SmartOS/IllumOS linux container environments are odd from a ZT implementation perspective. The underlying TUN/TAP drivers are from SmartOS/IllumOS base but everything else looks like proper Linux.

[adamierymenko]

What happens when you try to start it? Any errors?

[elcritch]

It's been a few months since I tried, but yes there were some specific errors. It was related to errors with how SmartOS named TUN/TAP's. Zerotier had difficulty instantiating the specific tun/tap in the linux lx-zones (LXC). I dug through the source code a bit and it seemed it'd be possible to modify the platform support, but I am not versed enough in either Solaris vs BSD vs Linux system calls or the way that ZT configures the tun/taps.

There's a couple of links with a little more info on the TUN/TAP situation on SmartOS containers (which should be similar to Illumos):

• https://gist.github.com/dsouzae/b54248996bcb747c43c6609cd7856423
• https://github.com/joyent/smartos-live/issues/626

[gdamore]

TUN/TAP is one hacky way to do this.

A much more elegant solution would be write a kernel level GLDv3 provider. Admittedly this is a fair bit of work -- probably the work of a couple of man weeks for someone knowledgeable in the GLDv3, and in the ZeroTierOne "low-level" APIs (the platform portability code).

And yes, lots of people use Solaris & illumos (and SmartOS) still. Many of them don't even use LX branded zones, but instead use "native" zones -- which are actually better in many respects. (LX has "minimal" overhead, but some system call thunking is still needed.)

I would be willing to do this work myself (to write a Solaris/illumos kernel driver) and am 100% confident in my ability to do so (I've worked with the bottom half of ZeroTier one as part of integrating it into nng), but would need commercial sponsorship to justify the effort. (Given adequate funding, I can make the cycles to make this happen.)

If folks are interested, please contact me out of band -- you can also send an email to info@staysail.tech.

[gdamore]

(I'd be willing to have this funded via kickstarter too, btw.)

[jbarfield]

If the driver could be written to run a native zone (such as in a Triton NAT zone) that would be a god send. Therefore you could utilize the nat zone as a full router to all of your fabric based vm/container infrastructure. Garrett what does a few weeks of funding look like in $$$? I would be willing to contribute and help to raise further money to contribute as well. I have a specific business use case for this functionality.

[gdamore]

Sorry, for some reason I didn't see the comment above. (It helps to have an "@" citation.)

Most likely a few man weeks (which translates to probably 4-6 weeks of calendar time) is about $8-10k.

[jbarfield]

@gdamore @sm-tradeboox

I started a fundraiser on fundly.com and have posted it on my facebook wall. I am currently working on linkedin, the Illumos mailing lists, and potentially the Zerotier mailing lists so long as this does not violate any mailing list rules.

I will personally donate $500.00-$1,000.00 to get it going Garrett. I need to check my budget.

https://fundly.com/zerotier-zerotierone-illumos-driver-development-project

Edit:

I just noticed that you requested kickstarter, should I cancel this one and start a kickstarter or would you rather kick it off on the site?

[gdamore]

Looks good to me.. we’ll see what happens…

Sent from Mail for Windows 10

From: jbarfield Sent: Wednesday, June 26, 2019 10:59 AM To: zerotier/ZeroTierOne Cc: gdamore; Mention Subject: Re: [zerotier/ZeroTierOne] Solaris support (#169)

@gdamore @sm-tradeboox I started a fundraiser on fundly.com and have posted it on my facebook wall. I am currently working on linkedin, the Illumos mailing lists, and potentially the Zerotier mailing lists so long as this does not violate any mailing list rules. I will personally donate $500.00-$1,000.00 to get it going Garrett. I need to check my budget. https://fundly.com/zerotier-zerotierone-illumos-driver-development-project — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.