Support Windows platform in vectorscan

[apismensky]

From https://networkbuilders.intel.com/docs/networkbuilders/accelerate-snort-performance-with-hyperscan-and-intel-xeon-processors-on-public-clouds-1680176363.pdf, section 2.2: Intel conducts continuous internal development and delivers new Hyperscan releases under Intel Proprietary License (IPL) beginning from 5.5 for interested customers. Please consider adding support to compile vectorscan on Windows platforms. Looks like vectorscan is the last resort option for people who wants to use hyperscan functionality as opensource on Windows.

[markos]

@apismensky Windows support is not something that is on the horizon at the moment. If it happens we will have to support both x86 and Arm, or not at all. It would have to be added as supported architecture on the CI and pass all the quality checks as well as supporting multiple versions of Windows. Even if someone sends a full PR we would probably not include it as it would be quite intrusive and full of special cases and we would be unable to actually support it. So, unless certain criteria are satisfied we will probably not include it soon. However, we would be willing to discuss future Windows support as part of a long-term contract work. Please send email to info@vectorcamp.gr if you are interested.

[gliwka]

@apismensky Even if someone sends a full PR we would probably not include it as it would be quite intrusive and full of special cases and we would be unable to actually support it. So, unless certain criteria are satisfied we will probably not include it soon.

@markos Do you refer here to the old MSVC compiler support, that complicated the hyperscan codebase with many ifdefs and specialities in the build script?

What do you think about enabling windows support with clang or mingw64 instead?

Today I was investigating windows support for vectorscan java bindings and was able to compile it using the MINGW64-toolchain on windows x64. I was also able to compile it on aarch64 using clang (woa64 builds against MSVC runtime). Unit tests pass on windows with both toolchains. The only patch needed was a single missing exported symbol (hs_compile_lit_multi) in the hs.def file.

Since the codebase in its current form on the master branch seems currently quite compatible with windows, it might be ideal to setup CI soon to ensure it stays that way.

[apismensky]

Even if the support for ARM64 is not fully functional, just supporting x64 would bring a lot of value here. @gliwka thanks for your contribution.

[markos]

@gliwka yes I was referring to MSVC, you are correct. Well, as long as we don't have to fill the place with ifdefs, I would be fine with supportin both Arm and x86 on Windows. Obviously limited compiler/windows version support, we cannot support every combination. But I have the impression that eventually someone will ask for MSVC support which is a big negative on our side.

We will investigate this for the release after (so 5.4.13).

Thank you for the clarification.

[gliwka]

Further note: Microsoft also ships clang (also cmake) with their Visual Studio Releases and has integration with MSBuild via clang-cl.exe. Maybe it’s possible to build it using that as well. Haven’t tried that yet.

@markos Feel free to reach out if you need pointers reproducing the toolchain.

[markos]

Thank you @gliwka I will do that when I'm ready to enable Windows support in the next version.

[apismensky]

@markos - any update on this issue?

[markos]

@apismensky sorry, Windows support is not in our plans. Even if someone would send us a PR, we would probably not accept it as we would not be able to verify it or support it. If someone could get it working on WLS or minimal changes to cmake and using compilers such as clang, we would be interested in reviewing it. But anything that involves significant changes to the code, is probably going to be rejected. Perhaps, I have been misunderstood in my previous comments. We will consider adding support for Windows in the next version, 5.4.13, but that's not a commitment that we will do so. If someone really needs Windows support, we might discuss a contract under some conditions.

[iseki0]

Could you share your reason to not support Windows? Even you want to reject the PR for it. Because windows is a popular platform, your decision might be looks strange. ( Sorry for my bad English, English is not my native language. @markos

[markos]

@iseki0 please let me be clear on this, as a platform Windows may be fine, but it's not really a platform one would choose for networking. BSD is a far more likely candidate (many routers are using BSD) and that's why we added support for it. And it was easy to do so. Windows has many peculiarities and doesn't follow POSIX standards and MS compilers are a complete mess to support, they are historically following their own standards and not the language standards (or at least not fully). Plus a lot of the SIMD stuff uses GNU extensions that are also supported by Clang/LLVM. We would need to modify the project with a ton of #ifdef everywhere to make it work with Windows. That's what the original project (Hyperscan) did and it was left to bit rot, no one was even checking if the Windows port worked for a long time.

Having said that, with WSL, it might actually be easier to do it nowadays. So if someone can provide with a PR that is easy to integrate and only involves a minor changes (eg add an include in a few places), we might accept it as an unsupported platform. But we will not accept anything else more invasive, nothing that even looks close as the old Windows support for Hyperscan.

If you really need proper Windows support, a proper implementation and support contract would have to be put in place, to add support for it, which would need:

• add server VMs, both for x86 and Arm to do constant compiles on every PR, so integration with buildbot, github
• setup development systems for both x86 and Arm to let developers work on the implementation.

So, the only way for this to happen is to either do the work yourself and request our review for inclusion -which may or may not happen- or negotiate a contract with us to make it work -which may also not happen as development on Windows is not within our immediate plans.

Again, as a last suggestion, if you could show us that the porting requires only minimal changes, I can promise you that we would at least consider adding it as an unsupported platform. But it would have to work on both x86 and Arm.

I'm afraid I'll have to close this as wontfix.

[iseki0]

Thank you for your cleaify, since I'm not familier with C++ development. 😊