search

OpenVPN / easy-rsa

easy-rsa - Simple shell based CA utility
Other
4k stars 1.19k forks source link

EasyRSA on Windows 11 with `mksh` hangs #1075

Closed TinCanTech closed 1 month ago

TinCanTech commented 7 months ago

The busybox installation detailed below is for historical reference ONLY.

This discussion has moved onto a debate concerning future development of MKSH. Skip-to: https://github.com/OpenVPN/easy-rsa/issues/1075#issuecomment-2270052361

Follow-up set: #1078

This is specifically:

Required testing:

Volunteers welcome.

If you have access to any version of Windows then you can help by testing.

TinCanTech commented 7 months ago

First test:

On Windows 11, open sh.exe.

To test command read, simply enter the word read into the terminal and press [enter]. Then press [enter] again, and you should return to the prompt.

This MUST be fixed -- There is NO work-around.

TinCanTech commented 7 months ago

Instructions for testing busybox.exe:

Requirements:

git setup and download TinCanTech-upload-busybox:

Install busybox.exe for use:

Change your current Windows Easy-RSA installation to use busybox.exe:

Start your current Windows Easy-RSA installation from EasyRSA-Start.bat.

TinCanTech commented 7 months ago

Follow-up set-to: #1076 #1077

1078

OutOfEspresso commented 1 month ago

Is this MKSH hangup still an issue or are you fine with busybox?

R39-w32-beta14 of the MKSH port was released in an early state where the developer was not happy with. Similar problems where observed in some situations when Win 10 was introduced.

There is a R39-w32-beta28 as of 2017 available. It contains some stability fixes which bay be worth a try. If anybody is interested, i can provide it. Unfortunaltely, i actually dont have access to a Win 11 instance but will try it out asap.

TinCanTech commented 1 month ago

@OutOfEspresso Hi,

Busy-box for Windows would be my preferred solution because we can build it ourselves. However, OpenVPN developers feel that this is another unnecessary dependency.

If we have a new version of MKSH then I can provide a .zip file for Windows testing. All I need is a link to the MKSH project.

OutOfEspresso commented 1 month ago

To be honest, there is no MKSH W32 project. The port was done since we had to replace the commercial "MKS Toolkit" with a free solution. We did not find anything suitable and thus, decided to do a port of MKSH our own and give the result back to the community as a one shot. This shot was beta14, which had to be released before 2013-06-30 due to legal reasons.

For the product we needed it for, it did its job. After the product requiring it was depreciated, the only remaining usecase was my personal use. As mentioned, i made some fixes until 2017 and thats it for now.

I attached the latest version just here mksh-w32-beta28.zip[Mod: URL Redacted]. Good luck :-)

Fun fact: As you may have seen, this port lacks filename completion and the ksh-like history using vi syntax. I never found the time to implement this. Today i came across this missing feature one more time and did a Google search to see if somebody else made a more complete port during the last years. What i found was this issue which forced me to create a GitHub Account and leave the above comment...

TinCanTech commented 1 month ago

@OutOfEspresso Sorry but your random binary is not suitable for EasyRSA.

Perhaps, if you publish the source code and build instructions on your new github homepage ..

OutOfEspresso commented 1 month ago

...i was afraid and i understand. But that is all I can offer for now.

If anybody tests, states that it is working and is interested in continue using it, i can provide the sources for this release too.

...just saw your edit...

Ok, we are thinking towards the same direction. But before i take this effort, please try out if beta 28 fixes the problem. If yes, i will get in touch with the mirbsd team to check, how to provide the sources.

TinCanTech commented 1 month ago

I am not in the habit of running random binaries from strangers on the internet and this is no exception.

If you are serious then do what you need to do.

avih commented 1 month ago

The port was done since we had to replace the commercial "MKS Toolkit" with a free solution. We did not find anything suitable and thus, decided to do a port of MKSH our own and give the result back to the community as a one shot. This shot was beta14, which had to be released before 2013-06-30 due to legal reasons.

Who is "we", and what's the context?

Is the current sh.exe which is R39-w32-beta14 in the current release the same binary which you or your colleagues compiled back then?

EDIT:

As far as I can tell (by the license) it was by Scalaris AG https://github.com/OpenVPN/easy-rsa/blob/7a372a4a8f84bb633389cdd117b0c0e434a1a90c/distro/windows/Licensing/mksh-Win32.txt#L4-L6

And Scalaris AG switched ownership on 2013-07-01 - https://www.post.ch/en/about-us/news/2013/swiss-post-acquires-scalaris-ag-software-company .

How did that binary get from Scalaris AG into EasyRSA for windows? Was it taken into the project back then as a binary without source code [patches] or build instructions?

OutOfEspresso commented 1 month ago

Exactly. "we" was Scalaris AG and I was the one who did the Port. In 2013, where Scalaris AG was due to be sold to Swiss Post Solutions AG, the port was ongoing but main functionallity working. On the other hand, nobody knew if Swiss Post Solutions is willing to contribute things as open source and I myself had no idea, who would be a person do decide this and how long it would take.

Thus, a Scalaris collegue with power of procuration decided to do this as long as Scalaris is on its own and we released the latest beta available end of June 2013:

$ echo "$KSH_VERSION" @(#)MIRBSD KSH R39-w32-beta14 $Date: 2013/06/28 21:28:57 $

To complete the history: End of 2021, Swiss Post Solutions was sold again (https://www.post.ch/en/about-us/media/press-releases/2021/swiss-post-transfers-swiss-post-solutions-to-new-owner) and is now SPS AG.

How did it get into EasyRSA? Most likely by downloading it from http://www.mirbsd.org/permalinks/wlog-10_e20130718-tg.htm#e20130718-tg_wlog-10. I sent the Zip to Thorsten Glaser, who released this way. This Zip contains binary, sources and build info.

And yes, the binary EasyRSA provides is identical to the one contained in this Zip and the one I compiled 2013.

My personal Email, Thorsten knows it, did never change but i did not hear anything and thus was nearly sure that nobody but me ever used it. When moving away my mail domain from the german provider GMX in 2023, the mksh-w32(at)gmx.net address, mentioned in the ReadMe was removed too. Since there was no incoming mail for lots of years, I did not expect this to become a problem. Btw: Did anybody try to contact me this way?

avih commented 1 month ago

Thanks for the info.

So basically this port was developed internally in Salaris AG for internal reasons (replace MSK toolkit), and together you decided to contribute the source patches and binaries back to the mksh project, which was then published at http://www.mirbsd.org/permalinks/wlog-10_e20130718-tg.htm#e20130718-tg_wlog-10 and later the EasyRSA people probably found that binary on that page and decided to use it?

And yes, the binary EasyRSA provides is identical to the one contained in this Zip

Yes, I can confirm that. Both have CRC32 of e09ed595 and md5sum of 66ff1278e852dedd3b005a54483cab42 .

So that's a pretty good start. At least now we have the sources (which need VS2008, and don't include nedmalloc_v105_svn1078.zip, but that file is still available on soureceforge), and presumably it would be possible to produce a binary.

After the product requiring it was depreciated, the only remaining usecase was my personal use. As mentioned, i made some fixes until 2017 and thats it for now.

This sounds to me like you worked on the public mksh-w32 sources for your personal needs, so I don't see how anyone could have an ownership claim on that development, other than yourself, right?

In that case, could you please publish the sources you have which were used to compile that last version from 2017? For instance at a github repo you'll create for it? (you could tag the 2013 sources as beta14, then overwrite the files with the beta 28 sources, and tag it as such).

If you're not familiar enough with git and/or otherwise don't have the time for it, would you publish the sources in a zip and attach it here or elsewhere and link to it?

I don't see a reason why nedmalloc_v105_svn1078.zip can't be included too. As far as I can tell it has the Boost license (so that license would have to be included too, which it is in that zip ffile).

mksh-w32(at)gmx.net address, mentioned in the ReadMe was removed too. Since there was no incoming mail for lots of years

I guess it just worked till this hang issue with windows 11...

avih commented 1 month ago

@OutOfEspresso Sorry but your random binary is not suitable for EasyRSA.

You say that, but that's exactly what distro/windows/bin/sh.exe is, which apparently suites EasyRSA quite well for the past 11 years.

It's the same binary from http://www.mirbsd.org/permalinks/wlog-10_e20130718-tg.htm#e20130718-tg_wlog-10 , and which the mksh project published as is without building it from the source themselves, and the EasyRSA project did exactly the same and used this binary without building it themselves, or else I'm quite sure the md5sum would have changed.

So basically the EasyRSA project trusted the mksh people, who apparently trusted "Michael Langguth and Scalaris AG" and publish their binary unmodified.

That's not at all to say that this beta28 binary should be taken without sources - it shouldn't IMO. Even if the sources do exist, then it would still be unwise to take an unknown binary.

But assuming @OutOfEspresso is indeed Michael Langguth, then it was still kind of him to publish it, and hopefully he'll be able to provide the sources as well.

And at least now we presumably have the sources to distro/windows/bin/sh.exe, and with some effort someone might be able to build them on a modern system, and maybe debug this win11 hang.

And if the beta28 sources would become available as well, then those might help too.

TinCanTech commented 1 month ago

@OutOfEspresso Sorry but your random binary is not suitable for EasyRSA.

You say that, but that's exactly what distro/windows/bin/sh.exe is

I attached the latest version just here mksh-w32-beta28.zip[URL Redacted]. Good luck :-)

Which is not the same version distributed by Easy-RSA.

@OutOfEspresso Thanks for the details you have provided, perhaps you could have done that initially.

@avih Thanks for pushing this.

Once again, I do not think the new binary is suitable under the current circumstances.

avih commented 1 month ago

I attached the latest version just here mksh-w32-beta28.zip[URL Redacted]. Good luck :-)

Which is not the same version distributed by Easy-RSA.

I did not say that. I said the current sh.exe in the distro is exactly such "random binary", presumably by @OutOfEspresso, which EastRSA didn't compile themselves, and the mksh project also didn't compile themselves and instead published it based on trust that it's OK.

I would have definitely expected EasyRSA to compile it themselves and not trust a random binary which even the mksh people say they didn't compile themselves.

But that's water under the bridge now.

I do not think the new binary is suitable under the current circumstances.

Agreed.

Which is why I think that as a starting point, EasyRSA should make an effort to build sh.exe from the available sources of beta-14 at the github CI. It might not be easy, but it should be possible, either to compile it using new VS, or maybe adapt it to mingw.

And if the beta-28 sources become available, check whether those can help too.

TinCanTech commented 1 month ago

I said the current sh.exe in the distro is exactly such "random binary"

No, it is not.

avih commented 1 month ago

I said the current sh.exe in the distro is exactly such "random binary"

No, it is not.

It has the same md5sum as the binary available at http://www.mirbsd.org/permalinks/wlog-10_e20130718-tg.htm#e20130718-tg_wlog-10 and which the mksh people claim they didn't compile themselves and instead publish it because someone sent it to them.

How would you call that anything other than "random binary which we didn't compile ourselves and we don't know who compiled it or from which sources" ?

TinCanTech commented 1 month ago

The binary offered by @OutOfEspresso came with zero verifiable details.

The binaries shipped by Easy-RSA have all been verified, validated, tested and approved.

@avih Your line of argument is not productive.

If you believe there is a security concern then please send details to:

Easy-RSA does not allow distribution of random binaries. That is final.

OutOfEspresso commented 1 month ago

This sounds to me like you worked on the public mksh-w32 sources for your personal needs, so I don't see how anyone could have an ownership claim on that development, other than yourself, right?

No, unfortunately. Further dedelopment followed the same agreement as the initial release. Thus, I am not the sole owner of it. I actually try to receive an Ok to publish the sources of beta 28 and get back if there are news.

If you're not familiar enough with git and/or otherwise don't have the time for it, would you publish the sources in a zip and attach it here or elsewhere and link to it?

Meanwhile I agree'd with the MirOS Project, that we most likely will release it at the same location as beta 14.

avih commented 1 month ago

Meanwhile I agree'd with the MirOS Project, that we most likely will release it at the same location as beta 14.

Sounds good.

Meanwhile, I've compiled beta 14 (after finding and extracting nedmalloc_v105_svn1078.zip into nedmalloc/, and applying the patch at README.1st), and I suspect that the sources don't match the binary, because as far as I can tell there's a leftover bug at the code which strictly prevent compilation. And if this diff crept in, how can we tell what other differences exist between the source and the precompiled binary?

Here's my patch to make it compile. Care to comment if I'm missing something?

patch to comment-out mtime() in liblan/systools.c ```patch commit baabf4e2aee04ce39c3ec23d2189d487bd557731 Author: Avi Halachmi (:avih) Date: Tue Aug 6 20:02:07 2024 +0300 liblan: comment broken and unused mtime() The file liblan/systools.c is required for the build (for "sleep", and more), however, it doesn't compile with WIN32 defined, because "timezone" in mtime() is unresolved (it looks like a WIP bug), and it doesn't compile without WIN32 defined because neither msvc nor mingw have . However, this function is unused by the code, so just comment it out. However also, this might be an indication that the binary mksh.exe in mksh-w32-beta14.zip might be built not from this exact source. The project can now be built using VS 2015 cl.exe (32 bit): - delete mksh/setmode.c (it's unused). - cl /Femksh.exe /D WIN32 /D LIBLAN /I liblan /I nedmalloc liblan/*.c mksh/*.c user32.lib winmm.lib advapi32.lib diff --git a/liblan/systools.c b/liblan/systools.c index a3b09bb..ca2d610 100644 --- a/liblan/systools.c +++ b/liblan/systools.c @@ -104,6 +104,11 @@ void msleep(long ms) /* * get the system time in ms since utc 1970 */ +/* + * mtime() doesn't compile with WIN32 because "timezone" is unresolved. + * however, it's unused by mksh, so disable it for now + */ +/* uint64 mtime(void) { uint64 ret; @@ -134,6 +139,7 @@ uint64 mtime(void) return(ret); } +*/ /* * unique interface to statfs like information ```

And here's the mtime function in liblan/systools.c. Note how timezone is not defined anywhere (also not outside that code):

uint64 mtime(void)
{
    uint64  ret;

#ifdef WIN32
    SYSTEMTIME  st;
    struct tm   tm;

    GetSystemTime(&st);

    tm.tm_year  = st.wYear - 1900;
    tm.tm_mon   = st.wMonth - 1;
    tm.tm_mday  = st.wDay;
    tm.tm_hour  = st.wHour;
    tm.tm_min   = st.wMinute;
    tm.tm_sec   = st.wSecond;
    tm.tm_isdst = 0;

    ret = (uint64)mktime(&tm)*1000;
    ret += (timezone * -1000) + st.wMilliseconds;
#else
    struct timeb    tb;

    ftime(&tb);

    ret= (uint64)tb.time*1000 + tb.millitm;
#endif

    return(ret);
}
OutOfEspresso commented 1 month ago

What VC Release do you use? If i remember correctly, the newest one I tried in the past was VC 2013.

Just tried my original upstream to the MirOS team using the old VC 2008 Express Installation and it compiles fine. The Zip downloaded from the official link also compiles liblan incl. systools.c but not mksh due to the missing nedmalloc, MirOS denied to include.

Thus, i assume that there may be a compatibility issue with newer VC releases.

From my "C:\Program Files (x86)\Visual Studio 9.0\VC\include\time.h":

#if     !__STDC__ || defined(_POSIX_)

/* Non-ANSI names for compatibility */

#define CLK_TCK  CLOCKS_PER_SEC

/*
daylight, timezone, and tzname are not available under /clr:pure.
Please use _daylight, _timezone, and _tzname or 
_get_daylight, _get_timezone, and _get_tzname instead.
*/
#if !defined(_M_CEE_PURE)
_CRT_INSECURE_DEPRECATE_GLOBALS(_get_daylight) _CRTIMP extern int daylight;
_CRT_INSECURE_DEPRECATE_GLOBALS(_get_timezone) _CRTIMP extern long timezone;
_CRT_INSECURE_DEPRECATE_GLOBALS(_get_tzname) _CRTIMP extern char * tzname[2];
#endif /* !defined(_M_CEE_PURE) */

And, believe it or not:

mksh@lanp > ls -l ../../mksh.exe ksh.exe
-rwxrwxrwx  1 lan 0 293888 2013-06-29 00:37 ../../mksh.exe
-rwxrwxrwx  1 lan 0 293888 2024-08-07 23:41 ksh.exe

mksh@lanp > cmp -l ksh.exe ../../mksh.exe
   241 375  44
   242 351  20
   243 263 316
   244 146 121
   321 321  22
   322 175 217

mksh@lanp > ./ksh -c "echo \$KSH_VERSION"
@(#)MIRBSD KSH R39-w32-beta14 $Date: 2013/06/28 21:28:57 $

As mentioned, i have a library (liblan) containing some personal "Swiss Knife" Tools I wrote over the time. Useful things and things, I smile about nowadays... Anyway, I just used this lib for the MKSH port and removed any unused source files. But i did not remove unused functions from the remaining C files.

Edit

Forgot to mention: Cool! As i see it, you are the second one, succsessfully compiling mksh.exe on the planet. :-)

avih commented 1 month ago

From my "C:\Program Files (x86)\Visual Studio 9.0\VC\include\time.h": _CRT_INSECURE_DEPRECATE_GLOBALS(_get_timezone) _CRTIMP extern long timezone;

Interesting.

I used msvc 2015 (but sort of portable thing, extracted from a full install, so I don't have the IDE etc, just the build tools and the SDK).

In time.h there's _timezone (and struct timezone) but not long timezone. FWIW, in mingw time.h it's the same (no timezone).

However, in MSVC 6 there is indeed timezone in time.h, but there were too many errors when trying to compile mksh.

So either 2013 is the last which has timezone, or my msvc 2015 SDK is lacking stuff. But anyway, that was the only blocker, and mtime is unused, so I just commented it out. Other than this, by default there were only 3 printf format warnings at mksh/win32.c.

Forgot to mention: Cool! As i see it, you are the second one, succsessfully compiling mksh.exe on the planet. :-)

I guess. It doesn't look like the EasyRSA people compiled it..

TinCanTech commented 1 month ago

@avih Your repeated attacks impugn not only Easy-RSA but also OpenVPN and MirBSD.