porting to windows

[shimondoodkin]

I had tried to compile it on MinGW, but it didn't work out because of signals

MinGW has only a small amount of signals

those are missing: I had defined them at the bottom of common.h , not sure what is the effect of this.

define SIGUSR1 26

define SIGUSR2 27

define SIGXFSZ 28

ideas for signals alternative: http://www.perlmonks.org/?node_id=1022364 https://social.msdn.microsoft.com/Forums/vstudio/en-US/8d128695-1727-434b-9fd1-7b74b2bf25ea/need-a-better-way-for-inter-thread-communication-?forum=vcgeneral

also had an error:

signals.c:78:20: error: storage size of 'act' isn't known struct sigaction act;

migw's signal.h

#ifndef _INC_SIGNAL
#define _INC_SIGNAL

#include <crtdefs.h>
#include <pthread_signal.h>

#ifdef __cplusplus
extern "C" {
#endif

#ifndef _SIG_ATOMIC_T_DEFINED
#define _SIG_ATOMIC_T_DEFINED
  typedef int sig_atomic_t;
#endif

#define NSIG 23

#define SIGINT 2
#define SIGILL 4
#define SIGABRT_COMPAT 6
#define SIGFPE 8
#define SIGSEGV 11
#define SIGTERM 15
#define SIGBREAK 21
#define SIGABRT 22       /* used by abort, replace SIGIOT in the future */
#define SIGABRT2 22

#ifdef _POSIX
#define SIGHUP  1   /* hangup */
#define SIGQUIT 3   /* quit */
#define SIGTRAP 5   /* trace trap (not reset when caught) */
#define SIGIOT  6       /* IOT instruction */
#define SIGEMT  7   /* EMT instruction */
#define SIGKILL 9   /* kill (cannot be caught or ignored) */
#define SIGBUS  10  /* bus error */
#define SIGSYS  12  /* bad argument to system call */
#define SIGPIPE 13  /* write on a pipe with no one to read it */
#ifdef __USE_MINGW_ALARM
#define SIGALRM 14  /* alarm clock */
#endif
#endif

  typedef   void (*__p_sig_fn_t)(int);

#define SIG_DFL (__p_sig_fn_t)0
#define SIG_IGN (__p_sig_fn_t)1
#define SIG_GET (__p_sig_fn_t)2
#define SIG_SGE (__p_sig_fn_t)3
#define SIG_ACK (__p_sig_fn_t)4
#define SIG_ERR (__p_sig_fn_t)-1

  extern void **__cdecl __pxcptinfoptrs(void);
#define _pxcptinfoptrs (*__pxcptinfoptrs())

  __p_sig_fn_t __cdecl signal(int _SigNum,__p_sig_fn_t _Func);
  int __cdecl raise(int _SigNum);

#ifdef __cplusplus
}
#endif
#endif

pthread_signal.h

#ifndef WIN_PTHREADS_SIGNAL_H
#define WIN_PTHREADS_SIGNAL_H

/* Windows has rudimentary signals support.  */
#define pthread_sigmask(H, S1, S2) 0

#endif /* WIN_PTHREADS_SIGNAL_H */

just to give some sense about the errors: the last one i did not knew how to solve .

$ make
Making all in lib
make[1]: Entering directory '/c/Users/user/Downloads/lbzip2-2.5/lbzip2-2.5/lib'
  GEN      alloca.h
  GEN      errno.h
  GEN      c++defs.h
  GEN      arg-nonnull.h
  GEN      warn-on-use.h
  GEN      fcntl.h
  GEN      inttypes.h
  GEN      math.h
  GEN      stdio.h
  GEN      stdlib.h
  GEN      string.h
  GEN      sys/stat.h
  GEN      sys/time.h
  GEN      sys/types.h
  GEN      time.h
  GEN      unistd.h
  GEN      wchar.h
make  all-recursive
make[2]: Entering directory '/c/Users/user/Downloads/lbzip2-2.5/lbzip2-2.5/lib'
make[3]: Entering directory '/c/Users/user/Downloads/lbzip2-2.5/lbzip2-2.5/lib'
  CC       dtotimespec.o
  CC       gettime.o
  CC       math.o
  CC       printf-frexp.o
  CC       printf-frexpl.o
  CC       stat-time.o
  CC       timespec.o
  CC       timespec-add.o
  CC       timespec-sub.o
  CC       unistd.o
  CC       utimens.o
  CC       xmalloc.o
  CC       xsize.o
  CC       asnprintf.o
  CC       fprintf.o
  CC       frexp.o
  CC       frexpl.o
  CC       fseterr.o
  CC       fstat.o
  CC       gettimeofday.o
  CC       isnanl.o
  CC       malloc.o
  CC       msvc-inval.o
  CC       printf-args.o
  CC       printf-parse.o
  CC       stat.o
  CC       vasnprintf.o
  AR       libgnu.a
make[3]: Leaving directory '/c/Users/user/Downloads/lbzip2-2.5/lbzip2-2.5/lib'
make[2]: Leaving directory '/c/Users/user/Downloads/lbzip2-2.5/lbzip2-2.5/lib'
make[1]: Leaving directory '/c/Users/user/Downloads/lbzip2-2.5/lbzip2-2.5/lib'
Making all in src
make[1]: Entering directory '/c/Users/user/Downloads/lbzip2-2.5/lbzip2-2.5/src'
  CC       compress.o
  CC       crctab.o
  CC       decode.o
  CC       divbwt.o
  CC       encode.o
encode.c: In function 'encoder_init':
encode.c:121:3: warning: implicit declaration of function 'bzero' [-Wimplicit-function-declaration]
   bzero(s->cmap, 256u * sizeof(bool));
   ^~~~~
encode.c:121:3: warning: incompatible implicit declaration of built-in function 'bzero'
encode.c: In function 'generate_initial_trees':
encode.c:829:5: warning: incompatible implicit declaration of built-in function 'bzero'
     bzero(&s->length[t][a], b - a);
     ^~~~~
encode.c: In function 'assign_codes':
encode.c:901:3: warning: incompatible implicit declaration of built-in function 'bzero'
   bzero(tree, sizeof(tree));
   ^~~~~
encode.c: In function 'generate_prefix_code':
encode.c:1060:5: warning: incompatible implicit declaration of built-in function 'bzero'
     bzero(s->frequency, nt * sizeof(*s->frequency));
     ^~~~~
  CC       expand.o
expand.c: In function 'on_input_avail':
expand.c:842:3: warning: implicit declaration of function 'bzero' [-Wimplicit-function-declaration]
   bzero((char *)buffer + size, missing);
   ^~~~~
expand.c:842:3: warning: incompatible implicit declaration of built-in function 'bzero'
  CC       main.o
main.c: In function 'info':
main.c:107:5: warning: implicit declaration of function 'flockfile' [-Wimplicit-function-declaration]
     flockfile(stderr);                          \
     ^
main.c:123:1: note: in expansion of macro 'DEF'
 DEF(info   (                                 const char *fmt, ...), 0,0,0,0,1)
 ^~~
main.c:117:7: warning: implicit declaration of function 'funlockfile' [-Wimplicit-function-declaration]
       funlockfile(stderr);                      \
       ^
main.c:123:1: note: in expansion of macro 'DEF'
 DEF(info   (                                 const char *fmt, ...), 0,0,0,0,1)
 ^~~
main.c: In function 'xstrtol':
main.c:173:12: warning: implicit declaration of function 'index' [-Wimplicit-function-declaration]
   endptr = index(suffix, *endptr);
            ^~~~~
main.c:173:12: warning: incompatible implicit declaration of built-in function 'index'
main.c: In function 'output_regf_uninit':
main.c:867:13: warning: implicit declaration of function 'fchown' [-Wimplicit-function-declaration]
   if (-1 == fchown(outfd, sbuf->st_uid, sbuf->st_gid)) {
             ^~~~~~
main.c:877:15: warning: implicit declaration of function 'fchmod' [-Wimplicit-function-declaration]
     if (-1 == fchmod(outfd, sbuf->st_mode & (S_IRWXU | S_IRWXG | S_IRWXO))) {
               ^~~~~~
  CC       parse.o
  CC       process.o
  CC       signals.o
signals.c: In function 'xempty':
signals.c:37:7: warning: implicit declaration of function 'sigemptyset' [-Wimplicit-function-declaration]
   if (sigemptyset(set) != 0)
       ^~~~~~~~~~~
signals.c: In function 'xadd':
signals.c:44:7: warning: implicit declaration of function 'sigaddset' [-Wimplicit-function-declaration]
   if (sigaddset(set, sig) != 0)
       ^~~~~~~~~
signals.c: In function 'xpending':
signals.c:58:7: warning: implicit declaration of function 'sigpending' [-Wimplicit-function-declaration]
   if (sigpending(set) != 0)
       ^~~~~~~~~~
signals.c: In function 'xmember':
signals.c:68:8: warning: implicit declaration of function 'sigismember' [-Wimplicit-function-declaration]
   rv = sigismember(set, sig);
        ^~~~~~~~~~~
signals.c: In function 'xaction':
signals.c:78:20: error: storage size of 'act' isn't known
   struct sigaction act;
                    ^~~
signals.c:84:7: warning: implicit declaration of function 'sigaction' [-Wimplicit-function-declaration]
   if (sigaction(sig, &act, NULL) != 0)
       ^~~~~~~~~
signals.c: In function 'halt':
signals.c:232:9: warning: implicit declaration of function 'sigsuspend' [-Wimplicit-function-declaration]
   ret = sigsuspend(&saved);
         ^~~~~~~~~~
signals.c: In function 'xraise':
signals.c:257:7: warning: implicit declaration of function 'kill' [-Wimplicit-function-declaration]
   if (kill(pid, sig) != 0)
       ^~~~
signals.c: In function 'bailout':
signals.c:308:1: warning: 'noreturn' function does return
 }
 ^
make[1]: *** [Makefile:1342: signals.o] Error 1
make[1]: Leaving directory '/c/Users/user/Downloads/lbzip2-2.5/lbzip2-2.5/src'
make: *** [Makefile:1308: all-recursive] Error 1

[kjn]

Some of the above warnings can be fixed easily:

• backporting commit 8702655 should help with some warnings about undeclared functions (index, bzero)
• flockfile/funlockfile can be replaced by any mutex calls (I used flockfile/funlockfile mostly because they can never fail)

Regarding signals: lbzip2 uses signals for both event handling and inter-thread communication. SIGUSR1 and SIGUSR2 can be replaced by other forms of inter-thread communication. From what I just read, SIGTERM, SIGXFSZ and SIGPIPE are not generated under Win32 and Win32 creates separate thread for handling SIGINT. Therefore it looks like all signal handling code could be reduced to a single handler for SIGINT (Ctrl+C) and some form of inter-thread communication (eg. mutex and condvar). There should be no need for signal masks as the only external signal (SIGINT) is always handled in a separate thread.

[Cyken-Zeraux]

Has any progress been made on this? lbzip2 would be incredibly useful on Windows,

[shimondoodkin]

I did all the rewrites I knew well and could do effectively.

I did not implemented the signalling rewrite.

The interthread communication requires to understand the communication why the threads communicate and what it tries to achieve and where it is in the source code. maybe the author can explain how the threads work.

בתאריך יום א׳, 2 באפר׳ 2017, 06:56, מאת CZauX ‏notifications@github.com:

Has any progress been made on this? lbzip2 would be incredibly useful on Windows,

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kjn/lbzip2/issues/16#issuecomment-290963082, or mute the thread https://github.com/notifications/unsubscribe-auth/AATMYMea95EdvKVVLO26bm4rnveXAS2sks5rrxxegaJpZM4Jewy0 .

[kjn]

I will try to make lbzip2 build and work on MinGW. @shimondoodkin It would help if you could share your existing code.

[shimondoodkin]

I believe all my changes are in my fork https://github.com/shimondoodkin/lbzip2

[nanoant]

@kjn @shimondoodkin I made a little attempt to build lbzip2 with MinGW, but there was quite enormous part missing, especially missing signal constants, file locking such as flockfile etc. I dunno it replacing these parts with dummy implementations that return success if good way to go. Additionally, gnulib dependency does not make it easier. Altogether looks like a lot of work to get it running natively on Windows.

Probably it would be easier to start when the project was based CMake for example not autotools, at least this would make it no more dependent on shell (Bash). This way pulling just GCC (e.g. x86_64-8.1.0-release-posix-seh-rt_v6-rev0) out of MinGW would be sufficient - no need to install full environment. Just thinking loud.

Also maybe having it C11 would make all the threading and time calculation easier?

[tansy]

As far as I'm aware of only analogical, parallel compressor that was, or shall I say can be compiled under windows is pigz; with some minor patching incorporatedin version 2.6, and works fine. I use it personally. It uses its own interface to (p)threads. Also plzip is known to be used under cygwin, mingw (since version 1.8), not msvc. It uses pread/pwrite that are not supported in mingw nor in msvc but with patch included in mentioned version 1.8 it does very well - I never had a problem with it. Anyhow it's not thread related issue per se.

Reason I'm talking about that, is that - there is a path already paved and maybe their solutions can be used with lbzip2.