Issue compiling with libX11 1.8.x

[tpwrules]

I was unable to build the latest commit of VirtualGL on my system which has libX11 1.8.1. I came across #205 which had the same error message I got but my source tree included that patch.

I did some sleuthing and saw that the test for libX11 1.8.x was failing even though it was the version used. Further sleuthing revealed that the test failed because my compiler was giving errors that the parameter names were omitted in the prototype.

It seems that CMake checks the prototype by defining a function with the prototype you give and I guess making sure the compiler doesn't complain the definition mismatches the prototype. For whatever reason GCC 11.3 this goes fine, but GCC 9.5 dies because the function definition is malformed.

For posterity, here is the faulty file it generated:

#include <X11/XKBlib.h>

static void cmakeRequireSymbol(int dummy, ...) {
  (void) dummy;
}

static void checkSymbol(void) {
#ifndef XkbOpenDisplay
  cmakeRequireSymbol(0, &XkbOpenDisplay);
#endif
}

Display *XkbOpenDisplay(_Xconst char *, int *, int *, int *, int *, int *) {
  return NULL;
}

#ifdef __CLASSIC_C__
int main() {
  int ac;
  char*av[];
#else
int main(int ac, char *av[]) {
#endif
  checkSymbol();
  if (ac > 1000) {
    return *av[0];
  }
  return 0;
}

I fixed this on my system by replacing Display *XkbOpenDisplay(_Xconst char *, int *, int *, int *, int *, int *) by Display *XkbOpenDisplay(_Xconst char *a, int *b, int *c, int *d, int *e, int *f) in server/CMakeLists.txt.

I am not sure if this changes the behavior if libX11 1.7.x or earlier is used, I did not check. But I assume GCC 9.5 is not the only compiler which has this behavior.

[dcommander]

Should be fixed now