Transparent unions break 3C declaration merging

[mattmccutchen-cci]

The following code (a simplified version of what is currently happening in sys/socket_checked.h) is valid C and Checked C:

struct sockaddr1 {
  int xs;
};
struct sockaddr2 {
  unsigned int xu;
};

union sockaddr_p_tu {
  struct sockaddr1 *sap1;
  struct sockaddr2 *sap2;
} __attribute__((transparent_union));

extern void mybind(union sockaddr_p_tu sap);
extern void mybind(struct sockaddr1 *sap);

but produces the following error from 3C (I presume because struct sockaddr1 * has a pointer at the top level but union sockaddr_p_tu doesn't):

/home/matt/test/transparent-union.c:14:13: fatal error: merging failed for 'mybind' due to transplanting between pointers with different depths during internal merge for parameter 0
extern void mybind(struct sockaddr1 *sap);
            ^
Failure occurred while trying to add variables. Exiting.

If the Checked C compiler isn't changed to outright reject this code as part of microsoft/checkedc#441, then ideally 3C should reject it with an error message clear enough for end users to quickly identify the problem.

[mattmccutchen-cci]

For whatever reason, I thought more about this issue this morning. I'm just recording my thoughts and not asking to change the prioritization of this issue.

If we want 3C to be able to process programs that #include <sys/socket.h> and set _GNU_SOURCE (such as Icecast and ImageMagick in their original form, which would be really nice), I propose that we proceed as follows:

• Change sys/socket_checked.h as described in microsoft/checkedc#441.
• Change 3C so that if a function foo has a declaration that uses a transparent union, that declaration is completely ignored by 3C and any other declarations of foo are treated as unwritable and constrained accordingly. That does mean that if foo has no declaration that doesn't use a transparent union, then 3C will treat it having no declaration at all, but for the socket functions, we should normally always be implicitly including the checked declarations that don't use transparent unions. Hopefully this workaround will be simple to implement now that the declaration merging process has been cleaned up in #463 and will have very little risk of breaking anything else. If we want, 3C could issue an error or warning if any declaration of foo is actually in a writable file; otherwise we might have to add new root-cause diagnostics for this situation. There are various ways we could make the workaround more narrowly targeted (and probably more complex to implement), but the above is probably enough for the socket functions, so I wouldn't bother with anything more complex.

Hopefully things would work at that point. However, since we know Microsoft hasn't tested _GNU_SOURCE at all, I wouldn't be surprised if there are remaining glitches that require minor changes to the Checked C compiler, for instance.