Windows support: strategy question (linux emulation or "native")

[steffengy]

Not sure if this is the right repository to place this issue, since it also affects ccir, but i feel it might belong a bit more here.

Is a windows implementation planned?

I see two very different ways of implementing it:

• emulating the required linux APIs ("mapping" to the windows equivalents at runtime or at a later stage of the project during compilation)
• implementing it natively (as the linux implementation, proxying WINAPI calls, this seems to require quite much work?) <---

Which of those (or maybe an other even better idea?) would you consider the favorable solution?

Other todos:

• [ ] Get rid of several win stubs (by not requiring them on win)
• [ ] get rid of workaround, temporary solutions, todos, ...

[steffengy]

BTW currently working on implementing open64 for windows and wondering what the reason to use syscall.syscall(SYS_OPEN over syscall.Open for linux is? (For windows that might lead to saving several lines of syscall-boilerplate)

[cznic]

BTW currently working on implementing open64 for windows and wondering what the reason to use syscall.syscall(SYS_OPEN over syscall.Open for linux is? (For windows that might lead to saving several lines of syscall-boilerplate)

Simply never considered. Direct mapping of libc syscallls to real syscalls was the first attempt tried - and kept if it happened to just work.

[steffengy]

Just wondering if there's a performance implication (not sure how aggressively GO inlines). I'll probably do a short benchmark and then use whatever seems the best solution for each _individual case.

[steffengy]

NOTE: Doesn't do inlining for syscall.Open after looking at the assembly. There's also additional overhead for checks the GO-code contains (which we might not want) so that route of direct mapping seems to make sense here too. Ofcourse it's still an individual decision.

[cznic]

Don't worry about performance too much - yet. The VM is already ~100 times slower than native code. Improving a particular libc function implemented in Go can thus have a rather limited impact on the overall speed, unless executed extremely often.

Even more importantly, I still hope to be able to soon-ish-ly translate the IR to Go. That's the strategic performance-wise optimization target. The ccgo project will use the ccir/libc generated .h/.go files and it will re-use the virtual/name_os_arch.go libc implementing stuff. Initially perhaps just by copy-pasting parts of it. Later some common crt (C run time) may become reasonable.

There's an intermediate step possible which I'm still contemplating: Instead of translating IR to Go, it's possible also to "unroll" the VM opcodes to the semantically equal Go code. According to some previous experiments, that would speedup most things by at least ~50%. But speedups are sometimes per se a bit mysterious mater ;-)

[cznic]

NOTE: Doesn't do inlining for syscall.Open after looking at the assembly. There's also additional overhead for checks the GO-code contains (which we might not want) so that route of direct mapping seems to make sense here too.

Be lazy as much as you can. There's still some work to do ahead where you'll probably don't have the luxury option of just calling the kernel.

OTOH, you're now about a dozen of "syscalls" away from having your Windows port complete. You are incredible. Especially when I observe you can easily understand the spaghetti code of mine you have to deal with :open_mouth:

Kudos.

[steffengy]

Still having some issues with builtin.h: __builtin_offsetof is required by windows.h but not found during go generate. (I added windows.h to generator.go)

If I add #define __builtin_offsetof(st, m) ((__SIZE_TYPE__)(&((st *)0)->m)) from builtin.h to predefined.h I get some errors because something seems to pull in the definitions of ___header etc. into stddef_windows_amd64.h, which then errors with cannot redefine macro using a replacement list of different length.

IMO This should be picked up from builtin.h, how does that work on linux?

[cznic]

On Linux: __builtin_offsetof is defined in builtin.h, which does #include <predefined.h>. Ie. defining it again in predefined is not the correct place. predefined.h should define only what cpp -dM shows and what was slurped when doing go generate.

Please try to run the test like, let's say the problem shows in TestGCCExec test named foo.c

$ go test -v -cpp -run GCC -re foo.c

and redirect stderr to a log. This way you can inspect exactly what the compiler sees after preprocessing foo.c and track exactly the source of every definition/redefinition. HTH

[steffengy]

How should I go about that since go generate for windows.h already fails with:

m:\mingw\x86_64-w64-mingw32\include\winnt.h:8359:100: unexpected typedefname NT_TIB, expected optional argument expression list or one of [&&, '!', '&', '(', ')', '*', '+', '-', '~', ++, --, _Alignof, character constant, floating-point constant, identifier, integer constant, long character constant, long string constant, sizeof, string literal]

648: #define FIELD_OFFSET(Type, Field) ((LONG) __builtin_offsetof(Type, Field))
8359:  FORCEINLINE struct _TEB *NtCurrentTeb(VOID) { return (struct _TEB *)__readgsqword(FIELD_OFFSET(NT_TIB,Self)); }

The code compiled during go generate AFAIK already includes a `#include ' but as we previously noted, the compiler doesn't seem to be very keen to use that here.

Had a similar issue which is why the following is also in predefined.h even though it also would be in builtin.h, but isn't enough for some reason:

  #define __builtin_va_start(ap, arg) ap = (__builtin_va_list)(&arg)
    #define __builtin_va_end(ap) ap = 0  

[cznic]

Ah, I thought go generate worked already. You're trying to remove some temporary solutions, I guess. There you can try go run generate.go -cpp and hopefully you will get the preprocessor output. But you seem to be doing that already.

m:\mingw\x86_64-w64-mingw32\include\winnt.h:8359:100: unexpected typedefname NT_TIB, expected optional argument expression list or one of [&&, '!', '&', '(', ')', '*', '+', '-', '~', ++, --, _Alignof, character constant, floating-point constant, identifier, integer constant, long character constant, long string constant, sizeof, string literal]

648: #define FIELD_OFFSET(Type, Field) ((LONG) __builtin_offsetof(Type, Field))
8359:  FORCEINLINE struct _TEB *NtCurrentTeb(VOID) { return (struct _TEB *)__readgsqword(FIELD_OFFSET(NT_TIB,Self)); }

In the above, is FORCELINE defined to be something? I would guess it should defined as nothing, i.e only #define FORCEINLINE. I also assume that at processing line 8359 __builtin_offsetof is already defined. I need to look at the windows stuff, please push your latest tree, whatever state it is. I can then also refer to a particular commint, if I see something. Thank you.

The code compiled during go generate AFAIK already includes a `#include ' but as we previously noted, the compiler doesn't seem to be very keen to use that here.

When go generate in ccir/libc runs, builtin.h is not used when generating .h files. It's used in emit() to sanity check the just generated header, because that generated header will be later used by ccir tests while including . The same is done by generator.go in cznic/sqlite.

[steffengy]

I'm trying to add two new headers (process.h, windows.h), which depend on other builtin's, which is why go generate doesn't work again. The current stuff for CCIR is commited. When figuring that out, I might be able to remove some temporary solutions from predefined.h also.

FORCEINLINE is effectively "" (and occurs on 7000 lines before which don't seem to error). go run generator.go returns https://gist.github.com/steffengy/906287910332c7dab2e5278724fce6a9

[cznic]

Thanks. Looking at the tree and the gist, nowhere I can find the definition of FIELD_OFFSET. That would explain the unexpected typedefname NT_TIB, expected optional argument expression list or one of [&&, '!', '&', '(', ')', '*', '+', '-', '~', ++, --, _Alignof, character constant, floating-point constant, identifier, integer constant, long character constant, long string constant, sizeof, string literal] error.

Maybe you can try adding #define FIELD_OFFSET(x, y) __builtin_offset(x, y) to predefine.h?

[steffengy]

@cznic It's already expanded as I read the log? Atleast the last line indicates that:

inline  struct _TEB *NtCurrentTeb(void) { return (struct _TEB *)__readgsqword(((LONG) __builtin_offsetof(NT_TIB, Self))); }

Adding your suggestion to predefined.h and running go generate:

cannot redefine macro FIELD_OFFSET: argument names differ

[cznic]

It's already expanded as I read the log?

You're right. I forgot the -ccp does not show #defne stuff :-(

cannot redefine macro FIELD_OFFSET: argument names differ

What I suggested was nonsensical. FIELD_OFFSET is obviously defined.

[steffengy]

Unfortunately for this case I cannot simply workaround it in predefined.h since there're quite a few builtuns (and because of the conflicts mentioned above), so this time we'll very likely have to get to solve this builtin.h issue.

I still don't understand how that works for the linux builds. I guess there should be references to __builtin stuff too, why does it work there but fails here?

[cznic]

I still don't understand how that works for the linux builds.

I will try to summarize how I think it works. Sorry for being somethimes wrong, I hope this time it's more correct.

1. generator.go first slurps cpp -dM. No .h stuff is used from ccir/libc. Acquired defintions are written to predefined_goos_goarch.h

2. generator.go attempts to generate ccir/libc headers for those listed here. When generating a ccir/libc header, <predefined.h> pulls the definitions created in step 1, but any header included is the system one as we need to get the system defintions for things in stdio.h, stdlib.h etc.

3. According to the mre (macro re) and dre (definition re) regexps, stuff is selected from the resulting AST from step 2. and written to the appropriate foo_goos_goarch.h in ccir/libc

4. What was just created, foo_goos_goarch.h is compiled again, this time using <builtin.h> provided in ccir/libc, effectively testing if it's gonna work later when really used without GCC.

5. If successful, the respective foo_goos_goarch.go file is written containg all the constants just seen in step 4. Here is a missing piece, it should generate also types for ccgo. Not implemented yet.

6. Later, in ccir tests and in sqlite generator.go, the situation when compiling C is the same as in step 4. It forces including <builtin.h> that pulls <predefined.h>, rest is the C input, whatever it is.

... I guess there should be references to __builtin stuff too, why does it work there but fails here?

__builtin_offsetof and friends are defined in builtin.h. On Linux GCC builtins are what the name suggest, they're built in, no need to define them so in step 2 they're known. When later doing ccir tests or sqlite go generate, we supply them from <builtin.h> because we are not GCC and cznic/cc has no builtins. (It has some which are not possible to define in a .h file.)

That's seems to explain how it works on Linux. Something is different [with GCC] on Windows.

I hope I reconstructed what the code does correctly...

[cznic]

Errata: The regexps selects the whole file where the match was found, not a single macro/definition.

[steffengy]

Compiling the following using MINGW-GCC also works:

#include <stdio.h>

typedef struct {
    int a;
    int b;
} Test;

int main() {
    printf("%d", __builtin_offsetof(Test, b)); // 4
}

Grepping through the all MINGW files I also find no definition of __builtin_offsetof, so I conclude it's also a GCC-builtin and very similar (or the same) as linux.

How are they [builtins] known in step 2. under linux? AFAIK they aren't defined in predefined_linux_amd64.h, so how does cznic/cc know about them? Does it somehow ask GCC about them? I think understanding that might help here to figure out what's going wrong.

[cznic]

How are they known in step 2. under linux?

I believe on Linux GCC knows __builtin_offsetof because it's hardcoded/predefined directly in the compiler and no "includable" definition of it exists, even that in this case it can exist. Some builtins are not possible to define in a .h file, that's why I think cpp -dM does not show any builtins.

The question is why GCC on Windows in step 2 does not know the same predefined builtin. My guess is that on Windows GCC is maybe trying to be compatible with C sources written for MS C/C++ compilers and the builtin has a different name. But no googling confirms that so far to me.

[cznic]

Compiling the following using MINGW-GCC also works:

That proves the builtin is known actually.

[steffengy]

Yeah my question is how cznic/cc thinks it's defined on linux, when it's not in cpp -dM (and therefore not in predefined_linux_amd64.h). Does it do some kind of other call to GCC to ask about what builtins its aware of, or how does it accomplish to resolve them on linux?

Or is step 2. actually "compiled" using GCC itself, that'd be unlikely since it's going through cznic/cc as far as I skimmed the code, so I just assume that's not the case.

[cznic]

AFAIK they aren't defined in predefined_linux_amd64, so how does cznic/cc know about them? Does it somehow ask GCC about them?

cznic/cc does not know __builtin_offset, it comes from in ccir/libc, but not in step 2. Which means I'm now lost, I do not understand how step 2. works :-( It does work here, just tried with your latest tree.

Perhaps, when generating a ccir/libc header on Linux, those macros are never used but on Windows they are?

Or is step 2. actually "compiled" using GCC itself, that'd be unlikely since it's going through cznic/cc as far as I skimmed the code, so I just assume that's not the case.

No, it's cznic/cc which is used, you're right.

[cznic]

Perhaps, when generating a ccir/libc header on Linux, those macros are never used but on Windows they are?

The more I think about this, the more it seems plausible to me. If it is so, the lack of them on Linux in step 2. would simply not be a problem. But it is and you Windows port unearthed it.

[steffengy]

Perhaps, when generating a ccir/libc header on Linux, those macros are never used but on Windows they are?

possible, sure. A quick grep into linux include's might shed some light onto this.

Is it possible that we need to call .gccEmu for cznic/cc or is that for someother use case?

[cznic]

Is it possible that we need to call .gccEmu for cznic/cc or is that for someother use case?

That's a horrible hack enabling to run some tests in cznic/cc. I dont believe it can help us. But in a sense it was a precursor of builtin.h in ccir/libc. Kind of.

You've already put some defintions into predefined.h, so maybe just carefully moving required definitions from builtin.h to predefined.h is the way to go?

But I have a feeling the split between builtin.h and predefined.h was necessary otherwise things fell apart on Linux. Anyway, can you please try it? There must be a proper solution somewhere. Exhaustive search is a painful way how to find it, but I have no better idea.

[steffengy]

Adding to predefine.h work so far for __builtin_va_start and __builtin_va_end (They temporarily are there since one of the first commits)

Adding #define __builtin_offsetof(st, m) ((__SIZE_TYPE__)(&((st *)0)->m)) to predefine.h somehow results in a bulk of additional definitions:

diff --git "a/stddef_windows_amd64-4e51852.001.h" "b/stddef_windows_amd64.h"
index 6575ab8..058e839 100644
--- "a/C:\\Users\\steff\\AppData\\Local\\Temp\\TortoiseGit\\stddef_windows_amd64-4e51852.001.h"
+++ "b/D:\\projects\\Go\\gopath\\src\\github.com\\cznic\\ccir\\libc\\stddef_windows_amd64.h"
@@ -32,6 +32,9 @@ errno_t _set_errno (int _Value );
 errno_t _get_errno (int *_Value );
 extern unsigned long __threadid (void );
 extern uintptr_t __threadhandle (void );
+typedef char *__builtin_va_list ;
+typedef __builtin_va_list va_list ;
+typedef void *FILE ;
 #define _INC_CRTDEFS 
 #define _CRT_PACKING (8)
 #define _CRTNOALIAS 
@@ -88,3 +91,25 @@ extern uintptr_t __threadhandle (void );
 #define __STDDEF_H__ 
 #define NULL ( ( void * ) 0 )
 #define offsetof(TYPE, MEMBER) __builtin_offsetof ( TYPE , MEMBER )
+#define _PREDEFINED_H_ 
+#define __str(x) # x // LINE 95
+#define ____header(name, os, arch) __str ( name ## _ ## os ## _ ## arch . h )
+#define ___header(name, os, arch) ____header ( name , os , arch )
+#define __header(name) ___header ( name , __os__ , __arch__ )
+#define _MSC_VER (1300)
+#define __restrict__ 
+#define __int32 long
+#define __int64 long long
+#define _INC_STDARG 
+#define __builtin_va_start(ap, arg) ap = ( __builtin_va_list ) ( & arg )
+#define __builtin_va_end(ap) ap = 0
+#define __builtin_offsetof(st, m) ( ( __SIZE_TYPE__ ) ( & ( ( st * ) 0 ) -> m ) )
+#define _VA_LIST_DEFINED 
+#define __inline inline
+#define __forceinline inline __attribute__ ( ( __always_inline__ ) )
+#define __USE_MINGW_ANSI_STDIO (0)
+#define index(s, c) strchr ( s , c )
+#define rindex(s, c) strrchr ( s , c )
+#define _FILE_DEFINED 
+#define __unaligned 
+#define _X86INTRIN_H_INCLUDED

So somehow this causes e.g. __str to be pulled in, which results in things falling apart like:

stddef_windows_amd64.h:95:9: cannot redefine macro using a replacement list of different length
stddef_windows_amd64.h:96:9: cannot redefine macro using a replacement list of different length
stddef_windows_amd64.h:97:9: cannot redefine macro using a replacement list of different length
stddef_windows_amd64.h:98:9: cannot redefine macro using a replacement list of different length
stddef_windows_amd64.h:99:9: cannot redefine macro using a replacement list of different length
stddef_windows_amd64.h:104:9: cannot redefine macro using a replacement list of different length
stddef_windows_amd64.h:106:9: cannot redefine macro using a replacement list of different length
stddef_windows_amd64.h:109:9: cannot redefine macro using a replacement list of different length
stddef_windows_amd64.h:110:9: cannot redefine macro using a replacement list of different length
stddef_windows_amd64.h:111:9: cannot redefine macro using a replacement list of different length
15:33:39.233141 global.go:205: too many errors

The errors itself are quite strange, not sure why the replacement list should have a different length. (In the best case this would just be some bug and without it redefinition would work)

But I guess even if I somehow manage to fix this I have a feeling that there might be more to pop up. (windows.h and the dependant headers are quite big, and since it's windows expected to do "unexpected" stuff)

[cznic]

Sad news.

FTR: Using https://github.com/cznic/ccir/commit/4e518522a97d1220eb4a3f45b8b01cb21e0afaa1 as a base and moving the defintion of __builtin_offsetof from builtin.h to predefine.h (https://github.com/cznic/ccir/commit/030b115d43286f40e7d1bd7d5eda955d1470f12f) results in:

jnml@4670:~/src/github.com/cznic/ccir/libc$ go generate
15:51:37.848794 generator.go:119: stddef_linux_amd64.h:81:9: cannot redefine macro using a replacement list of different length
stddef_linux_amd64.h:82:9: cannot redefine macro using a replacement list of different length
stddef_linux_amd64.h:83:9: cannot redefine macro using a replacement list of different length
stddef_linux_amd64.h:84:9: cannot redefine macro using a replacement list of different length
stddef_linux_amd64.h:85:9: cannot redefine macro using a replacement list of different length
exit status 1
libc.go:2: running "go": exit status 1
jnml@4670:~/src/github.com/cznic/ccir/libc$ 

So unfortunately, it does break Linux, I recalled that correctly. Anyway, now I have a broken Linux branch, which I can investigate independently from having Winows. At least something.

[steffengy]

That's the same issue (origin) I'm having on windows. I first though that because predefine.h is included before that #define __str(x) #x would expand to #define #x #x, but that would throw a different error.

Hopefully now that we both can look into it, we can figure something out

[cznic]

This fixes the Linux build

diff --git a/libc/generator.go b/libc/generator.go
index 229b106..e604d77 100644
--- a/libc/generator.go
+++ b/libc/generator.go
@@ -847,7 +847,7 @@ func extractCopyright(f string) string {
 }

 func skip(f string) bool {
-       return strings.HasPrefix(f, "predefined_") || f == "<predefine>"
+       return strings.HasPrefix(f, "predefined") || f == "<predefine>"
 }

Please try the same on Windows.

[steffengy]

I get a different error, which seems to be unrelated, so I say very likely yes. Still a bit funny that a possible typo is the cause of this^^

[cznic]

I get a different error, which seems to be unrelated, so I say very likely yes.

Yes, like in 'it fixes the __builtin_offsetof problem on Windows'? :smile:

[steffengy]

Yeah now it seems to related to some other MINGW specific intrinsic stuff... Looking at ifdef's/include to figure out what I need to do and similar stuff (still the same line but __readgsqword this time)

[cznic]

Great! Every progress counts.

[steffengy]

__MINGW_INTRIN_INLINE void __cdecl __debugbreak(void)
{
  __asm__ __volatile__("int {$}3":);
}

It seems strange to me that something like this is included now. If I understand the VM correctly, I'd doubt that assembly instructions would even work like that? (or are they also "translated"?)

[cznic]

No, assembler is not supported and there are no intentions to do so. But the generated headers have no function bodies, so this should be harmless. Or does it trigger some problem?

[steffengy]

It seems to fail during parsing (probably because EnableAsm() / EnableAlternativeKeywords() isn't specified). The actual problem is that I need a definition of __readgsqword.

__debugbreak (and maybe other [likely unreferenced] assembly-intrinsics (including __readgsqword which seems to use gs/segment registers using assembly instructions) are pulled in by setting __MINGW_INTRIN_INLINE.

So I'm wondering if it'd make sense to define __readgsqword as an "external" builtin, which I then would define/handle in cznic/virtual, do you agree that this would be the correct way to handle such builtins?

[cznic]

From here:

These intrinsics are only available in kernel mode, and the routines are only available as intrinsics.

We surely should not be in kernel mode, me thinks.

So I'm wondering if it'd make sense to define __readgsqword as an external builtin, which I then would define/handle in cznic/virtual, do you agree that this would be the correct way to handle such builtins?

Regardless of the kernel mode mystery, the VM has no concept of the GS segment register, so I don't see what the builtin could possibly do.

We can surely enable the assembler and tweak ccir to translate any assemble containing function to a panic instruction, hoping nothing will ever call it. If it happens anyway, there will be a stack trace so we can see where/why it happens and what to do about it. I don't expect SQLite to do anything which ends in calling this intrinsic. WDYT?

[steffengy]

The documentation seems to be wrong there. MSVC TLS (as in thread local storage) uses the GS segment. Reference

Nevertheless I don't expect it to be actually called anytime by SQLITE, it's just necessary because sqlite3.c -> windows.h -> winnt.h which defines functions, which could potentially be used like:

 FORCEINLINE struct _TEB *NtCurrentTeb(VOID) { return (struct _TEB *)__readgsqword(FIELD_OFFSET(NT_TIB,Self)); }

Potentially I see multiple solutions 1) Your proposed assembly+panic solution which would allow __MINGW_INTRIN_INLINE stuff to be defined (and therefore will likely cover most of those) 2) Stubbing the bare-minimum in predefined.h to panic in some kind 3) Stubbing the bare-minimum in predefined.h and handling (panicing) them in cznic/virtual

What would be a good solution in your opinion?

[cznic]

No strong opinion here. Feel free to choose what you feel is better or easier. Stubbing a panic (2) using a define can use the __builtin_abort, FYI.

[steffengy]

I think 2. seems like least work. The only issue I see is that I would want to include the called function so if it should actually happen you can figure out why. Is there a possibility so somehow given __builtin_abort an abort reason? (Something like "__readgsqword stub called")

If not, maybe 3. is a solid alternative.

[cznic]

Is there a possibility so somehow given __builtin_abort an abort reason? (Something like "__readgsqword stub called")

No, but you can copy and adjust the assert macro. It decorates the fail using __func__ and __LINE__. And the VM panic per se should report something useful. (A build tag may be necessary for that, not sure.)

Additionally, FTR, there is a TLS in cznic/virtual: https://github.com/cznic/virtual/blob/769db7bb1f04e2dfad6964c3e4c0961044f4860f/thread.go#L70

If needed, it can be extended or even made run-time configurable in its size etc.

[cznic]

Also, if a builtin is registered, but not dispatched in the giant switch case in vm/cpu.go, there should be also an instruction trap with stack trace citing the opcode (== the builtin) name.

[steffengy]

The generator starts to be really annoying because it doesn't output in the correct order (typedefs are after the type usages) and the workaround grows:

https://github.com/cznic/ccir/commit/cadbea0e7f830d3138fa5fd3fbbe04280e26b362

Unfortunately this workaround might not work anymore (currently struggling with rpc.h, rpcdce.h) since there're so many cross-dependencies.

Any idea if there's a better solution?

[cznic]

The generator starts to be really annoying because it doesn't output in the correct order (typedefs are after the type usages) and the workaround grows:

Do not workaround that. If it's a bug it has to be fixed. But I recall experiencing something similar and the fix was as simple as to add the type name in question to the respective regexp.

Long version: We rely on that the GCC headers include, define and use/refer to things in the correct order (define before use, except where the language somehow permits that [typedef struct foo foo;]). Adding the required type to the dre regexp parameter of header in generate.go should simply include that definition (and its whole source file) in the preprocessing, ie. correct order. Please try and let me know the outcome.

I see you're already doing maybe exactly that. Then it may be necessary to remove any forced includes. I recall falling into that trap as well. The lesson learned was: do not add any #include unless no other way around. Using the regexp to get the definition in the correct order worked except for I think one single case (forgot where).

Sorry for the troubles. The above will hopefully improve things.

[steffengy]

Removed the workaround entirely. It seems like the ordering doesn't work though:

Example for fcntl.h (since thats currently what fails for me):

[...]
_off64_t lseek64 (int fd ,_off64_t offset ,int whence );
[...]
typedef long long _off64_t ;

{"fcntl", "F_WRLCK", "_off64_t|__off_t|open|struct flock|__time32_t"},

Adding a println like here results in the correct ordering on the console (without the filtering applied, logically, so I assume there's something wrong with that):

diff --git "a/src\\github.com\\cznic\\ccir\\libc\\generator.go"
index ea52de1..9d7824a 100644
--- "a/generator-cadbea0.002.go"
+++ "b/\generator.go"
@@ -956,6 +956,7 @@ func header(nm, mre, dre string) {
            // nop
        case 1: // Declaration                  // Case 1
            d := declaration(n.Declaration)
+           fmt.Println(d)
            if dbg || re.MatchString(d) {
                if !m[f] {
                    m[f] = true
@@ -972,32 +973,6 @@ func header(nm, mre, dre string) {
        }
    }

[cznic]

Note the dbg || .... It reminds me that when solving probably exactly the same problem on Linux, I added that option to verify the respective type definition really comes before it's use. Because if that's the case and the definitions from the appropriate files are later on emitted in the discovery order (fo = append(fo, f)) everything should just work. IOW, please try go run generate.go -debug, it may show something useful. Using that option should define more than necessary and can introduce other bugs, but it should hopefully allow to verify if the define/use order is correct before we output to libc/foo.h. If the input order is correct, we just need to find why/where we do not respect that order on output, which should be a simpler problem.

There's an exception to the concept above. Macros, which can effectively define/name/map types, have to be emitted after other definitions. There are other problems when keeping them in the original order. Not sure ATM what was the reason, but I think that was one of the last tweaks needed before finally the generating started to work on Linux.

Please experiment using the above hints/thoughts. The solution, once you find it, will be probably simple. And surely my fault ;-)

PS: I know I've just reiterated parts of your earlier reasoning. It helps me to write it down anyway because formulating thoughts sometimes reveal the related mistake.

[steffengy]

go run generate.go -debug generates the wrong order, as before.

I think the issue is that we output ordered by files, which might not be correct for cross-dependencies. (Atleast that's what I get from adding the fmt.Println(d) mentioned previously)

I'll try to refactor that to order by declaration-order and we'll see what happens

[cznic]

go run generate.go -debug generates the wrong order, as before.

Surprising, but if that means there's a bug in generator.go, it's good to have this piece of information. But you've earlier said that using the debug print the order was okay. Using -debug should include everything in original, preprocessing order. I must be missing something.

I think the issue is that we output ordered by files, which might not be correct for cross-dependencies.

Hmm, now I think you've nailed it. Source files order is not the same as the order the compiler sees after preprocessing b/c #includes in C are quite powerful and not limited to effectively include the whole file or nothing. GNU libc actually uses things like #define need_foo before including things to include very specific parts of other headers.

So it starts to look like we need to record the original order in full and "and" that with the selected-by-regexp property but still output in the same order. That will probably also remove the macros-after-defintions wart.

Do you feel like implementing this particular change or do you want me to have a stab on it?

I'll try to refactor that to order by declaration-order and we'll see what happens

Ah, you already decided on that ;-)

---

Once again, I'm sorry you have to deal with my bad and buggy code. I'm very grateful you are helping in improving it!

[cznic]

BTW: Emitting just everything in incoming order would by my first attempt. If that, by chance, works, I see no real reason why not to keep that solution.The final IR/binary does in no way contain the potentially redundant definitions so it would not hurt anything, I think. (Just a 2c thought about the simplest implementation.)

[steffengy]

Making those changes (not final) fixes it for fcntl.h:

diff --git "a/generator-cadbea0.005.go" "b/generator.go"
index ea52de1..14bdc34 100644
--- "a/generator-cadbea0.005.go"
+++ "b/generator.go"
@@ -943,6 +943,7 @@ func header(nm, mre, dre string) {
    re := regexp.MustCompile(dre)
    dset := map[string]scanner.ErrorList{}
    var fo []string
+   i := 0
    for l := ast; l != nil; l = l.TranslationUnit {
        n := l.ExternalDeclaration
        pos := position(n)
@@ -962,6 +963,10 @@ func header(nm, mre, dre string) {
                    fo = append(fo, f)
                }
            }
+           // make sure we can sort by declaration order later
+           pos.Filename = ""
+           pos.Line = i
+           i += 1
            dset[f] = append(dset[f], &scanner.Error{Pos: pos, Msg: d})
        case 2: // BasicAssemblerStatement ';'  // Case 2
            log.Fatalf("%s: TODO %v", pos, n.Case)
@@ -1024,6 +1003,7 @@ func header(nm, mre, dre string) {

    var a, more []string
    cp := map[string]bool{}
+   aggregatedDecls := scanner.ErrorList{}
    for _, f := range fo {
        if s := extractCopyright(f); s != "" {
            cp[f] = true
@@ -1034,14 +1014,11 @@ func header(nm, mre, dre string) {
 %s
 `, f, s))
        }
-       set := dset[f]
-       set.Sort()
-       for _, v := range set {
-           if *oPos || *oDebug == nm {
-               v.Msg = fmt.Sprintf("%s // %s", v.Msg, v.Pos)
-           }
-           a = append(a, v.Msg)
-       }
+       aggregatedDecls = append(aggregatedDecls, dset[f]...)
+   }
+   aggregatedDecls.Sort()
+   for _, decl := range aggregatedDecls {
+       a = append(a, decl.Msg)
    }
    for _, f := range fo {
        if !cp[f] {

The only issue I see with this is handling the copyright information since we cannot just insert them at file boundaries can we?

While writing this I actually have a potential solution of inserting them at the top of the file and referencing them at bounds which might be best illustrated with an example:

/**
 * [FILE_1] Copyright Super Cool Corporation 
 * ...
*/
/**
  * [FILE_2] Copyright XYZ! DO THAT! DONT DO THAT 
  * ...
**/

// BOUNDARY: BELOW HERE [FILE_1] TILL NEXT BOUNDARY:
int company_1_test();
int company_1_test2();
// BOUNDARY: BELOW HERE [FILE_2] TILL NEXT BOUNDARY:
int company_2_test();
int company_2_test_2();
// BOUNDARY: BELOW HERE [FILE_1]
int company_1_test();
// BOUNDARY: BELOW HERE [FILE_2]
int company_2_test();

Any idea about that?