[RFC] generator.cs refactoring

[chamons]

Our primary code generator, which I affectionately call generator.cs after it's primary source file, is in many ways showing it's age.

Some examples of this:

• It is almost completely in one single large file 8200+ lines long.
• It only has integration tests, and not that many given the complexity.
• Difficult to debug - It is not straight forward to debug why it's failing. Often it is faster to just add debug comments and re-run a given build, which can be painful given the long 10+ second cycle.
• Difficult to trace failures - Often crashes don't show any context on which namespace/type/memeber they were processing, only saying "NRE".
• It is very complex
  • Pop quiz PrintPlatformAttributes takes two argument MemberInfo mi, Type type = null, in which situations is type null, and what do they imply about the context in which we're generating platform arguments.
  • Generate (Type type) way over 1000 lines long.

This is not to say generator.cs is bad - It is amazing. It's power is what allows our team to tackle binding such as massive API surface.

If, in the future we had bandwidth I would suggest the following:

• Write a test script that makes it easy to verify changes to the generator produce identical output
• Review the source code, and determine what complexity can be cleaned up with refactoring and what needs rewritten. Possibly rewrite sections, but never the entire thing, and never at once.
• Check in those changes in gradual PRs, always keeping everything building and identical (unless the old output can be shown to be actually wrong).

I'm sure we could come up with a list of improvements we'd like to make while under the hood. Mine include:

• Either removing the two pass "we compile the assembly and then use reflection to read it" system for a roslyn front end (there are no trivia we'd need to care about to ruin the day unlike my last project), or making it easier in our build to generate the first step in our build and then debug through the generator.
• Add a set of real data structures instead of passing raw types/strings around. Right now, trying to parse "where do we handle inlined protocol members vs members found directly on the type" took too long.
• Add logging/traceability. I've grown fond of telemetry style logging in my personal projects, where you can define "spans" for a given request and then shovel in metadata and in crashes you can see "in processinging type x, in processing member y, in span z, we crashed", but in the end anything is better than what we have today.

[rolfbjarne]

Additional points:

• Move the source code to its own directory instead of being in the src/ directory (easier to see what's generator code and what's bindings)
• More tests, more tests, more tests.
• Extract the "compile the api definitions" out of the generator.
• Make it faster / performance work.

• Difficult to debug - It is not straight forward to debug why it's failing. Often it is faster to just add debug comments and re-run a given build, which can be painful given the long 10+ second cycle.

It should actually be fairly easy to debug: open src/generator.sln, select the configuration that matches the platform you're running the generator for, set breakpoints, run.

That said, it might still be hard to figure out what's going wrong.