Exposing flatbuffer compilation internals

[CasperN]

The maintainers had some conversations and discussions around a significant refactoring of flatc to expose more internals somehow.

However, we're not sure if there are enough use cases to justify such an effort, or even guide its technical direction. So, I'm now crowd sourcing ideas. So... Where does flatbuffers tooling fall short, and can these be addressed by exposing more information that currently exists in flatc? And also how do people feel about the priority of this effort relative to some other large projects such as implementing the ideas in #5875 or #6053

Some use cases:

• U1. Implementing new code generators for new languages in those languages
• U2. Easier front end for projects like flattools and dflats
• U3. Generated validators: example
• U4. Generated fieldmasks (like this but not based on strings)
• U5. Might make an XML to/from flatbuffers tool easier to implement #6033
• U6. Generated custom flatbuffer -> string methods
• U7. Invoke code generators owned by other codebases, e.g. swift-grpc

Some methods

• ~M1: dynamic linking~
  • This does not improve how many languages flatc can be developed in and forces us to deal with cross platform dynamic linking and does not seem to provide enough marginal benefits per cost
• ~M2: scripting languages~
  • We will move forward with M3 since M2 does not support U7 and since M3 does not actually prevent M2, should we choose to pursue it in the future.
• M3: exposing a json/bfbs intermediate representation (We currently have reflection.fbs which may be extended or replaced)

Some positive side effects

• P1: It'll make flatbuffers internals easier to understand (idl_parser pretty organically grown and messy right now).

Some preferences / potential requirements

• R1: ~If we go with M3,~ we should try to integrate with reflection.fbs rather than starting from scratch
• ~R2: If we go with M2, we should reimplement our code generators in the scripting language.~
• R3: We should support be able to specify multiple levels of information stored in reflection.fbs; optionally include attributes, comments, etc. This can be configured to "give enough for reflection" and "give enough for compilation / everything"
• R4: The IR should be JSON compatible, in case a generator is to be implemented in a language without current flatbuffers support

@dbaileychess @mikkelfj @aardappel @krojew @paulovap @adsharma

(I intend to keep this top comment up to date with discussion below)

[github-actions[bot]]

This issue is stale because it has been open 6 months with no activity. Please comment or label not-stale, or this will be closed in 14 days.

[github-actions[bot]]

This issue was automatically closed due to no activity for 6 months plus the 14 day notice period.