Clean up julia.h

[Keno]

Looking through julia.h, I think we should go through an decide exactly what is part of our public ABI and what isn't. At the moment, our ABI for embedding is essentially completely unstable and just using julia.h will bake in various struct offsets into the code making use of it. I think we could probably cut the header down quite a bit and end up in a state where we're mostly ABI compatible across versions, which could help embedders that want to link against different julia versions, particularly as julia becomes more stable.

In particular, I think it would be worth:

1. Moving all type definitions out to a separate julia-types.h header
2. Where we have static inline functions, determine whether having them be static inline is critical for embedders (if so, make a minimal ABI that has the fields being accessed) or not (in which case we can have static inline in the private header and a definition in jlapi.c)
3. Cleaning up any utilities intended for internal use only (there's a few of them)

Tentatively adding to the 1.6 milestone, since I think it would be nice to start the LTS with a cleaner public ABI.

[fingolfin]

I was just about to open my own issue when I saw this, so I am going to tack this on here.

First off: yes, please, to all suggested here. Regarding julia-types.h, I'd love if there was a way to include "julia.h minus julia-types.h". That way, I cannot "accidentally" access struct member -- be it because I was careless and did so directly, or be it because I unwittingly called macros or static inline functions which access struct members.

Also, I'd appreciate if there was explicit documentation regarding whatever ABI stability guarantees Julia is willing to make (or not!) for the Julia C kernel ABI. As in:

1. Is code that was compiled against Julia 1.x.y guaranteed (modulo bugs, mistakes, etc., of course) to be runnable with all later 1.x versions, without need for recompilation? My guess is yes, but I didn't find this explicitly in the documentation (may have been my mistake). So even if the answer is "No", having this stated explicitly would be helpful. (But please, please, let this be "yes", otherwise dealing with this becomes even harder).
2. Is there any ABI promise when going from 1.x to 1.(x+1)? I just learned that JL_GC_PUSH1 (and variants) change from Julia 1.3 to 1.4, thus CxxWrap.jl compiled against Julia 1.4 is broken in Julia 1.3). Now, my guess is, there is no such promise -- while of course I'd love one, I totally understand why one may not want to make any promises. But then it would still be nice to have this explicitly stated.
3. By doing what @Keno says, there is a great opportunity to start defining a useful subset of the API/ABI for which one may be willing and able to make bigger promises; as in: I totally get that you wouldn't want to promise anything about all content of julia.h, that'd tie you down far too much. But perhaps there is a sizeable subset where this is no big issue, and where such a promise would be possible.

[barche]

As indicated by @fingolfin this would be very useful for CxxWrap and all packages that depend on it. To support embedding from MSVC, issue #34201 also needs attention.

[vtjnash]

FWIW, https://github.com/JuliaLang/julia/pull/36588 has now made everything effectively private, except a few very specific things (which is pretty much just jl_init IIUC)

[vtjnash]

We realized in discussions of #36588 that it hasn't yet changed anything. So we need to re-export things before making those changes, but it is not yet impactful for embedding in v1.6.

[fingolfin]

@vtjnash I don't understand how PR #36588 could be related to this issue: it doesn't touch julia.h nor add any new header one could use instead, so while it seems to be doing something for improved ABI support, whatever it does seems to be orthogonal to what is discussed in this issue?