missing --js is disruptive

[mikkelfj]

In the 2.0 release the --js option was removed. This is rather disruptive for live production sites requiring the need to use multiple versions of the flatc tool for different languages.

After some digging it becomes apparent that you can use the tsc tool to convert --ts output to javascript, but this is not straight forward because the output is not the same format, and worse, the tsc toolchain is invasive and not always easily installable.

Apart from short term breakage, it is problematic that it requires heavy tooling in addition to flatc.

The argument that there are many flavors of javascript and modules doesn't really hold. It would be possible to simply support Ecmascript 6 which would be acceptable to all modern browsers and presumably node.js.

Also, the now lost bundled single file output is fairly signficant for web site inclusion without extra tooling, but I could see a --js and --output-file option combined to achieve that, if multi file output like --ts is preferred.

@aardappel @krojew @bjornharrtell

[krojew]

I disagree on several points:

• tsc being disruptive can be argued is a very niche case, given it's installed like every other JS dependency using npm. If you're already using JS and npm, you can install tsc. It's as disruptive as any other thing you install that way, which leads me to believe we're talking about some very specific situation, rather than the norm.
• The argument for different flavors most definitely hold - you yourself arbitrarily chose one, which suits your needs, but might not be the proper one for other projects. That's why we leave that decision to the end user, rather than forcing anything. And believe me - incorporating tsc in the pipeline is much less work that having to deal with incompatible JS style.
• Can you elaborate on not being the same format? TS support has been reworked to a modern module-based approach in 2.0 and it should be the same after transpilation.

[mikkelfj]

Not everyone is using node.js, let alone npm. On web site builds it now requires node.js and npm to be installed. FlatBuffer from web to server is a very real thing.

For developers generating code from schema and check it in to a repo, these all require node, npm and tsc to be installed in order to generate code.

Regarding the same format, I cannot elaborate and I may have been mistaken. A coworker used the swc tool instead of tsc due to installation issues and could not get this to work (trivially, and reverted to flatc 1.12). As I recall the FlatBuffers doc mentions that the format might be different.

You cannot argue that ECMA Script 6 is an abritrary choice. A few years back it would be different, yes.

As nice as TS is, it lends itself to the all too heavy tooling needed to build modern web sites (looking at webpack). I think it is nice, and relevant, to have TS output. But omitting JS is not good.

[krojew]

If you can't install from npm, you can install tsc via yarn or system packages or in any other way. Granted, it is one more dependency, but given npm is THE standard right now, it's a safe assumption it will be available. Exceptions are just that - exceptions.

Regarding JS flavor - ES6 is the same arbitrary choice as any other. Is you want to argue we should target specific JS flavor, then we can also argue to target specific JS environment. Also, modules were only one of the arguments of going with TS.

In the end, I still think we're talking about a niche case, rather than the norm. This will eventually lead to the exact same discussion python cryptography recently had after adding Rust dependency which doesn't support all platforms. The outcome is - support of niche cases falls of niche case users.

[mikkelfj]

With that line of argument, you could just use C++ instead of typescript since typescript is a niche case.

[krojew]

Reducing to absurd doesn't really apply here.

[bjornharrtell]

While I can agree that removing --js is somewhat disruptive I also feel it's defendable and I agree with all argumentation from @krojew. Some additional thoughts:

• We could keep/reintroduce --js but it would be a thin and awkward wrapper around tsc (+ a bundler for single file target) anyway. It was discussed when doing the modernization of flatbuffers js story and all involved decided against.
• Regardless if the source is TypeScript or JavaScript the user needs to make a decision about what the target platform is and instruct the compiler/transpiler of that target. There is no good default here. It might have been ES5 a while back when baseline IE11 support was "common ground" but it is no longer. So there is no reasonable/usable "just give me JavaScript in a single bundle" output anymore, in my opinion.
• TypeScript vs JavaScript is a non issue. As per previous bullet point you would still need to involve transpilation using babel or similar and that is no less complex than using TypeScript, in my opinion.

[bjornharrtell]

That said I wouldn't oppose a convenience --js single file build switch but it would have to depend on npm, tsc and rollup (or similar) with some sensible defaults to produce the output, I don't see any way around that.

[mikkelfj]

@bjornharrtell I agree a tsc wrapper would be pointless. I do not accept the argument that you always need transpilation. If that were the case, I could accept ts as the only output. But the usecase is to dump a .js file in a simple webpage and send and receiver flatbuffers from an API endpoint. The previous js did that just fine.

[krojew]

"Dumping" a plain .js file into a page is a concept which has been abandoned in practice years ago. While there surely are legacy projects which still do that, we would like to support a more contemporary approach of using modules. Nevertheless, you still can transpile TS and create one giant blob of js, if you wish. If, for any reason, you don't want to do that, you can stick with a previous version, which is binary compatible with 2.0.

[mikkelfj]

It's kind of opening a can of worms, but there is also a host of security implications in forcing the use of npm.

[bjornharrtell]

Including/reintroducing both a legacy pure ES5 or ES6 level JavaScript implementation and code gen in flatbuffers, which essentially is the only way to get what you wish @mikkelfj would require alot of effort and introduce a very heavy maintenance burden on the project.

[krojew]

It's kind of opening a can of worms, but there is also a host of security implications in forcing the use of npm.

Then use alternatives.

[bjornharrtell]

Regardless, we do lack a single entry point to create single file bundles for gode gen which is kind of related to this. Made an issue for that at https://github.com/google/flatbuffers/issues/6673.

[aardappel]

As a programmer who likes to keep things incredibly simple myself (and a disliker of dependencies / extra tools), I can really understand that that just being able to get a single .js out of flatc and having a working site was really nice to have.

That said, I also understand that is not the way most people work anymore, and being able to use tsc as your universal transpiler for whatever flavor makes a lot of sense.

The alternative, to maintain a TS and ES6 generator separately, or a combined one with lots of conditionals, is too high a price to pay for the convenience of having this older path.

[mikkelfj]

I'm not going to beat a dead horse, but I would have been more understanding if the --js didn't exist to begin with and if I wasn't already depending on the --js option for live products.

[finnhvman]

I'm bumping into this as well. Is there a migration guide from 1.12.0 JS to 2.0.0 TS?

I've inherited a project built with JS and it seems I have to update all the references when using TS

[AndrewJDR]

I have to agree with @mikkelfj that this was a disruptive change. Implementations of javascript exist outside of the node/typescript ecosystem, including a host of lightweight/embedded machine implementations. Flatbuffers, being a library/file format that is all about packing and reading data in performant way is a very natural pairing for many of these embedded and minimalist contexts.

I could get behind a "npm/node is the standard, everything else is niche" attitude if we weren't talking about a performance oriented data serialization library, but we're talking about a performance oriented data serialization library.

[krojew]

Are you running flatc on your target embedded devices, or in some kind of CI workflow?

[bjornharrtell]

I don't understand your argument @AndrewJDR. You get your choice of plain JavaScript after transpiling (and possibly bundling) to your target platform.

[AndrewJDR]

@krojew I don't have an active project using the --js switch, but I've used it in the past on lightweight projects that didn't use npm, typescript, or bundlers, and would have liked to know that I could do so again in the future.

@bjornharrtell I'm saying that installing a package manager and learning about bundling will never be wanted/needed by the types of developers I described (non-node / plain js developers), so in exchange for a burden being lifted from this project's developers by not having to maintain a js output codepath, a new burden has been created for all current and future developers of the kind I mentioned. The aggregate cost of that new burden is not something we can easily quantify, as it's unlikely that every developer affected by it will report to this github issue.

Another unfortunate bit orthogonal to this is that you can't just download tsc without also pulling in npm or its ilk. If there were an official tsc executable download available for every platform, it would make things like this a lot easier. But that's a different story :-/

[krojew]

I don't have an active project using the --js switch, but I've used it in the past on lightweight projects that didn't use npm, typescript, or bundlers, and would have liked to know that I could do so again in the future.

In such cases, you can still use the 1.x version. In 2.x we are targeting more contemporary workflows.

[AndrewJDR]

@krojew Sure, but miss out on any other improvements/fixes moving forward. So that option will become less and less attractive as more time passes.

[krojew]

That is true, but this is the price of progress. At some point technology needs to move forward.

[AndrewJDR]

Eh, this is just too hand-wavy to me, and feels like it ignores the kinds of users flatbuffers is likely to attract. flatbuffers is a performance centric library, and perormance-centric developers disproportionately concern themselves with minimizing both build time and runtime deps because they understand that all deps have a cost to a project. So it stands to reason that the "flatbuffer users" vs. "developers that care about minimizing dependencies" venn diagram is going to have a fairly large middle section, rather than a thin (i.e. niche as you've suggested). Whereas devs that don't care about things like performance and dependency cost will be more likely to use json parsers everywhere and eschew things like flatbuffers altogether.

[krojew]

Build performance is orthogonal to runtime performance. I'd argue one sacrifices the other in most cases, but that's not the point here. FB targets the best runtime performance. Let's not make baseless assumptions.

[AndrewJDR]

Not really speaking about build speeds, I'm referring to the toll that learning and maintaining additional build time dependencies can take (in aggregate) on developers and cause them to lose precious time they could be spending on building a really performant runtime. Performance centric devs, particularly on smaller projects, think about those kinds of tradeoffs all the time, and those are the kinds of devs that a project with the goals of flatbuffers would reasonably tend to attract.

[krojew]

If someone wants to use plain non-modular js and doesn't want to use external tools, then FB 1.x is the way to go. Maintaining niche cases is a heavy burden on this project, hence the focus on more generic ones.

[AndrewJDR]

Yeah the crux of the argument is that it's not niche for a project like this. But that horse is dead i think.

[krojew]

Nothing prevents you from creating and maintaining a generator for such particular case. We simply don't have the resources to fit every need, so we decided to support what is a de facto standard in the industry now.

[bjornharrtell]

I have a hard time accepting the argument that performance centric developers to not want to learn or use a compiler/transpiler or that it is so difficult or heavy that it significantly reduces the ability to produce performance oriented code. Mabye we should write everying in assembly again eh?

Personally I'm convinced that old school non-modular JavaScript is dying fast and for good reasons. Taking a quick look into gaming related JS resources modular JS or TS does not seem exotic in any way, for example three.js has been modular JS for many years and PixiJS is TypeScript.

I would rather say FlatBuffers is lagging behind because what would make more sense to call for in a performance/gaming context is native WebAssembly support.

The reality of what you are saying is throw away the work me and @krojew have been through to modernize the FlatBuffers JavaScript support which I started on almost a year ago with https://github.com/google/flatbuffers/issues/6094, because there is no way to support both paths, and that's not very helpful or appreciative feedback at this point.

But perhaps I know nothing because I'm not a game developer.

[aardappel]

@AndrewJDR I very much empathize with developers that want to keep their builds and code bases simple, I am most definitely one of them. Also agree that generally if the developer of a library can do something that makes things simpler for 1000s of users, they generally should.

But FlatBuffers doesn't have a large team dedicated to JS/TS development, it is largely volunteer driven, so our options for keeping development of JS & TS going as two forks (or shared with a ton of conditionals) are not great, it would likely slow forward progress, or we'd be in the situation where some features would be in one and then take forever to show up in the other. So making everything go thru TS is what we need to do to keep things going forward.

That, and if more JS/TS contributors show up to work on it we can consider further options, but in recent times its been mostly @krojew and @bjornharrtell doing most of it.

[AndrewJDR]

Hi @aardappel thanks for chiming in. One thought I had -- is there another place where possibly disruptive deprecations can be announced? Maybe the mailing list? If the visibility of something like this is raised, especially if it's prior to it going into effect, I think it can in turn increase the chances of additional maintainers or perhaps even creative ideas coming to the surface. At minimum, maybe some interesting perspectives would be shared so everyone (project devs and user devs alike) knows better what we're about to be getting into.

I should mention that the typescript contributions are greatly appreciated, thanks @krojew and @bjornharrtell

[aardappel]

I don't think we have a good way to reach FlatBuffers users. I doubt the mailing list would suffice. It is also hard to know ahead of time how many people will be affected, most users are very quiet :)

This JS/TS transition has been discussed for close to a year on various issues/PRs, really the only way to stay intimately up to date with what is happening in FlatBuffers land is to watch this repo (or some other method of checking on PRs).

[reneroth]

The documentation should be updated at least :) https://google.github.io/flatbuffers/flatbuffers_guide_using_schema_compiler.html