search

CesiumGS / cesium

An open-source JavaScript library for world-class 3D globes and maps :earth_americas:
https://cesium.com/cesiumjs/
Apache License 2.0
12.74k stars 3.45k forks source link

Cesium 2 #2524

Open pjcozzi opened 9 years ago

pjcozzi commented 9 years ago

There is not official plans for Cesium 2 yet, but @mramato and I have been discussing potential breaking changes and this is the place to track them.

We've also started to label some existing issues as marked for 2.0. Link

mramato commented 8 years ago
mramato commented 8 years ago
pjcozzi commented 8 years ago
kring commented 8 years ago

Move imagery providers to Core? The terrain providers are there.

I considered doing this when I moved the terrain providers to core, but didn't. My thought process was that surface geometry is useful for things other than rendering a scene. I couldn't really say the same of imagery providers.

pjcozzi commented 8 years ago

Yes, I can see both perspectives. I don't have a strong opinion, but was thinking of things like computer vision, image processing, etc..

hpinkos commented 8 years ago

Some geometry improvements:

pjcozzi commented 8 years ago

Perhaps useful reading:

pjcozzi commented 7 years ago
pjcozzi commented 7 years ago

For a bit more on ES 6 modules, Leaflet 1.1.0 now uses ES 6 modules and Rollup.

http://glmatrix.net/ also moved to modules

pjcozzi commented 7 years ago

Recent podcast on modules (and their history) that is pretty good: https://changelog.com/jsparty/16

pjcozzi commented 6 years ago

The State of JavaScript 2017 has lots of good states on trends for ES 6, TypeScript, build tools, etc.

pjcozzi commented 6 years ago
pjcozzi commented 6 years ago
TJKoury commented 6 years ago

Is there an official branch / repo with progress on converting the current code to es6 modules? I'd be interested in helping out.

ggetz commented 6 years ago

@TJKoury No official branch yet. If you'd like to start one, go ahead and open an incremental PR.

mramato commented 6 years ago

Just to prevent anyone doing a bunch of work that we end up being unable to merge. Please keep us in the loop in terms of exactly what you would like to do. I have a bit of a roadmap laid out in my head and I'd be happy to expand on it if you really are interested in undertaking this. Switching our source code to ES6 is trivial, but there are lots of gotchas the moment you are done step 1 (build, jasmine, module loading, backwards compatibility, node, etc...)

TJKoury commented 6 years ago

Sure thing.

Even 'step 1' as you call it is very problematic; there is a lot of 'code smell' issues in Cesium when removing AMD (like 'return' outside of a function when there is no longer a function as the second input to the 'define' function).

Some other things I want to accomplish are:

  1. Moving things out of ThirdParty folders into node_modules as dependencies
  2. Removing buildModuleUrl entirely and replacing the functionality with Webpack. User should be able to configure code splitting as required.
  3. Removing the gulp build process entirely to be replaced by Webpack

Any input would be helpful.

mramato commented 6 years ago

Even 'step 1' as you call it is very problematic; there is a lot of 'code smell' issues in Cesium when removing AMD (like 'return' outside of a function when there is no longer a function as the second input to the 'define' function).

I'm not really sure what you mean here. The define function at the top goes away (and is replaced by import statements) and the return at the bottom goes away completely. What am I missing? This should be a pretty basic script

Moving things out of ThirdParty folders into node_modules as dependencies

I would like to see this happen, but I think it's a completely separate issue that can either be done before or after a move to ES6, I definitely wouldn't do them at the same time. It will also have it's own hurdles that need to be overcome.

Removing buildModuleUrl entirely and replacing the functionality with Webpack. User should be able to configure code splitting as required. Removing the gulp build process entirely to be replaced by Webpack

We definitely don't want to require devs to use webpack in order to use our modules, and I'm not sure we want to move to webpack as the build system in general. When developing Cesium locally, we don't want to introduce a heavy build step or add lag time when a file is changed (because of all workers needing to be rebuilt for example). Something like SystemJS might be a good solution here, it would mirror our current set up but support ES6 modules. More investigation is needed though. I'm not against using webpack in Cesium itself if the iteration speed is sufficient and friction-less.

Porting the unit tests will also be a big part of this, so don't underestimate the work that might have to happen there (it could be straight forward, but I haven't looked into it at all).

mramato commented 6 years ago

Two more things;

  1. Coverage will also need to be updated to work. I'm not sure if the old and busted jscoverage that Cesium is using can be made to work or if we'll need to switch to karma-coverage. The plan was to do the later at some point, but it only supports the old istanbul API (and not the new nyc api) which has a major scalability bug that prevents the tests from completing. Again, I'm open to ideas here, they just need to be explored. Upgrading coverage could also happen in a separate PR before (But not after)
  2. Whatever build system we move to (webpack or otherwise) should probably happen in it's own PR before the ES6 switchover to keep the review manageable. So moving to webpack can happen before ES6 (but again, we need to evaluate the impact that has on day to day development)
TJKoury commented 6 years ago

This is an example of the first issue. This pattern is used throughout the code base.

The third party module stuff is an interesting problem; most of these are native AMD or have been wrapped, so that the user has to choose between a build tool that can handle both AMD and es6, or add another build step. Might as well address it now.

I should have been more specific about the code-splitting and Webpack; what I meant to say is compiling the modules into one file or requiring them individually at run time should be a deployment option. The current setup doesn't allow for that choice.

I test in production, so no problems there.

mramato commented 6 years ago

This is an example of the first issue. This pattern is used throughout the code base.

This is a coarse grained check so the code can actually load in browsers that don't support required features (otherwise the script would fail to parse and the entire page would break). I wasn't aware that ES6 didn't support early return so I didn't realize this was a problem, but I only think it affects a dozen files, so it should be pretty easy to deal with. I wouldn't consider this a code smell as much as a limitation of ES6 (which is understandable because one of their design goals was static module analysis)

TJKoury commented 6 years ago

No problem, it's just a shift returning from the AMD closure to specifying which exports belong to the module. Of course what goes along with this is deciding what the behavior should be in these modules (throw an error?) as well as the calling code sanely dealing with whatever is returned.

Most of these checks are redundant (like looking for TypedArray support) so I would think they could all be pulled out, deduped, and handled up front. Any objections to that approach?

mramato commented 6 years ago

The issue is that we can't throw an error. The code needs to load cleanly in browsers that don't support TypedArray (like old IE). The code does not need to be functional because Cesium itself won't work if they are not available, but the script itself needs to be able to be parsed by the browser..

TJKoury commented 6 years ago

That sounds reasonable, maybe have a singleton Cesium.ERRORS array that gets populated or something.

pjcozzi commented 6 years ago

Perhaps interesting reading:

pjcozzi commented 6 years ago

Consider coarse-grain CPU-intensive subsets in WebAssembly (with asmjs fallback)

BabylonJS team is checking out assemblyscript, a TypeScript to WebAssembly compiler.

Perhaps long-term interesting to Cesium.

pjcozzi commented 6 years ago

Consider coarse-grain CPU-intensive subsets in WebAssembly (with asmjs fallback)

Sounds like JS/webasm interop may become very fast, making more fine-grained use of webasm possible.

See Mozilla's Making calls to WebAssembly fast and implementing anyref.

pjcozzi commented 6 years ago

Perhaps xeogl's upgrade to ES6 & rollup will have some lessons learned: https://github.com/xeolabs/xeogl/issues/272

TJKoury commented 6 years ago

@pjcozzi Looks good! I have a rollup version working now, just finishing up some rough edges.

thw0rted commented 5 years ago

Has the team thought about moving to semver once this all gets worked out? I saw #2422, and 4 years ago, Cesium was in the "move fast and break things" phase, so I understand the hesitation at that time to increase the major version for every breaking change. The arguments against semver presented in that issue made sense when they were presented, but I would argue that they do not anymore. I work in government and appreciate the historical difficulty in approving a new major version, but ever since Google started putting out a new Chrome every month, there's a little more understanding that maybe version X+1 isn't something that requires all the paperwork to start over from scratch.

Upthread, people are talking about making Cesium more modular -- a great idea! -- but this also means you need to carefully consider the surface area of the public API and what guarantees you're making with them. In the older semver issue, Matt argues that breaking changes are made slowly over time and that API consumers have time to adjust, so publishing them under a minor version increment is fine. The thing is, I consume Cesium via the (official, supported) NPM package, and because every other dependency I use follows semver, I can just npm update to get the latest non-breaking changes... except for Cesium. For that, I have to go pull up the changelog, where nearly every release has a section titled "Breaking Changes" (that title appears 72 times there!).

So: as part of your 2.0 roadmap, I'd like to ask that you a) carefully consider the surface of your public API, and b) make a contract with API consumers to minimize breaking changes between minor versions. The rest of the world has finally started getting actual use out of their major version number, and I'd love to see Cesium follow them.

pjcozzi commented 5 years ago

@thw0rted we're on board with semver; we'll just need to confirm with major customers supporting the project that they will still have an upgrade path with a worse cast, I imagine, of a niche wrapper project with different versioning.

thw0rted commented 5 years ago

Great to hear. I saw a brief mention of Typescript upthread (and I think there's a related issue) -- has there been any further discussion on that front? I've basically moved all my code to TS and I'm really happy with the result. The best part is that with the right transpiler options, your emitted code is basically the exact ES2015 (ES6?) module code you would have written in the first place.

TJKoury commented 5 years ago

I’m not a fan of Typescript, but I get why it is popular. A project as well documented as Cesium could certainly be converted with a little effort.

thw0rted commented 5 years ago

I don't want to derail the thread, but I think that a 2.0 rewrite (because that's what it would be, if you're going from AMD to proper modern modules) would be the last hope to change over, because you'd be reconsidering the public APIs.

One of the biggest problems I foresee if Cesium went to TS is the getter/setter typing issue. Short version: in TS, you have to treat accessors the same as properties. That means that if your setter accepts types A, B, or C, your getter by definition returns all those types. So if you want to be able to set ent.position to a Cartesian3 or a Property<Cartesian3>, even with the understanding that Cesium will internally wrap the Cartesian3 in a ConstantProperty transparently for you, as far as TypeScript is concerned, this means that reading ent.position must give you a "union type" that is either a Cartesian3 or Property and you can't count on it being one or the other until you narrow the type with a guard / check.

Even shorter version:

ent.position = Cartesian3.fromDegrees(3,4);
console.log(ent.position.getValue(viewer.clock.currentTime));

is a compile-time error, because either you declare Entity#position as Cartesian3 | Property, in which case the second line flags (you accessed getValue without checking that it returned a Property), or you declare it as Property, in which case the first line flags (you can't assign a Cartesian3 to a Property).

This is a limitation of the language as written and the TS team has declared it "too complex" to work any other way. So, to port Cesium to TS, you have three choices:

Obviously all are bad, but it would be up to the team which is least-bad.

pjcozzi commented 5 years ago

@thw0rted @TJKoury there is certainty interest in supporting TypeScript well even if that just means officially maintained definition files, but nothing has been deeply explored or decided at this point. Please add more thoughts and resources to this issue and we will review when we have bandwidth to kick off this effort.

pjcozzi commented 5 years ago
pjcozzi commented 5 years ago
mramato commented 4 years ago

ES6 PR is open, see #8224

TJKoury commented 4 years ago

Now that @mramato has a good way forward with es6, is anyone interested in our solution to roll Cesium up into a single-file, so it doesn't need a server?

thw0rted commented 4 years ago

I sorta-kinda do this with Webpack DLLs but still have to distribute assets (textures, meshes, data files, web workers) alongside. Do you have a workaround for all of that?

TJKoury commented 4 years ago

@thw0rted I do, it's the same process I used to create webpack-cesium as a test. This was the original reason I started my own internal es6 version.

A compressed version is on our project at celestrak.com.

markerikson commented 4 years ago

Yeah, I did Webpack DLLs also:

https://blog.isquaredsoftware.com/2017/03/declarative-earth-part-1-cesium-webpack/

On a new React project, I used https://github.com/darwin-education/craco-cesium to enable Cesium usage with Create-React-App (and also their https://resium.darwineducation.com/ React wrapper components).

TJKoury commented 4 years ago

Sounds like people have it covered! Great write-up BTW @markerikson, I’ll have to check out craco-cesium.

markerikson commented 4 years ago

@TJKoury : thanks. I did that post a couple years ago based on a work project. The post talks about ejecting CRA, when today I'd use one of the many CRA override tools (craco, react-app-rewired, customize-cra) to modify the Webpack config without ejecting. craco-cesium just happens to use craco to apply a reasonable bunch of Webpack tweaks without having to further add those in by hand.

Doing a CRA build with craco-cesium appears to result in a smaller final output than my original Webpack DLL does.

ezze commented 4 years ago

It's a big disappointment for me, guys, that migrating to ES modules you still didn't care about semantic versioning. All my projects relying on Cesium are broken now. :-(

thw0rted commented 4 years ago

How were you consuming the code? I was under the impression that the distro still includes built (UMD?) output that shouldn't be any different from before, and the API hasn't changed.

markerikson commented 4 years ago

Bundled apps won't be consuming the UMD file. The bundler will pull in whatever entry point is pointed to in package.json. UMD builds are only for use as script tags, really.

In this case, @ezze hasn't actually given any details on what actual problems occurred when they migrated to 1.63, so it's hard to say what's going on.

thw0rted commented 4 years ago

That's down to how the bundler is configured, which is why I asked ezze how it's being consumed. Cesium 1.64 ships with main: index.js, which exports either Build/cesium/Cesium or esm("Source/Cesium.js") depending on the environment flag. Either way I believe this basically gives you UMD. I haven't used them in a while, but I thought the "U" in UMD stood for Universal, because it should work with CommonJS or as a script tag.

If the bundler understands ES modules, the package file includes module: Source/Cesium.js which means it can also grab the ES modules directly.

ezze commented 4 years ago

We are relying on Cesium as a peer dependency in a set of our private libraries and using it like this:

import Cesium from 'cesium';
const imageryProvider = new Cesium.ImageryProvider(...);

Cesium was declared in externals of Webpack configuration:

{
  externals: {
    cesium: 'Cesium'
  }
}

It doesn't work since 1.63.0 because ES6 module is imported instead of original dist with global Cesium object and there is no default export there. Our code should import Cesium components in another way now:

import { ImageryProvider } from 'cesium';
const imageryProvider = new ImageryProvider(...);

Additionally, some of our libraries rely on Source/Core and Source/Workers of Cesium to build custom workers that are also bundled to Workers directory by automated scripts. They were written as AMD modules while Cesium used them before 1.63.0 and were also prepared with Require.js optimizer. As I can see now in Cesium, built workers are still shipped as built AMD modules in Build directory but source code is migrated to ES6. Therefore my AMD modules were broken too due to imported ES6 modules couldn't be recognized by Require.js. I had to rewrite custom workers in ES6 and build them with Rollup to amd to be able to use again.

Now, I fixed all these issues in my 47 libraries and everything is quite good right now but I spent a day before I realized that something is went wrong with Cesium and fixed all my related stuff (though I had to do another unrelated work).

I still have some issues with removing externals from my Webpack config and building Cesium ES modules directly into my vendor libraries' bundle (to get some benefits such as tree shaking) but, probably, I will report it in a separate issue later if I have no luck with resolving them.

nshen commented 3 years ago

consiter vite , next generation bundler ,it is really fast. https://github.com/nshen/vite-plugin-cesium