Aligned to Revisiting Spacetrack Report #3

So @ezze has helped me get smart enough to be dangerous! I still need to make more SGP4 tests for this PR, but I want to have this conversation before I invest a lot of time into it.

I started reading through python-sgp4 and realized that while satellite.js is based on python-sgp4, there are some simplifications that were made that reduced the number of options passed. One that caught my attention was the opsmode in satrec initialization. It was set to default to the improved SGP4 code instead of the legacy code. That seems like a great idea, but it has one huge problem, it prevents us from using Revisiting Spacetrack Report (RSR) #3's verification data to validate our library actually functions the same as the original. Our test for sgp4 was useful in verifying that none of the changes (like the millisecond one I was playing with) to the code base affect the results, but it does nothing to validate the code base actually matches the original its based on.

So I changed the opsmode, but was still getting different results than RSR #3. Next thing I found was that the constants.js file removed the original call to getgravityconst and defaulted to the modern WGS-84 constants. Again, this seems like a good idea - use the newest values, but it A) prevents verification against RSR #3's dataset and B) makes the assumption that USSF is publishing TLEs based on WGS-84 instead of maintaining WGS-72 to avoid having to upgrade all of their systems to use WGS-84 (I haven't seen anything public clarifying which route they went).

I see two options:

1) Accept that it isn't 100% accurate, but the margin of error is pretty low and the goal of this repo is to maintain the status quo. 2) I can start making pull requests to make the code more closely match python-sgp4.

My concern is that without the 2020 OMM changes Vallado made to support Alpha-5 catalog objects, this library will become obsolete by the end of the year for anyone using objects created by Space Fence which are slated to use Alpha-5 numbering to the best of my knowledge.

The middle ground between those two options is that I can fork the project and start doing my rewrite to match David Vallado's most recent C++ code for SGP4. I will focus on making it match and then it might need a smaller wrapper to apply default options for backwards compatibility with satellite.js...

@ezze and @davidcalhoun let me know what you guys think (or anyone who wants to chime in).

@thkruz Sorry, I can't answer in details at the moment but hope that I will do it in a couple of days.

I ended up doing the "middle ground" I mentioned above. https://github.com/thkruz/sgp4-js. I needed to better understand the code for my own purposes so it just made sense to rewrite it. Ended up finding a bunch of useful functions that were left out of satellite.js.

There are three files in my src directory:

sgp4.js is a rewrite of the 2020 version of the c++ code for sgp4
sgp4-utils is all of the functions that were added in satellite.js but not in the c++ version
satellite.js is a wrapper that maps all of the satellite-js functions to the corresponding functions in sgp4.js and sgp4-util.js

I used a quick webpack script to make versions in the dist folder that add them to the window.

My goal is to fill in any gaps I might have made (I know I ditched typescript temporarily) and then start adding more functions to the util folder and reworking the c++ code to optimize it for javascript.

Happy to do some PRs to incorporate any of the testing I add (should be very easy), but I am going to split my code off to start adding features and changing things rather than making major PRs on satellite.js.

@thkruz Thanks for the update.

I think you've made the right decision. Probably, it's worth not forking satellite.js but start a new independent project (if you feel a strength and motivation to maintain it) and (maybe) start it from a scratch. As I already said in another thread, I know just a little bit more than nothing in orbital mechanics and it's very difficult for me to realize what happens behind the scene. But reading your first message from the issue (and knowing you as one of the most active users, supporters and contributors of satellite.js, and remembering your keeptrack.space) I believe and 100% sure that all these things make sense. @shashwatak doesn't participate in the project anymore, @tikhonovits seems busy too and I'm unable to do it for the reasons mentioned above. In other words, it's obvious that satellite.js becomes obsolete and a better actual alternatives are required. So if you have some luck to provide a library to replace satellite.js by (even with a new API) I will be happy to support it as I can and migrate to it. We can deprecate satellite.js then and recommend other users to switch to the new library.

If it's not very difficult for you, could you please clarify few things just for me (like for an idiot that doesn't understand anything in it, haha :) ):

Are your updating the library still relying on Python version of sgp4? You mentioned some C++ code but I can't find it.
I faced with the problem that I can't verify that changes I made to satellite.js don't break anything. That's why tests are required. Initially I wrote some tests relying on test data provided by @shashwatak but later and didn't know how to extend them based on my poor knowledges. How do you plan to provide (or already provide) test data for your tests?

Another thought I had: having C++ codebase it's possible to create a binding library for Node.js. It would simplify the library maintenance but the downside is that, unfortunatelly, this approach doesn't allow to use it in the browser.

As a final note, these are my little recommendations that might (or might not) be useful for a better library usage experience:

Port the whole source code to TypeScript (if it's possible and there are no significant obstacles). Thanks to @kylegmaxwell and @iamthechad satellite.js already has TypeScript definitions but having source code in TypeScript can simplify library maintenance and stability in the future.
Don't use airbnb code style (at least, overwrite some rules). I had very bad experience trying to resolve its brackets hell in satellite.js.
Separate additional transformation functions introduced by @shashwatak from ported sgp4 functions (as you already did, I see). Probably it's worth moving them to a separate npm package depending on sgp4.js.
Choose a better npm package name without .js suffix. Unfortunatelly sgp4 is busy and seems unmaintained. Probably we should ask @joshuaferrara whether he is ready to share the name for a new library.

Thank you for the big impact you made for satellite.js! If you need some help from me don't hesitate to ask.

@ezze I see you are leading me down another path of learning I have avoided haha. It took me 5 years to come around to learning ES6+ and now you have me sitting here working on TypeScript.

I definitely feel the motivation to maintain the library. My only hang up was not wanting to appear snobbish in going my own route. Orbital mechanics is a huge part of my full-time job. JavaScript is only a hobby of mine. So while I can spot gravity constant errors, I will often overlook a much simpler way of writing the code. The biggest support I would need is just pointing out my mistakes - I enjoy learning better coding and fixing your mistakes is how you learn.

Very happy to explain!

I used the C++ code that is included in the python library. Much like you suggested, they allow users who can support it to run the native code. Unfortunately my entire motivation is to use this in a browser, but since understanding the changes in the code helps me at my full-time job, I have no qualms in manually updating it as changes are made.

The wonderful team who published the C++ code also published all of their test results here. The C++ code is at the bottom and their test results are in a table. I have been manually copying the state vector (x,y,z,xdot,ydot,zdot) data from the table into the json file you made for testing. At some point I'd love to have a 1 for 1 copy of their test results to validate against, but for now I focused on having the two extreme cases and safely assuming if it gets those right then the middle is probably fine (they were mainly establishing that at the extremes it maintains its accuracy.

As I look to find possible improvements to the accuracy of the code at longer propagation times -- probably a good place to explain for anyone else reading that doesn't know much about SGP4 -- the amount of variables that go into calculating the exact position of a satellite is insane. A mountain range will pull a satellite more than the ocean (more mass). If you wanted 100% accuracy you would need to have different gravity values for every inch of the earth....and in 1965 that wasn't happening (not even today lol). Instead we made generalizations like "the earth is fatter at the equators (more mass)" and we use those generalizations to model the satellites position. Every generalization makes it less accurate and those inaccuracies build on themselves over time. Most of the changes to SGP4 in the last 20 years have been to account for some obscure scenario we encountered where it got it really wrong. Rocket bodies that have an apogee towards the moon but a perigee really close to the earth cause two different extreme conditionals in the code to both get it wrong. So we add a "unless its one of those weird satellites" in the code and apply a different logic.

So how do I build new test cases? I maintain a copy of the catalog every day. So one test case could be taking a satellite TLE from Jday 100 and then propagating it to Jday 200. Then take the TLE from Jday 200 and propagate it to Jday 200. Compare the error in the positions. Now we start fiddling with some additional conditionals "after jday 50 then..." and see if we can generate a propagator that gets it closer --- then test it against 1000s of satellites. Luckily I have been collecting the catalog for about a year now, so plenty of test data.

Second method involves NASAs satellites that are tracked with lasers. They have some that we KNOW...EXACTLY...where it was. So again, we start with a TLE from today and see how close it gets to the right answer. Go back to a TLE from 100 days ago and see how close it gets to the right answer. Fiddle around and see if we can get a closer answer. You would have to sacrifice computational power, but I think I focus on conditionals that only apply to old TLEs then it won't matter 98% of the time.

Beyond accuracy the much bigger challenge is to increase performance on the browser. I had developed a gpu powered sgp4 script (nightmare to make) and it could do 10000 propagations like nothing....BUT...there was a 5-6ms delay in reading the answer from the GPU which made it worthless for my purposes. My plan is to use a webworker thread pool to simply divide the problem of propagating 17000 satellites into 4-8 pieces. Along the lines of your suggestion to pull the non-cannon SGP4 utilities out of the code I will also be making the multi-threading an extension so you only have to download what you want.

Love the suggestions! My experience with TypeScript tonight has been "o that is neat"..."this is so frustrating to figure out, why would anyone use this"..."yea this would fix so many problems in the code". So I am committing to using it and not looking back.

Airbnb was one of the first things I got rid of. I am sure it has a purpose and if people love it then they can use it - I found it extremely frustrating.

I was leaning towards sgp4js even before you mentioned sgp4 (funny how I didn't even consider the shorter name). While I agree the period in the name is a bad idea, my inclination was to add the js since there are tons of libraries porting sgp4, but only a small few in js. Ultimately the code is more important than the name at this stage of the game. I will get working on the typescript conversion while we both ponder the name issue lol.

I will be sure to reach back for feedback as I go. Only a few questions up front:

1) AMD is outdated, no? I'd prefer to drop it unless you tell me its commonly used. 2) Is there a reason you picked rollup vs webpack? I am inexperienced enough at both to not know the pros/cons. I have webpack on there at the moment because it was the first thing I googled and it mentions typescript support.

I appreciate it and thank you for pulling me into the world of open source. While I have always been happy to answer questions, I don't think I would have ever started pushing code if you didn't explain the ES6+ piece to me.

Just my two cents here! I read through some of your comments, will try to come back later and read the rest!

Note that I've also been scraping Celetrak's TLEs and have an archive going back to December 20, 2019. Sounds like you have a similar dataset, but let me know if you're interested and I can provide a download link!

It's generally true that AMD is old and outdated, but IMO this is part of a larger discussion whether to support an older style UMD export. UMD is a wrapper format that supports AMD, CommonJS, as well as the old browser global (where the entrypoint is something like window.foo). I believe it's still useful to have a UMD export in addition to a modern ES Modules (ESM) export of the library, then export both at build time - here's a simple example of one of my projects where I'm using Rollup to export both UMD (for legacy support) and ESM. In general ESM is way more desirable these days because it can be properly treeshaken (removal of unused code) by dependent projects.
For a few years Rollup has been more desireable than Webpack for library code, primarily because it supports ESM exports, which can be treeshaken. Webpack itself knows how to consume ESM code and treeshake it, but it still can't export ESM code. Eventually I'd love to move over to using all-Webpack for simplicity myself, but for now I'm still using Webpack for apps and Rollup for libraries.

Let me know if any of this is vague or needs clarification! Kind of wrote this in a hurry.

@davidcalhoun is right and I can't answer better. Just add my three kopeks to David's two cents: as the result, usually a size of a bundle created by Rollup is less than Webpack's one.

I think it's easy enough to provide UMD library build with Rollup just like David has done for his project...or just like it's done in satellite.js. If you already started with Webpack and don't want to switch to Rollup it also can do the job easily. Another point to protect AMD from me: I work with Cesium for years, it has been written with Require.js until 1.63. Some people who use Cesium functionality directly may be too intertial to migrate to ES6+ quickly so it's a good idea to keep AMD support in order to provide an ability to use the library in legacy projects.

If you want to stick to TypeScript (by the way, I don't force you to choose it, Flow is another option for you, haha...although TypeScript seems to be a first choice for most developers nowadays) you probably may want to use neither Webpack nor Rollup for such a simple project which doesn't rely on a bunch of external dependencies. TypeScript has its own reliable compiler already with a set of required options. To be honest, I was also too lazy to learn TypeScript until recently but it became so popular and "annoying" so I gave it a try, and it overcame my expectations. It may seem too complicated at first but it's worth it. You can take a glance at my simple project in which I managed to use TypeScript tools for compilation, watching and testing. You can specify multiple targets for your library there as well as you do it with Webpack or Rollup.

@davidcalhoun Perfect explanation - thank you. Took me a little bit, but I got umd support added with webpack following @ezze's link. I will have to do some reading tomorrow to understand the treeshake concept.

Totally in love with TypeScript. It forced me to resolve some of the silly parts in the code.

To @ezze's earlier point about a better name, I settled on ootk (Orbital Object ToolKit) and plan to make it a monorepo with various modules (SGP4 and Transforms) to start. Should let you pull the core features you need without getting bogged down in all the extra stuff I want to make available (my version of suncalc for example). My plan is to have everything branch off the ootk object, but you can import any of the branches individually if you don't want the whole thing.

I have off from work tomorrow and should be able to get everything published and proper documentation uploaded. Thank you both again for the support on all of this.

@thkruz If you want to make a monorepo then probably Lerna is your friend. But keep in mind that it works well only for centralized git workflow. I had some issues and patches before making it to work with gitflow. Here is my outdated answer on it but I hope that you can get an idea in general if you decide to use it.

shashwatak / satellite-js

Aligned to Revisiting Spacetrack Report #3 #85