brandon-rhodes / python-sgp4

Python version of the SGP4 satellite position library
MIT License
370 stars 87 forks source link

Support OMM file format #65

Closed benjaminglass1 closed 3 years ago

benjaminglass1 commented 4 years ago

As documented at Celestrak's website, an XML/JSON based format called OMM (spec: https://public.ccsds.org/Pubs/502x0b2c1.pdf) is being pushed as a more modern replacement for the TLE format. Since the parameters are still reported in terms of sgp4 elements, and given the dominance of CelesTrak as a source of orbit information, it seems reasonable imo to add support to this project for the OMM format.

I'd be happy to open a PR for this with some guidance on how you'd like this structured. Would it be more reasonable to:

  1. Add this functionality into io.py?
  2. Create a new submodule, omm.py or something similar?
  3. Something else?

This is my first time trying to contribute to an open-source project, so apologies if there's something I'm missing.

ckuethe commented 4 years ago

Go ahead and just start coding something and make a PR. You'll probably get more useful feedback with actual code that we can talk about. If it was me, I'd start by prototyping it with just the JSON format because I have an unreasonable loathing of XML, and because it's way easier to ingest JSON objects into python than XML. I kind of accidentally independently invented a crappy version of this because I wanted to store TLEs in a MongoDB instance and breaking out all the parameters into fields in a document made it very easy to work with. So thanks for highlighting that there is a standard that I can develop against.

Part of me really wants to say "XML should f*ck off and die in a hard drive shredder". I was recently developing some monitoring for my home photovoltaic system that has an API that is all JSON... except for one stupid XML endpoint, and there's no good reason for it. The code to handle that one endpoint is way more complicated than all the rest of the endpoint handlers. xmltodict can be one useful tool, if my rant hasn't discouraged you from fully implementing OMM including XML.

Parsing the JSON version is literally a one-liner, and once you know the OMM JSON reader works you can extend it to the other formats and cross-validate. It looks like Satrec.twoline2rv(line1, line2) is the way to create a satellite from a TLE, so maybe Satrec.omm(format=Satrec.JSON, data=buf) might be a reasonable interface. Or maybe ommjson, ommcsv, ommxml, so that it it absolutely clear to users what the input format is supposed to be.

One thing to consider is what happens when I do something like the example below and feed in a JSON object that contains multiple sets of parameters... food for thought :)

>>> import requests
>>> from pprint import pprint
>>> data = requests.get('https://celestrak.com/NORAD/elements/gp.php?NAME=MICROSAT-R&FORMAT=JSON-PRETTY').json()
>>> pprint(data)
[{'ARG_OF_PERICENTER': 299.7926,
  'BSTAR': 0.00014326,
  'CLASSIFICATION_TYPE': 'U',
  'ECCENTRICITY': 0.0478768,
  'ELEMENT_SET_NO': 999,
  'EPHEMERIS_TYPE': 0,
  'EPOCH': '2020-06-28T16:19:09.412320',
  'INCLINATION': 96.0469,
  'MEAN_ANOMALY': 55.6493,
  'MEAN_MOTION': 14.85824771,
  'MEAN_MOTION_DDOT': 1.4741e-06,
  'MEAN_MOTION_DOT': 0.00011768,
  'NORAD_CAT_ID': 44134,
  'OBJECT_ID': '2019-006V',
  'OBJECT_NAME': 'MICROSAT-R DEB',
  'RA_OF_ASC_NODE': 338.5026,
  'REV_AT_EPOCH': 6671},
 {'ARG_OF_PERICENTER': 256.5117,
  'BSTAR': 0.00091395,
  'CLASSIFICATION_TYPE': 'U',
  'ECCENTRICITY': 0.012578,
  'ELEMENT_SET_NO': 999,
  'EPHEMERIS_TYPE': 0,
  'EPOCH': '2020-06-28T20:46:26.885280',
  'INCLINATION': 95.6997,
  'MEAN_ANOMALY': 102.2117,
  'MEAN_MOTION': 15.85547186,
  'MEAN_MOTION_DDOT': 6.2965e-05,
  'MEAN_MOTION_DOT': 0.00417568,
  'NORAD_CAT_ID': 44147,
  'OBJECT_ID': '2019-006AJ',
  'OBJECT_NAME': 'MICROSAT-R DEB',
  'RA_OF_ASC_NODE': 328.5679,
  'REV_AT_EPOCH': 6717},
 {'ARG_OF_PERICENTER': 317.662,
  'BSTAR': 0.0019312,
  'CLASSIFICATION_TYPE': 'U',
  'ECCENTRICITY': 0.0132193,
  'ELEMENT_SET_NO': 999,
  'EPHEMERIS_TYPE': 0,
  'EPOCH': '2020-06-28T20:05:33.531360',
  'INCLINATION': 95.2336,
  'MEAN_ANOMALY': 41.4508,
  'MEAN_MOTION': 15.81383415,
  'MEAN_MOTION_DDOT': 0.00019069,
  'MEAN_MOTION_DOT': 0.00728927,
  'NORAD_CAT_ID': 44160,
  'OBJECT_ID': '2019-006AX',
  'OBJECT_NAME': 'MICROSAT-R DEB',
  'RA_OF_ASC_NODE': 285.6139,
  'REV_AT_EPOCH': 6534},
 {'ARG_OF_PERICENTER': 19.2135,
  'BSTAR': 0.0010387,
  'CLASSIFICATION_TYPE': 'U',
  'ECCENTRICITY': 0.0627756,
  'ELEMENT_SET_NO': 999,
  'EPHEMERIS_TYPE': 0,
  'EPOCH': '2020-06-28T03:01:34.669344',
  'INCLINATION': 94.6349,
  'MEAN_ANOMALY': 343.1631,
  'MEAN_MOTION': 14.53240269,
  'MEAN_MOTION_DDOT': 1.5337e-06,
  'MEAN_MOTION_DOT': 0.00085286,
  'NORAD_CAT_ID': 44381,
  'OBJECT_ID': '2019-006DC',
  'OBJECT_NAME': 'MICROSAT-R DEB',
  'RA_OF_ASC_NODE': 240.1106,
  'REV_AT_EPOCH': 5819},
 {'ARG_OF_PERICENTER': 178.9618,
  'BSTAR': 0.00098388,
  'CLASSIFICATION_TYPE': 'U',
  'ECCENTRICITY': 0.0297548,
  'ELEMENT_SET_NO': 999,
  'EPHEMERIS_TYPE': 0,
  'EPOCH': '2020-06-28T05:46:39.979200',
  'INCLINATION': 94.687,
  'MEAN_ANOMALY': 181.2319,
  'MEAN_MOTION': 15.34885226,
  'MEAN_MOTION_DDOT': 6.0216e-06,
  'MEAN_MOTION_DOT': 0.00155296,
  'NORAD_CAT_ID': 44382,
  'OBJECT_ID': '2019-006DD',
  'OBJECT_NAME': 'MICROSAT-R DEB',
  'RA_OF_ASC_NODE': 274.1937,
  'REV_AT_EPOCH': 6156},
 {'ARG_OF_PERICENTER': 99.511,
  'BSTAR': 0.00055609,
  'CLASSIFICATION_TYPE': 'U',
  'ECCENTRICITY': 0.0818287,
  'ELEMENT_SET_NO': 999,
  'EPHEMERIS_TYPE': 0,
  'EPOCH': '2020-06-28T20:09:59.593248',
  'INCLINATION': 96.1852,
  'MEAN_ANOMALY': 269.9417,
  'MEAN_MOTION': 14.06299142,
  'MEAN_MOTION_DDOT': 7.4286e-07,
  'MEAN_MOTION_DOT': 0.00033161,
  'NORAD_CAT_ID': 44383,
  'OBJECT_ID': '2019-006DE',
  'OBJECT_NAME': 'MICROSAT-R DEB',
  'RA_OF_ASC_NODE': 310.1751,
  'REV_AT_EPOCH': 5715},
 {'ARG_OF_PERICENTER': 153.7762,
  'BSTAR': 0.0034551,
  'CLASSIFICATION_TYPE': 'U',
  'ECCENTRICITY': 0.0738757,
  'ELEMENT_SET_NO': 999,
  'EPHEMERIS_TYPE': 0,
  'EPOCH': '2020-06-28T20:15:39.529728',
  'INCLINATION': 95.8539,
  'MEAN_ANOMALY': 210.2896,
  'MEAN_MOTION': 14.27265072,
  'MEAN_MOTION_DDOT': 9.4047e-06,
  'MEAN_MOTION_DOT': 0.00255039,
  'NORAD_CAT_ID': 44463,
  'OBJECT_ID': '2019-006DM',
  'OBJECT_NAME': 'MICROSAT-R DEB',
  'RA_OF_ASC_NODE': 281.0475,
  'REV_AT_EPOCH': 5031},
 {'ARG_OF_PERICENTER': 307.8541,
  'BSTAR': 0.00049837,
  'CLASSIFICATION_TYPE': 'U',
  'ECCENTRICITY': 0.0493635,
  'ELEMENT_SET_NO': 999,
  'EPHEMERIS_TYPE': 0,
  'EPOCH': '2020-06-28T12:08:12.064416',
  'INCLINATION': 93.5021,
  'MEAN_ANOMALY': 47.8974,
  'MEAN_MOTION': 14.87166929,
  'MEAN_MOTION_DDOT': 1.0167e-06,
  'MEAN_MOTION_DOT': 0.00054252,
  'NORAD_CAT_ID': 45478,
  'OBJECT_ID': '2019-006EG',
  'OBJECT_NAME': 'MICROSAT-R DEB',
  'RA_OF_ASC_NODE': 190.5519,
  'REV_AT_EPOCH': 2398},
 {'ARG_OF_PERICENTER': 155.4871,
  'BSTAR': 0.0010172,
  'CLASSIFICATION_TYPE': 'U',
  'ECCENTRICITY': 0.0242905,
  'ELEMENT_SET_NO': 999,
  'EPHEMERIS_TYPE': 0,
  'EPOCH': '2020-06-28T04:11:37.695840',
  'INCLINATION': 95.2289,
  'MEAN_ANOMALY': 205.8168,
  'MEAN_MOTION': 15.47520475,
  'MEAN_MOTION_DDOT': 8.4421e-06,
  'MEAN_MOTION_DOT': 0.0017698,
  'NORAD_CAT_ID': 45480,
  'OBJECT_ID': '2019-006EJ',
  'OBJECT_NAME': 'MICROSAT-R DEB',
  'RA_OF_ASC_NODE': 313.4374,
  'REV_AT_EPOCH': 4134},
 {'ARG_OF_PERICENTER': 174.072,
  'BSTAR': 0.001169,
  'CLASSIFICATION_TYPE': 'U',
  'ECCENTRICITY': 0.0249643,
  'ELEMENT_SET_NO': 999,
  'EPHEMERIS_TYPE': 0,
  'EPOCH': '2020-06-28T07:27:46.340928',
  'INCLINATION': 94.9261,
  'MEAN_ANOMALY': 186.3591,
  'MEAN_MOTION': 15.44819616,
  'MEAN_MOTION_DDOT': 9.2105e-06,
  'MEAN_MOTION_DOT': 0.00188567,
  'NORAD_CAT_ID': 45481,
  'OBJECT_ID': '2019-006EK',
  'OBJECT_NAME': 'MICROSAT-R DEB',
  'RA_OF_ASC_NODE': 293.2403,
  'REV_AT_EPOCH': 4421},
 {'ARG_OF_PERICENTER': 221.7175,
  'BSTAR': 0.0010721,
  'CLASSIFICATION_TYPE': 'U',
  'ECCENTRICITY': 0.027363,
  'ELEMENT_SET_NO': 999,
  'EPHEMERIS_TYPE': 0,
  'EPOCH': '2020-06-28T11:58:59.788704',
  'INCLINATION': 94.7739,
  'MEAN_ANOMALY': 136.2925,
  'MEAN_MOTION': 15.38912648,
  'MEAN_MOTION_DDOT': 6.8359e-06,
  'MEAN_MOTION_DOT': 0.00162102,
  'NORAD_CAT_ID': 45482,
  'OBJECT_ID': '2019-006EL',
  'OBJECT_NAME': 'MICROSAT-R DEB',
  'RA_OF_ASC_NODE': 275.316,
  'REV_AT_EPOCH': 4764}]
>>>
brandon-rhodes commented 4 years ago

@ckuethe — Thanks for the thoughts on the merits of XML and JSON; the satellite folks’ decision to use XML is indeed a sad and very out-of-date one. You might, though, preface your reply with a note to the effect that you are not the Skyfield project lead? Instructions like “Go ahead and just start coding” and expletives directed at XML might have rather a different weight if the reader understands that they are advice from a fellow contributor, and not official contribution instructions.

@benjaminglass1 — Thanks for your interest in this SGP4 library! Before we write any code, let me look more closely at:

https://celestrak.com/NORAD/documentation/gp-data-formats.php

Because it looks like there might already be Python code we can look at. I’ll comment here in more detail after reading!

[Edited: my first reply mistakenly thought this was a Skyfield issue.]

brandon-rhodes commented 4 years ago

@benjaminglass1 — I don't see any useful Python code at https://spacedatastandards.org/ and I don't see any update for the traditional SGP4 C++ library to support the new formats, so it does look appropriate for this library to add code itself for reading the format.

There does seem to have been some code for this feature, though, offered last week over on the Skyfield project that uses this library:

https://github.com/skyfielders/python-skyfield/pull/398

I'll see about getting that author to instead make a pull request here — in which case there might not be much new code needing to be written, @benjaminglass1; we’ll see!

glangford commented 4 years ago

Hello - as mentioned above, I moved my pull request at Skyfield over here.

The goal of this code is to test that TLE and XML sources give similar results, which appears to be the case from what I have seen so far. It shows how OMM XML can be parsed by SGP4 applications and evaluated further, until an API is defined and support is added into the library at some point in the future. Looking forward to discussing further and hearing your comments and suggestions.

dmopalmer commented 4 years ago

One other mode that would be useful is a way of making Satellite object where all the Keplerian elements are arguments to the function, rather than embedded in a TLE, XML, JSON etc. string. Then for each input format, you can extract the values and pass them to that single function. Shared code is better.

It also provides some future-proofing when, e.g., Space Message Tweets becomes the new standard.

This would also simplify, e.g., extracting the elements, modifying them, and making a new satellite record. This is useful for refining a satellite orbit based on observations.

brandon-rhodes commented 4 years ago

@dmopalmer — So to recast your proposal in terms of code: input formats could produce sgp4init() argument lists as their output, allowing new Satrec objects to be initialized with those elements?

https://rhodesmill.org/skyfield/earth-satellites.html#build-a-satellite-from-orbital-elements

ckuethe commented 4 years ago

@brandon-rhodes Noted. I could have made it more clear that I was replying only as just some guy on the internet, albeit one who thinks OMM support is a good idea, not with any level of authority.

There's a reason I specifically said "unreasonable loathing of XML"... because I know I'm not being logical about it.

At the same time, if I was writing the code I'd suck it up and implement an XML code path because that's what the standard says and I'd want it for completeness. There will be people who'd use it, and I don't want to leave them out. Just as a developer forced to interact with XML-based protocols, I find XML transport much less pleasant to work with than JSON or packed binary formats like gRPC or structs on the wire.

dmopalmer commented 4 years ago

I didn't know sgp4init() existed. (Ah, 2020-04-24 version 2.7)

Yes, input format parsers that produce sgp4init() argument lists (or preferably argument dicts if sgp4init can take all named arguments) would be useful. Have ommxml2satellite() call ommxml2_parse() and then sgp4init().

Alternately, if an EarthSatellite could spit out an sgp4init() argument dict (that can round-trip without loss) then you can process an input format all the way, then back up a bit to get the arguments.

(Documentation note: epoch is not days since... it is YYdoy.fracday, if it is the same value as used in TLEs. Also ndot is revs/day^2, not revs/day.)

brandon-rhodes commented 4 years ago

@ckuethe — My only fear was that a new contributor be set to work writing code, thinking they had received a go-ahead from a maintainer. Thanks for understanding!

@dmopalmer — No problem on not having seen the new contributed routine yet! I just wanted to ground your suggestion in code, or else learn what else might be necessary to fully accomplish the approach you are suggesting.

I am likely to make a tuple of arguments the glue between parsers and builders, if only because they are vastly faster in Python, and I already get questions from folks with hundreds of satellites, wondering if there might be edges where the library could be faster. But I'll make it easy for such tuple elements to be addressed with names — you are entirely correct in foreseeing that users will often want to dig in and see what's going on without needing to use integer indexes to reach each orbital element!

brandon-rhodes commented 3 years ago

I have just committed c3f9ee8308f13916dec25b4eb6f18f857b065388 which splits parsing arguments from interpreting them, so that CSV and XML inputs can be used symmetrically in code; see test_omm_xml_matches_old_tle() and test_omm_csv_matches_old_tle() for examples of how the parsing and interpretation calls fit together.

There's a bit of a snag for fields like classification which are (a) not set through sgp4init() but (b) also not directly writable into the C++ structure from Python. I should either switch them to be directly writable or add a special method for setting them. The former option is attractive, as it's simpler, and as I see no danger in letting them be directly set — since they aren't actually orbit parameters, they presumably can't break things too badly or wind up inconsistent if we open the floodgates to users setting them directly. They are:

'classification', 'elnum', 'ephtype', 'intldesg', 'revnum', 'whichconst'

I am likely to release the next version of sgp4 with this new module in place but not documented yet, so followers of this issue can try it out before it becomes publicly advertised and thus set in stone.

astrojuanlu commented 3 years ago

If I understand correctly, there's already support in master to parse OMM XML and CSV into OMM JSON, and to initialize a Satrec object from this JSON. Would you find interesting to have also exporting capabilities? For example, export_tle gaining a format parameter, by default set to "tle" but with a "omm_json" option?

astrojuanlu commented 3 years ago

(In fact, export_satrec, export_gp or export might be better names at this point, to signal that TLE is only one of the many "general perturbations data formats")

brandon-rhodes commented 3 years ago

Would you find interesting to have also exporting capabilities?

I myself have no immediate need for them, but if your own project could use them, I think they would be a straightforward and easy-to-maintain enough addition that it would not burden the library to have them added.

I agree that three routines would be better than a single routine with a 3-way switch. They could all call a common hidden function to do their validation, so that the validation code wouldn't have to be repeated.

A difficulty with the API might be that the JSON and CSV formats necessarily involve many satellites, not just one, if I'm thinking about the file format correctly? Whereas each 3-line TLE is separate from every other, the JSON and CSV files have extra text that stands outside of the individual elements: the square brackets and commas in the JSON, and the header line in the CSV. (But double-check me on those, I'm not looking at the formats myself as I say that, so I could be misremembering.)

Some API questions would come into play, but before I tackle those: is this a feature your project could actively use? Or were you just asking if I needed them? :)

astrojuanlu commented 3 years ago

is this a feature your project could actively use? Or were you just asking if I needed them? :)

As part of one of the sub-activities of the OpenSatCom project we develop with the Libre Space Foundation, my first proposal was to explore the current status of OMM support across the Python ecosystem, and I think having export capabilities to OMM will help us move towards this new wonderful format. So that's the general reason why I'm asking this :)

I see myself using this in at least one internal project, and maybe more public projects going forward.

I agree that three routines would be better than a single routine with a 3-way switch. They could all call a common hidden function to do their validation, so that the validation code wouldn't have to be repeated.

Sounds good!

A difficulty with the API might be that the JSON and CSV formats necessarily involve many satellites, not just one, if I'm thinking about the file format correctly?

From CCSDS 502.0-B-2:

4.1.5 The OMM shall be a plain text file consisting of orbit data for a single object. It shall be easily readable by both humans and computers.

brandon-rhodes commented 3 years ago

The OMM shall be a plain text file consisting of orbit data for a single object.

Oh, wow, you're not allowed to have a CSV file with multiple satellites inside? That's surprising. It does make more sense for the JSON case, though, where otherwise you'd need an outer array to wrap the series of satellites which makes the format complicated in a way the CSV multi-satellite case isn't.

I would like sgp4 to support CSV files with multiple satellites, even if it's not part of the official standard, because it seems such a sensible and flexible way to store a file of satellites. :slightly_smiling_face:

So, my further comments to get you going on a PR:

Let me know what the answers to these questions look like when referenced against the standard (and thanks for reading the standard!), and let me know your opinions about the ideas, and we can make some final API decisions.

astrojuanlu commented 3 years ago

The term JSON does not appear a single time in the standard :sweat_smile: ~And, in fact, XML is not mentioned either.~ Edit: See below. The ony thing that 502.0-B-2 specifies is a list of keywords with specific meaning, and then apparently there is freedom of how to put that into text.

My understanding is that "OMM JSON" is "just" a JSON translation of the OMM keywords (which are standardized), and therefore there are no specific constraints on how the JSON should be formatted. Same goes for OMM XML, OMM KVN, and OMM CSV.

Maybe we can have a export_omm that returns a dictionary with the standard OMM keywords (or a namedtuple, or a dataclass, whatever - but I think a dictionary is simpler) and stop there. Then the user can decide how to serialize that to JSON (trivial), XML (easy? I guess) or KVN (as difficult as one pleases).

For XML:

Description of the message formats based on the use of Extensible Markup Language (XML) is detailed in an integrated XML schema document for all Navigation Data Message Recommended Standards. (See reference [4].) [4] XML Specification for Navigation Data Messages. Recommendation for Space Data System Standards, CCSDS 505.0-B-1. Blue Book. Issue 1. Washington, D.C.: CCSDS, December 2010.

astrojuanlu commented 3 years ago

Sorry, my mistake - there is mention of a XML schema in a separate document. (Putting a quick comment before I add more information so there is no misunderstanding, and editing my previous one)

astrojuanlu commented 3 years ago

Well, I was in the mood of reading the OMM standard, but not so much for an XML standard :) Anyway, I stand by my proposal: having a export_omm that produces a dictionary. After that, proper XML serialization according to CCSDS 505.0-B-1 could be implemented as part of python-sgp4... or as a separate project.

brandon-rhodes commented 3 years ago

Maybe we can have a export_omm that returns a dictionary with the standard OMM keywords…

Yes, I think that makes sense! And it keeps the library as simple as possible for now. We can always add formatting routines later if lots of folks seem to need them; but it might be that we can get away with simple mentions in the README of how to produce each format.

I would be happy to see a PR for an export_omm() routine.

astrojuanlu commented 3 years ago

I have some half-working code, but stumbled upon the problem you described in this comment, namely that satrec.{intldesg,ephtype,classification,elnum,revnum} are not set by Satrec.sgp4init (and therefore, sgp4.omm.initialize does not set them). Setting them from Python, again as you said, doesn't work:

Traceback (most recent call last):
  File "/home/juanlu/Personal/python-sgp4/test_nusat8.py", line 27, in <module>
    initialize(sat, nusat8)
  File "/home/juanlu/Personal/python-sgp4/sgp4/omm.py", line 45, in initialize
    sat.intldesg = fields["OBJECT_ID"]
AttributeError: attribute 'intldesg' of 'sgp4.vallado_cpp.Satrec' objects is not writable

However, some of these keywords happen to be mandatory according to the standard. Therefore, my proposal was to do something like this:

def export_omm(satrec, object_name, object_id, ephemeris_type, classification_type, element_set_no, rev_at_epoch):
    ...

or to wait until you make these attributes writable.

brandon-rhodes commented 3 years ago

Oh, that makes sense. Okay, let's assign this issue to me and I'll take a look at that earlier issue sometime in the next week — or, by the latest, next weekend, when I should definitely have some time — and I'll choose a direction. Thanks for making progress till you reached a block!

brandon-rhodes commented 3 years ago

@astrojuanlu — All right, the two commits you’ll see just above this comment have added the ability to write those 5 attributes, and have expanded the OMM import routine to try writing them. The results look good so far, as you'll see from the tests. I'll un-assign myself on the assumption that this unblocks you, but let me know if you run into further snags!

brandon-rhodes commented 3 years ago

I have just released a new version of the library that, thanks to some code contributed by @astrojuanlu, can do OMM export to a Python dictionary, which the user is then free to output as JSON or CSV or more exotic formats as they please. While users might run into snags that will require a bit more documentation or code, I think that we can now close this issue, since the basic round-trip to and from OMM is now possible.

Feel free to make further comments here in this issue, though, if you have questions about the new support that don't seem to warrant an entire new issue!

And thanks again, everyone, for pointing my attention at the new format, which hadn't been on my radar yet.