Creation of a custom epheremis object

[fraserw]

Tagging @brianmajor in the CADC who is helping me with this coding.

I am looking for input from the TOMs dev team on the approach I have taken to handling the following observing situation which requires custom ephemerides for a target. I have worked on this some already, but don't want to get too far down the path if the approach I have taken is deemed inappropriate.

Describe the bug

Here I describe the need for this feature, and the work to date.

In object discovery, a state is often reached where an object is known, and it's ephemeris is well enough known to get follow-up or tracking observations in the not too distant future, but the orbit is not yet certain. In these cases, it is more practical to think of the object's trajectory (ephemeris) rather than it's orbital elements, which will be massively uncertain.

In that sense, from an observer standpoint it is better to be able to provide an ephemeris directly, rather than the uncertain orbital elements. Specifically, it is better to provide the ephemeris uncertainty, rather than an ephemeris uncertainty which itself is estimated from the uncertain orbit.

The feature I am working on addresses the above observing scenario.

In Summary: The ability to create a new TOMs target for a moving body not yet in the JPL or MPC databases, and also doesn't have well defined orbital elements. That is, create a new target based on a user-provided ephemeris rather than a user provided set of elements.

To Reproduce Create target drop down -> Custom Ephemeris -select local ephemeris file (in the JPL format)

Expected behavior Create a new target based on name in the ephemeris file and store the ephemeris in the TOMs.target object.

Screenshots I have provided two screenshots. The first is the targets page, showing the dropdown item added. That shot also shows a target (Makemake) created after the upload.

The second shot shows the simple file upload page modelled after the targets csv upload page.

Description of work to date

The work I have done to date is in extending the target object in tom_targets/models.py to include django database objects capable of storing the ephemeris. Specifically, I have created a new target type "EPHEMERIS", and added additional fields:

• centre-site name (CharField)
• Ephermis Right Ascensions (TextField)
• Ephemeris Declinations (TextField)
• Ephemeris Modified Julian Dates (TextField)

centre-site name contains the geographic telescope code (eg. 568 for Mauna-Kea), and the latter three objects contain the ephemeris itself in space-delimited text.

Description of needed work Two additional text fields will be added to handle uncertainties in ra and dec.

Modify the EPHEMERIS target type to handle ephemerides for different locations. This could simply be the requirement of a new EPHEMERIS target for each telescope site in question, or alternatively an extension of the above fields for multiple sites. Input on this would be much appreciated.

Modify tom_observations to make use of the new EPHEMERIS target type.

[fraserw]

Oops. Forgot the screenshots.

Dropdown.pdf Import target.pdf

[dmcollom]

Hi Wes,

Much appreciate you taking a look at this! I think I understand what you're getting at--you effectively want a way to store a series of RA/Dec/Time values for a target, most likely uploaded from a file.

I have a thought on how to implement it, and it should be relatively simple. I do want to mention that, in general, for a Django reusable app, extending models can be a tricky thing, because all of the views that rely on Target would have to be changed. However, something we do in multiple other places (ObservationRecord, ReducedDatum) is implement a TextField on the model, which we actually use to store JSON.

So instead of creating a new model, I would suggest implementing ephemerides as a new field on Target called, you guessed it, ephemerides, and have it store JSON. Then, when a user uploads a supported format, it can serialize it like so:

{
  "ephemerides": [
    {
      "time": 52000,
      "ra": 269.73,
      "dec": -29.65
    },
    {
      "time": 52001,
      "ra": 269.73,
      "dec": -29.65
    }
  ]
}

Furthermore (since I'm unsure how common it is to store ephemerides by site), this would enable someone like yourself to modify the upload code to include site information as well (alternatively, the code handling the upload could configure the JSON to have a site-based hierarchy if site information is present). To take it even further, the hierarchy could be ephemeris -> site -> time -> ra/dec, to ensure that there aren't duplicate entries for a timestamp.

Just for a couple more details, this also seems like the type of thing that we'd only want to be present for the moving objects case, so I'd probably suggest hiding some of the UI elements if settings.py includes a TARGET_TYPE that isn't NON_SIDEREAL. Ensuring the logic for processing the file is contained in the view itself would also be essential, because it would enable anyone that wants to modify the processing logic to simply inherit from the view and just override the one function that handles it.

Does all that make sense? Do you think it would work for your use case? Let me know what you think, and I'm very happy to collaborate with you on getting something implemented.

[dmcollom]

I'm adding this as a separate comment, just to ensure that my thoughts aren't conflated. Here's a completely separate suggestion for how to achieve this, and it wouldn't require significant code changes (or even a PR!)

The ReducedDatum model has foreign keys for Target and DataProduct (as well as ObservationRecord). We currently use it for storing photometry and spectroscopy--spectra in particular are stored as lists of magnitude/flux/error sets. The classes managing the processing of photometry and spectroscopy exist in tom_dataproducts/processors/, and they both implement the generic DataProcessor class in tom_dataproducts/data_processor.py.

I think the easiest way to get what you're looking for going would be to add another DataProcessor (the documentation has a section on doing that here), and then implement the data processing in a similar way to my above comment. Ultimately, the decision is yours--I think that the first solution treats ephemerides as permanent Target attributes, whereas the second treats them as observed data, so it's up to you for which you think fits your mental model better and which is more easily implemented.

[fraserw]

Hi David. Thanks for getting back to us.

I will respond to your second comment in a second response. To the first one...

Two clarifications:

a) I am not creating a new model. I have just added four additional fields to the target object in tom_targets, along with the new target type EPHEMERIS (just like SIDEREAL or NON-SIDEREAL). This is already as textfields like stated.

b) ephemerides are 100% site specific. Bluntly, how the angle you'd point a death laser at to melt the Eiffel tower would be quite different from New York then from the moon. This is true of asteroids and telescopes as well. So no matter how we implement this, it has to be done in a site specific way.

I see the benefits of JSON format, but I am not sure how to work with it, or even how to implement it, and store it as a text field. I can do the space delimited stuff pretty easy. But if we want JSON, I'll need help or an example on that one.

It would be nice if we could just store things as nice simple python objects then stored as pickles, but that seems not so Django-esque.

As for the UI, are you suggesting that I implement a view specific to the EPHEMERIS object? It's a good idea, though I am not yet familiar enough with TOMs to do that. TBI.

[fraserw]

Replying to the second comment.

I am not sure I know what you're talking about. Or at least, very confused. It seems you've proposed a chicken-egg problem. How does one create reduced data from data that doesn't exist. Or alternatively, how does one "process data" to create telescope observations.

Can you try again?

[dmcollom]

I'm sorry, Wes, I'll try to do a better job explaining. And I'm going to drop the second suggestion--so don't worry about that one.

A few things in the first case, the JSON field option:

• It makes sense that the ephemeris fields belong as fields on the Target, but if I'm understanding correctly, then although the orbital elements are uncertain, they may become certain. If that's the case, it seems that a Target defined as an EPHEMERIS type wouldn't not necessarily stay so. Because of that, as well as the fact that EPHEMERIS type seems to overlap with NON_SIDEREAL type, I would say that this can be implemented without expanding the available types.
• So JSON isn't too much more difficult to work with than space-delimited fields, and we use it as a standard everywhere else in the TOM. It also makes a lot more sense in this case because it's hierarchical, so you'll be able to group ephemerides by sitecode. The thing that makes it so easy is that Python dicts are essentially JSON, so you can do the following:

in tom_targets/models.py

class Target(models.Model):
# various fields
  ephemerides = models.TextField()

ephemerides = {
  "mko": [
    {
      "time": 52000,
      "ra": 269.73,
      "dec": -29.65
    },
    {
      "time": 52001,
      "ra": 269.73,
      "dec": -29.65
    }
  ]
}

t = Target(name='Makemake')
t.ephemerides = json.dumps(ephemerides)
t.save()

So what's great is that any Python dict can be converted to a string with json.dumps(), and any valid string of JSON can be converted back to a dict with json.loads().

To bring it back to the feature at hand, I can give a couple of quick highlights on how I would implement this:

1) As you suggested, add a dropdown item to the existing menu to upload a set of ephemerides. 2) Create a view virtually identical to tom_targets.views.TargetImportView, just one that calls a new method called import_ephemerides(csv_stream) and handles the successes a little differently. 3) Create a corresponding new method in tom_targets.utils.import_targets called import_ephemerides that reads in the csv and stores it as a Python dict, before converting it to JSON and saving a new Target object.

Does that make sense? There are a couple of little extra items that will come up (like modifying the validation on a non-sidereal target to require either ephemerides OR orbital elements), but hopefully it doesn't seem too bad. Also, you mentioned you've started to work on this--do you have a branch as a reference?

[fraserw]

Hi David. I wasn't confused by your suggestion about json and views, only your suggestion of using the ReducedDatum class. Fortunately you are suggesting nearly exactly the same thing I have already done!

That json.dump/load and the dictionary key were all I needed. I implemented items 1, 2, and 3 last week already using space delimited strings. Modifying that to json is a trivial task. Excellent.

One very important point First I apologize for misspeaking. I actually made a different scheme for the non-sidereal type, not a whole new target type. Still only SIDEREAL or NON-SIDEREAL.

I then modified the required fields for NON-SIDEREAL to be only scheme and epoch_of_elements. The elements themselves were moved to the various element schemes (MPC_MINOR_PLANET, MPC_COMET, JPL_MAJOR_PLANET) and EPHEMERIS scheme has its own requirements.

I presume that is TOMs-esque enough?

[dmcollom]

Oh, adding it as another scheme is great. And yeah, that all seems perfect. Very excited to see the finished product!

[fraserw]

Excellent.

One last thing - is there a max textfield limit I should be sticking with? One could imagine running into limits pretty easily, depending on the input ephemeris. For example, half a year of hourly ephemerides could easily work out to be XX MB per object.

The Math

MJD XXXXX.XXXXX - 11 characters RA XXX.XXXXX - 9 DEC sXX.XXXXX - 9 dRA (in ") XX.XX - 5 dDec (in") XX.XX - 5 PA (deg) XXX.XX - 6

Labels require another ~25 characters, with JSON occupying a similar number.

Total characters per entry: ~84

For half a year of hourly ephemerides at 8 telescope sites, we need nearly 3 million characters, or ~6 MB for the database entry. lol

This is just one potential use case, but not atypical of the things I have done with Gemini in the past.

Do we want to store these as static files, or stick with DB entries with some length limit?

[dmcollom]

Ah. Yes. That amount of data seems a little prohibitive. I believe that the current solution you're pursuing will work for some cases and still have a lot of value, but that for this amount of data, we may want to consider something else. (Also, side note, the advantage of a TextField is that you don't need to specify a max_length, unlike a CharField).

So I'm going to take this opportunity to explain in detail the DataProduct solution, of which you were understandably skeptical.

To give a brief overview of the data model, there are four core models--Target, ObservationRecord, DataProduct, and ReducedDatum. While I'm sure that what jumps to mind immediately with a DataProduct is something that is sourced from an observation, the concept behind a DataProduct is that it's really any file containing data. The file itself is saved to disk and the DataProduct includes file information. Meanwhile, a ReducedDatum is where an individual point of data would be stored. A ReducedDatum has a timestamp, and then a TextField for arbitrary data storage.

It seems like for this level of data, you could, upon uploading a file of ephemerides, save the file as a DataProduct. You could then store each ephemeris from the DataProduct as a ReducedDatum, which would enable you to do a range search for a time when necessary. It'd probably be a similar amount of coding to what you already have.

Does that make more sense than when I explained it the first time? I mentioned it to Rachel and she suggested that if you have more questions, we should set up a brief video call to discuss some options.

[fraserw]

Yep! That makes perfect sense, Thanks! And I agree with you, would be a safer implementation. For now I have the json working all the way up to (but not including) creating observations with an EPHEMERIS scheme, so I'll leave it there. If I have time in the project, I'll swing back and look at using the DataProduct idea!

Question about the calls made in ObservationCreateView. Unsurprisingly, I am getting the "scheme: "EPHEMERIS" is not a valid choice." error. What I can't figure out yet is what code is called that checks for target type and scheme, and from there, how RA and Dec are extracted for the MPC/JPL orbit schemes.

Can you point me to the correct module/code blob that at least throws that error?

Thanks again David

[dmcollom]

Well, if you happen to have a Github branch I can look at, that would be helpful! But otherwise, the error message you're seeing is likely the built-in Django validation. At the risk of stating the obvious, you would need to add EPHEMERIS to TARGET_SCHEMES. The follow-up, since you almost certainly did that, is that you need to create a migration with ./manage.py makemigrations and then run the migration with ./manage.py migrate. The valid choices are part of the table information, so the DB needs to be updated. That would be my first guess at what's missing to cause that.

As far as the extraction of RA/Dec, are you referring to pulling things from the catalog, or in the upload?

[fraserw]

Hmm I did all of that already. Will continue to dig. When back to my laptop I'll push the changes.

On Tue., Feb. 11, 2020, 4:56 p.m. David Collom notifications@github.com wrote:

Well, if you happen to have a Github branch I can look at, that would be helpful! But otherwise, the error message you're seeing is likely the built-in Django validation. At the risk of stating the obvious, you would need to add EPHEMERIS to TARGET_SCHEMES. The follow-up, since you almost certainly did that, is that you need to create a migration with ./manage.py makemigrations and then run the migration with ./manage.py migrate. The valid choices are part of the table information, so the DB needs to be updated. That would be my first guess at what's missing to cause that.

As far as the extraction of RA/Dec, are you referring to pulling things from the catalog, or in the upload?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/TOMToolkit/tom_base/issues/247?email_source=notifications&email_token=ABRJZPRZ5TAURBOBHL6LOM3RCNCL7A5CNFSM4KSWUDP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELO3KWA#issuecomment-584955224, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABRJZPXNAH3W6FFMQKP2IKTRCNCL7ANCNFSM4KSWUDPQ .

[fraserw]

Hi @dmcollom. I have pushed my changes my fork of tom_base. You should be able to see my changes there.

I also did confirm that I added the scheme "EPHEMERIS" to the TARGET_SCHEMES list in tom_targets/models.py. So not sure what is causing the invalid scheme error.

Thanks for the help!

[dmcollom]

Very happy to help! This is a cool thing to be added.

So I'm actually not even able to get to the ephemeris upload--I'm not entirely sure that all the files are currently pushed (I'm missing the target_ephemeris_import.html template). And can you clarify, where are you seeing the error scheme: EPHEMERIS is not a valid choice? You mentioned the ObservationCreateView--is this in a submission to LCO, or is this a different form? If it's LCO, the LCO Observation Portal doesn't accept ephemerides, because the scheduler doesn't know when the moving object will be observed, so it has to calculate the ephemeris when it's scheduled. If this is the case, the error you're seeing is from a validation by the Observation Portal, not something in the TOM.

One other thing I might mention is that I would prefer that a single target should store the ephemerides for all sites for that target. To me, it doesn't make sense that a TOM would have multiple Target objects for a single Target simply because it has data for more than one site. Again, this can all be addressed with the JSON hierarchy, and including site information in the JSON. That will allow the data to be more agnostic to the TOM-supported facilities, as well--that is, a facility doesn't have to exist in the TOM to have a set of ephemerides associated with it.

[fraserw]

Oh duh. Forgot to add the ephemeris file. Added and pushed. Further comments in-line

So I'm actually not even able to get to the ephemeris upload--I'm not entirely sure that all the files are currently pushed (I'm missing the target_ephemeris_import.html template). And can you clarify, where are you seeing the error scheme: EPHEMERIS is not a valid choice? You mentioned the ObservationCreateView--is this in a submission to LCO, or is this a different form? If it's LCO, the LCO Observation Portal doesn't accept ephemerides, because the scheduler doesn't know when the moving object will be observed, so it has to calculate the ephemeris when it's scheduled. If this is the case, the error you're seeing is from a validation by the Observation Portal, not something in the TOM.

I am seeing the full error

scheme: "EPHEMERIS" is not a valid choice. when I attempt to create an observation, eg at the observations/LCO/create/ link.

So if you are right, how do we move forward? That is, how can we provide the ephemeris json data to the scheduler such that it can pull the correct coordinates? It must be that the scheduler has built in a way to estimate those from a set of orbital elements, so I guess we just need to make it read ephemerides instead?

One other thing I might mention is that I would prefer that a single target should store the ephemerides for all sites for that target. To me, it doesn't make sense that a TOM would have multiple Target objects for a single Target simply because it has data for more than one site. Again, this can all be addressed with the JSON hierarchy, and including site information in the JSON. That will allow the data to be more agnostic to the TOM-supported facilities, as well--that is, a facility doesn't have to exist in the TOM to have a set of ephemerides associated with it.

Yup, this is already how I did it, as you'll see from the code. Currently, the provided epheremeris file is actually a set of JPL formatted files, appended one after another. The code checks for ephemerides for the appropriate telescope locations and stores accordingly. The json format is a dictionary of dictionaries, with first set of keys corresponding to telescope locations, and the second set corresponding to ephemeris entries (mjd, ra, dra, dec, ddec) for each telescope location.

[dmcollom]

Yup, this is already how I did it, as you'll see from the code.

Ah, my mistake, I saw that the Target model has a new field for site and I just assumed. Would you mind uploading an example file of ephemerides as well? I'm sure they're easily gotten from JPL Horizons to someone who knows, but it's not exactly clear to me where.

It must be that the scheduler has built in a way to estimate those from a set of orbital elements, so I guess we just need to make it read ephemerides instead?

Unforunately, LCO as an organization doesn't really have a need or a plan to read ephemerides from for an observation. I presume that the scheduler derives the position of a moving object from the orbital elements (how could it not, after all), but given that the scheduler runs the scheduling process roughly every 15 minutes and doesn't know which ephemerides it needs until it schedules the observation, it isn't practical to configure it to read from an ephemerides file at the moment.

Now, I recognize that that hardly solves your problem, but presumably you have a way to approximate orbital elements even for a moving object with uncertainties? One thing that would work is to ensure that the orbital elements are up to date upon observation submission and submitting with shorter windows. If the elements are from the MPC or can be calculated on the fly, you could definitely write a bit of TOM-specific code to update the elements prior to observation submission.

[dmcollom]

One other note: my colleague Joey Chatelain mentioned to me in the past that this package, sbpy, may be of value. It looks like it has capabilities to pull orbital elements from both JPL Horizons and the MPC, which would make the on-the-fly update more feasible.

[fraserw]

Morning. The example ephemeris file is in static/tom_targets. I had added that in a second push. It's for Makemake, just because that's the one I randomly chose.

OK this complicates things greatly. I think a code rewrite will be necessary.

The whole point of this exercise was to be able to observe objects that were not yet of the quality to be submitted to the MPC, and therefore would not be in either the MPC or JPL catalogs.

I am not sure if it's possible to generically create a set of elements that accurately reproduces an ephemeris (especially accurately reproduces the ephemeris uncertainty). This particular requirement - that ephemerides are determined from elements alone - might just make this use case impossible with the LCO. Which would suck.

[rachel3834]

Hi Wes, Do you have time to talk me through your use-case in detail here? I'd like to understand the workflow a bit better and see what we can do to support you.

Thanks Rachel

On Thu, Feb 13, 2020 at 10:36 AM Wesley Fraser notifications@github.com wrote:

Morning. The example ephemeris file is in static/tom_targets. I had added that in a second push. It's for Makemake, just because that's the one I randomly chose.

OK this complicates things greatly. I think a code rewrite will be necessary.

The whole point of this exercise was to be able to observe objects that were not yet of the quality to be submitted to the MPC, and therefore would not be in either the MPC or JPL catalogs.

I am not sure if it's possible to generically create a set of elements that accurately reproduces an ephemeris (especially accurately reproduces the ephemeris uncertainty). This particular requirement - that ephemerides are determined from elements alone - might just make this use case impossible with the LCO. Which would suck.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/TOMToolkit/tom_base/issues/247?email_source=notifications&email_token=ACPJA3HP2KEES6MYXKPPOY3RCWHK5A5CNFSM4KSWUDP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELWDSKI#issuecomment-585906473, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACPJA3EQMXIR3VAXIWYO7S3RCWHK5ANCNFSM4KSWUDPQ .

[fraserw]

Thanks Rachel. In a little bit I will send out a doodle to the relevant folks on this end (JJ and Brian) and you and David. Anyone else I should include?

[fraserw]

Since I don't have everyone's email addresses: doodle poll

[fraserw]

Just for everyone's info, here's what I'm thinking about in regards to ephemeris sampling. I'd resample frequently enough to ensure that the object doesn't move no more than ~1/3rd of a camera FOV (fraction being user tuneable). So for KBOs and SINISTRO that's every 2 hours, with 5-6 observations created per target, per night.

For asteroids which move an ord r of magnitude faster, 50 per night, and for comets like 2/I, as many as 100 per night.

Then multiply those numbers to account for how many images are required to cover a full error ellipse.

Just an FYI.