Should be able to specify order of metadata fields

[tfmorris]

The current API delivers the metadata as a JSON object, but this precludes specifying the ordering of the fields which are presented to the user and instead makes them effectively random. This leads to orderings like Latitute (sic), Elevation, Longitude in the current season spotter.

There needs to be a way to specify the ordering of the fields to be presented to the user. One possibility would be to make the metadata an array of name/value tuples. Another would be to include ordering info in the object. Unfortunately both are breaking changes to the API.

[mkosmala]

+1

I imagine the easiest thing to do is to have a(n optional) field in the metadata (called "order"?) that is a list of integers representing the order the project builders want to see the metadata displayed.

[camallen]

I agree with @mkosmala here. We could provide an array of ordered keys or ordering object with the metadata json object that can be used to preserve this.

{
  "RA": "ra_value",
  "DEC": "dev_value",
  "orderArray": ["field1", "field3", "field2"],
  "orderObject": {
    "0": "field1",
    "1": "field3",
    "2": "field2"
  }
}

[marten]

I agree with @camallen but I think this should be on the Project or Workflow rather than on the Subjects themselves. I can see why you would do it on the subjects themselves, but in practice the aim is consistency in presentation. Doing it per subject would also mean so much repetition of data in our database that it'll just waste tons of space.

[eatyourgreens]

A given workflow could have different subject sets, each with different subject metadata. Even within a subject set, I don't think there's a requirement for all the subjects to have the same metadata.

Off the top of my head, Penguin Watch is an example of a project where the workflow has stayed constant, but the subject metadata has changed with each batch of uploaded subjects.

[chrislintott]

@camallen Can you make a decision about strategy here and assign appropriately?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

Closed by probot-stale due to a lack of recent activity. Please feel free to re-open if you wish to take on this change.

[eatyourgreens]

Ordered Maps are a possible way to handle this in JavaScript. I don't know how that would serialise to Panoptes.

[marten]

https://github.com/zooniverse/Panoptes/issues/2933 suggests that there is a JS library that would serialize and deserialize Maps into JSON correctly, which is planned to be used for the implementation of workflow steps.

[marten]

But of course using that might not work since PFE isn't the only one that creates subjects. There are a bunch of different ways of uploading subjects with metadata.

I still stand by my 2016 argument: this should go on the workflow or project level to avoid repetition. Missing keys in some subjects' metadata shouldn't matter, whatever keys there are should adhere to the given ordering, and any keys not specified by the ordering should be ordered in some sensible way (eg alphabetically but sorted after all the ones which do have a set ordering).

[eatyourgreens]

PFE could be updated to use new Map() instead of {} here: https://github.com/zooniverse/Panoptes-Front-End/blob/0dfddd2ce1e80401158cf0ed0f9dc9536af589e5/app/pages/lab/subject-set.cjsx#L306-L311 That would involve rewriting the subject set lab component in JS, since new Map() won't work in coffeescript, and the JSON-API client will probably have to be updated to support Maps too.

[marten]

Yes, but then we'd probably also have to add support for Map to the Python and Ruby clients.

[eatyourgreens]

True, but we're using Map for workflow steps so that's going to have to happen anyway.

[marten]

Makes me wonder if that's a good idea.

[marten]

Having looked more into the library referenced by @srallen I'm now worried this might be a bad idea. That libary does not at all specify how Maps will be converted into JSON. There is no specification at all there. This means that it's difficult for libaries in other languages to adhere to the same standards.

I can't tell if the ADR considered interoperability between PFE and non-JS applications?

[srallen]

I didn't consider interoperability when choosing since I was only considering the new steps property on workflows. The lab page for editing workflows would be redone eventually to support this new feature in the lab app and the plan was to use the same library there.

We could DIY the serialization and deserialization and document how to do it instead?

[srallen]

@marten the library's serialization method converts Map to JSON Object: https://github.com/sonnyp/JSON8/tree/master/packages/json8#ooserialize

The parse method is configurable to convert JSON Object back to Map: https://github.com/sonnyp/JSON8/tree/master/packages/json8#ooparse

[eatyourgreens]

If it's serialised to Object before posting to Panoptes, then doesn't that lose the key order? Metadata is serialised as a JSON Object at the moment, which is why we can't guarantee the order of the keys when we request it back from Panoptes.

[srallen]

Yeah, this is my bad at not looking more closely at the library. We can DIY it as a JSON array of pairs.

[eatyourgreens]

Thinking about it, that's how subject.locations is stored, in order to preserve the order of the uploaded files. That's a pretty easy change to make in PFE, but will be a breaking change for anything else that uses the API.

[srallen]

I'm working on implementing the modal in the rewrite using Grommet's DataTable. It comes with a bunch of functionality built in including sorts, search, etc. Perhaps some of this issue can be solved with this UX. Here's an example:

Staging's I Fancy Cats project's subject metadata is not in alphabetic order:

With Grommet's DataTable sort, you can click the column header to put it in alphabetic order:

@mrniaboc Would this be a good intermediate type solution for ordering?

Would anyone be interested in any of the other functionality Grommet's DataTable has like search for the subject metadata table?