Open bch29 opened 4 years ago
Thanks for the report!
I understand from #1187 that it would be very difficult to preserve key order from dhall records, but I am hoping that since the dhall normal form of a map is an ordered list, this case would be easier to handle.
Normalization of Dhall expressions, which sorts the record fields, is one issue, but as you correctly point out, it's not the problem in this case.
After normalization, we translate the normalized expressions to the JSON Value
format of the JSON library we use, aeson
. aeson
uses a hash map for storing objects, and I think it is at this stage that the order of keys is lost in your example.
Thank you for responding quickly!
It does not seem like the reordering is caused by hashing because it is always sorted in the output of dhall-to-json
.
However, if aeson
stored objects as a ordered Map
rather than a HashMap
, that would explain this behaviour.
We use aeson-pretty
for the Value -> ByteString
conversion. Apparently that uses lexicographic sorting.
Apparently we could customize the field order in that step, but I don't see how to use that to solve this issue.
Oh, so what you could do is encode your schema with the Prelude.JSON
type, and then use Prelude.JSON.render
to convert to Text
:
@sjakobi Do we lose value ordering after processing dhall expression?
I think that we could enumerate every key from the dhall expression and use that enumeration to sort. In @bch29 example:
"foo"
comes first in dhall expression, we number it 1"bar"
comes last, we number it 2That goes on a Map Text Int
and we could aim sort using that. That could be an option for the CLI
@german1608 We translate these Prelude.Map
s to RecordLit
s in convertToHomogenousMaps
:
The translation from RecordLit
to Value
then happens here:
I think that we could enumerate every key from the dhall expression and use that enumeration to sort.
Yeah, that could work, but I think it would be tricky to make it reliable. I think we should rather use a JSON or YAML encoding that preserves key order. So encode Expr -> OrderPreservingYAML
, and then translate OrderPreservingYAML -> Aeson.Value
if needed. HsYAML
should be a good fit, but its GPL licence is a problem: https://github.com/haskell-hvr/HsYAML/issues/45
Maybe we should just create our own YAML type. Better leave parsing and encoding to other libraries though. It's a huge mess.
EDIT: I made a question about available libraries on r/haskell.
@sjakobi: The Data.Aeson.Value
type is so simple that it probably wouldn't be that much code to define our own inline variation on that type, except replacing HashMap
with Dhall.Map.Map
. Then we could provide a conversion from our type to Data.Aeson.Value
that discards the order by converting to the HashMap
Indeed. We could consider allowing some non-Text
key types too, as requested in https://github.com/dhall-lang/dhall-haskell/issues/1379.
Oh, so what you could do is encode your schema with the
Prelude.JSON
type, and then usePrelude.JSON.render
to convert toText
:
That is quite awkward for my use case because I have a few more layers of records around what I gave in the example, and those work just fine being left as dhall records. I would have to convert them all to the JSON type, which is a lot of boilerplate.
I am trying to use dhall to generate JSON config for an application in which key order actually matters. It is a GUI program for visualizing tabular data, and the columns are configured like
where the order of the keys in the "schema" object determines the order of the columns in the GUI.
My dhall representation of this JSON object is
however,
dhall-to-json
reorders the fields in the map because "bar" sorts before "foo". Is it possible to havedhall-to-json
preserve key order in this case? I understand from #1187 that it would be very difficult to preserve key order from dhall records, but I am hoping that since the dhall normal form of a map is an ordered list, this case would be easier to handle.