circuithub / rel8

Hey! Hey! Can u rel8?
https://rel8.readthedocs.io
Other
150 stars 38 forks source link

Support nested `catListTable` (by representing nested arrays as text) #242

Closed shane-circuithub closed 9 months ago

shane-circuithub commented 1 year ago

This is a possible fix to #168. With this we can catListTable arbitrarily deep trees of ListTables.

It comes at a relatively high cost, however.

Currently we represent nested arrays with anonymous records. This works reasonably well, except that we can't extract the field from the anonymous record when we need it (PostgreSQL theoretically suports .f1 syntax since PG13 but it only works in very limited situations). But it does mean we can decode the results using Hasql's binary decoders, and ordering works how we expect to (array[row(array[9])] < array[row(array[10])]).

What this PR does is instead represent nested arrays as text. To be able to decode this, we need each DBType to supply a text parser in addition to a binary decoder. It also means that ordering is no longer intuitive, because array[array[9]::text] > array[array[10]::text]. However, it does mean we can nest catListTables to our heart's content and it will always just work.

ocharles commented 1 year ago

Is this basically a variant on what we used to do when encoding to JSON?

shane-circuithub commented 1 year ago

I don't recall that we ever did actually use JSON for this but we did talk about it. As far as I know PostgreSQL doesn't have a way to reliably convert arbitrary things from JSON so I don't think JSON could really be used for this. There's also no guarantee that the FromJSON instance on the Haskell side will line up with the JSON that Postgres would convert the array to. At least the text encoding is stable and we can convert things to and from it.

ocharles commented 10 months ago

I don't recall that we ever did actually use JSON for this but we did talk about it.

Yea, I had it working but couldn't find anything in the old rel8 branches. Oh well. Would have been around Mar 2021.

As far as I know PostgreSQL doesn't have a way to reliably convert arbitrary things from JSON so I don't think JSON could really be used for this.

I think what I was doing previously was (for non-JSON things): casting to ::text, then casting to ::jsonb. This way you're basically always using the text encoding, but building structure in JSON.

There's also no guarantee that the FromJSON instance on the Haskell side will line up with the JSON that Postgres would convert the array to. At least the text encoding is stable and we can convert things to and from it.

FromJSON isn't involved with this. The TypeInformation would know that we have our own internal JSON structure that's used by various things, just like your Decoder. That said, Decoder is basically what I had before, and I didn't like it because we've now got to maintain two parsers - one for binary and one for text. This seems like a lot of complexity. We're also duplicating all of the parsing that postgresql-simple does. This all makes me a little uneasy, and I think is what you're alluding to when you say "comes at a high cost"