search

streamingfast / substreams-sink-sql

Apache License 2.0
11 stars 14 forks source link

Binary value transfer and persistence #8

Closed let4be closed 1 year ago

let4be commented 1 year ago

I've been looking through code and it seems all values exported from substream are encoded as a string map...

It could be convenient but inefficient and it doesn't help with DB insertion either - having all insert types to be defined purely from golang's Reflect definitions(which in turn are deduced by database/sql) limits ones abilities to use database specific types(like bytea)

I would like to see an ability to export protobuf's bytes from substream and persist postgres bytea. Proper data binding should be utilized instead of trying to format params yourself, this needs to die - https://github.com/streamingfast/substreams-sink-postgres/blob/019a993580ddca8d1be6514db6a98c53ea4534a7/db/operations.go#L129

maoueh commented 1 year ago

I think the current way of doing things today gives room to use any Postgres type, simply format as a string how psql would accept it.

The bytes optimization will be fixed by #14 which defines more transfers data type and where bytes could be transferred without being hex encoded at all.

Today, for bytea, returns a hex encoded string to be correctly inserted in Postgres.