Problematic parsing of simple string arrays

[zolting]

The pinax fork of substreams-sink-sql now supports ArrayClickhouse types, but the current implementation implies :

• In the Substream Rust code
  • Building a string representing the array (e.g "[123, 345, 678]" or "['one', 'two', 'three']"
  • Passing the string in a row.change() call with substreams-database-change Rust library
• substreams-sink-sql then JSON unmarshals the array string to then insert the array into Clickhouse

This approach works perfectly in most cases, but fails with arrays that contain strings like :

Program log: Some(AnchorError(AnchorError { error_name: \"AggregatorAccountNotFound\", error_code_number: 6028, error_msg: \"Aggregator account missing from provided account list.\", error_origin: Some(Source(Source { filename: \"programs/switchboard_v2/src/actions/crank_pop_action.rs\", line: 150 })), compared_values: None }))

In this situation, a simple solution could be to simply make a transaction_log_messages Clickhouse table instead of using a log_messages field in the transactions table, but this comes at the expense of making the schema more complex than necessary

Or, if it's not too much work, adapting the substreams-database-change library and substreams-sink-sql so that it better handles arrays could be great. Maybe a method like change_array() ?

[zolting]

Managed to fix most parsing issues by simply using serde_json but there are still some edge cases

pub fn string_array_to_string_with_escapes(values: &[String]) -> String {
    serde_json::to_string(values).unwrap_or_else(|_| "[]".to_string())
}