Convert String Representations into Logical Type

[sabbysabs]

I have a JSON structure coming in and it has string representations of Booleans, Ints, Longs. I believe that avsc has the capability to infer if given the right options on the parse function. However, I'm not entirely sure how to go about doing this and would greatly appreciate any assistance.

My sample JSON: input.json

My Schema: runscope.json

My code:

    const type = avro.parse(await schema);

    // Add additional fields needed for pub/sub
    message.IngestionTimestamp = new Date(Date.now()).toISOString();
    // this accepts UPSERT and DELETE, but I expect that DELETEs should not come through.
    message._CHANGE_TYPE = 'UPSERT';
    // this needs to be done before we change the finished_at value
    message._CHANGE_SEQUENCE_NUMBER = Math.ceil(message.finished_at);
    // these come in as floats but we want to change them to a TIMESTAMP string
    message.finished_at = new Date(Math.ceil(message.finished_at)).toISOString();
    message.started_at = new Date(Math.ceil(message.started_at)).toISOString();
    // avsc doesn't change a string boolean to a boolean object
    message.agent_expired = Boolean(message.agent_expired);
   console.log(type.toString(message));

The error message I get is this:

Error: invalid ["null","long"]: "1258"
    at throwInvalidError (/Users/H476998/Desktop/work-items/digital-uptime-results/node_modules/avsc/lib/types.js:3042:9)
    at UnwrappedUnionType._copy (/Users/H476998/Desktop/work-items/digital-uptime-results/node_modules/avsc/lib/types.js:1383:7)
    at RecordType._copy (/Users/H476998/Desktop/work-items/digital-uptime-results/node_modules/avsc/lib/types.js:2481:26)
    at ArrayType._copy (/Users/H476998/Desktop/work-items/digital-uptime-results/node_modules/avsc/lib/types.js:2052:31)
    at RecordType._copy (/Users/H476998/Desktop/work-items/digital-uptime-results/node_modules/avsc/lib/types.js:2481:26)
    at Type.toString (/Users/H476998/Desktop/work-items/digital-uptime-results/node_modules/avsc/lib/types.js:678:30)
    at publishMessage (/Users/H476998/Desktop/work-items/digital-uptime-results/avro-helper.js:19:22)
    at publishAvroRecords (/Users/H476998/Desktop/work-items/digital-uptime-results/index.js:69:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

[mtth]

Hi @sabbysabs. avsc doesn't expose an option to auto-convert types to their string representation. You could achieve it with the combination of a logical type to do the transformation and a type hook to apply it where it makes sense, but I would not recommend it as first option. If possible, it would be better to update the schema to reflect the true types, or explicitly convert the fields that need it.