After Upgrading azure package to 1.4.0, QueueInputSource failed to read messages written by older (<1.4.0) CC QueueOutputSink with default preprocessor #324
It appears that the latest 1.4.0 azure package is using @azure/storage-queue lib instead of azure-storage lib used by the older CC version. Apparently, there is an inconsistency between how messages are encoded (by default) and how they are decoded between @azure/storage-queue and azure-storage. Newer library failed to decode message encoded by default TextXmlQueueMessageEncoder with older library.
Error:
SyntaxError: Unexpected token & in JSON at position 1 at JSON.parse (<anonymous>) at EnvelopeQueueMessagePreprocessor.process (/Users/yug0003/Documents/store-initialization-checkpoints/services/store-initialization-checkpoints/node_modules/@walmartlabs/cookie-cutter-azure/dist/utils/QueueClient.js:43:21) at /Users/yug0003/Documents/store-initialization-checkpoints/services/store-initialization-checkpoints/node_modules/@walmartlabs/cookie-cutter-azure/dist/utils/QueueClient.js:221:56 at Array.reduce (<anonymous>) at QueueClient.<anonymous> (/Users/yug0003/Documents/store-initialization-checkpoints/services/store-initialization-checkpoints/node_modules/@walmartlabs/cookie-cutter-azure/dist/utils/QueueClient.js:219:57) at Generator.next (<anonymous>) at fulfilled (/Users/yug0003/Documents/store-initialization-checkpoints/services/store-initialization-checkpoints/node_modules/@walmartlabs/cookie-cutter-azure/dist/utils/QueueClient.js:11:58) at runMicrotasks (<anonymous>) at processTicksAndRejections (internal/process/task_queues.js:93:5)
Encoded message written with older CC azure queue sink:
{"payload":{"context":{"store":{"tenantId":"walmart-us","storeId":"3014"},"time":{"eventTime":{"seconds":1645131671,"nanos":770000000}},"correlation":{"parentEventIds":[{}],"api":{}},"location":{"point":{}},"camera":{}},"laneNumber":3},"headers":{"event_type":"kepler.store.checkpoints.initialization.events.InferenceRunningCompleted"}}
It appears that the latest 1.4.0 azure package is using @azure/storage-queue lib instead of azure-storage lib used by the older CC version. Apparently, there is an inconsistency between how messages are encoded (by default) and how they are decoded between @azure/storage-queue and azure-storage. Newer library failed to decode message encoded by default TextXmlQueueMessageEncoder with older library.
Error:
SyntaxError: Unexpected token & in JSON at position 1 at JSON.parse (<anonymous>) at EnvelopeQueueMessagePreprocessor.process (/Users/yug0003/Documents/store-initialization-checkpoints/services/store-initialization-checkpoints/node_modules/@walmartlabs/cookie-cutter-azure/dist/utils/QueueClient.js:43:21) at /Users/yug0003/Documents/store-initialization-checkpoints/services/store-initialization-checkpoints/node_modules/@walmartlabs/cookie-cutter-azure/dist/utils/QueueClient.js:221:56 at Array.reduce (<anonymous>) at QueueClient.<anonymous> (/Users/yug0003/Documents/store-initialization-checkpoints/services/store-initialization-checkpoints/node_modules/@walmartlabs/cookie-cutter-azure/dist/utils/QueueClient.js:219:57) at Generator.next (<anonymous>) at fulfilled (/Users/yug0003/Documents/store-initialization-checkpoints/services/store-initialization-checkpoints/node_modules/@walmartlabs/cookie-cutter-azure/dist/utils/QueueClient.js:11:58) at runMicrotasks (<anonymous>) at processTicksAndRejections (internal/process/task_queues.js:93:5)
Encoded message written with older CC azure queue sink:
{"payload":{"context":{"store":{"tenantId":"walmart-us","storeId":"3014"},"time":{"eventTime":{"seconds":1645131671,"nanos":770000000}},"correlation":{"parentEventIds":[{}],"api":{}},"location":{"point":{}},"camera":{}},"laneNumber":3},"headers":{"event_type":"kepler.store.checkpoints.initialization.events.InferenceRunningCompleted"}}