walmartlabs / cookie-cutter

An opinionated micro-services framework for TypeScript
https://walmartlabs.github.io/cookie-cutter
Other
58 stars 35 forks source link

After Upgrading azure package to 1.4.0, QueueInputSource failed to read messages written by older (<1.4.0) CC QueueOutputSink with default preprocessor #324

Closed YashGajera closed 1 year ago

YashGajera commented 2 years ago

It appears that the latest 1.4.0 azure package is using @azure/storage-queue lib instead of azure-storage lib used by the older CC version. Apparently, there is an inconsistency between how messages are encoded (by default) and how they are decoded between @azure/storage-queue and azure-storage. Newer library failed to decode message encoded by default TextXmlQueueMessageEncoder with older library.

Error: SyntaxError: Unexpected token & in JSON at position 1 at JSON.parse (<anonymous>) at EnvelopeQueueMessagePreprocessor.process (/Users/yug0003/Documents/store-initialization-checkpoints/services/store-initialization-checkpoints/node_modules/@walmartlabs/cookie-cutter-azure/dist/utils/QueueClient.js:43:21) at /Users/yug0003/Documents/store-initialization-checkpoints/services/store-initialization-checkpoints/node_modules/@walmartlabs/cookie-cutter-azure/dist/utils/QueueClient.js:221:56 at Array.reduce (<anonymous>) at QueueClient.<anonymous> (/Users/yug0003/Documents/store-initialization-checkpoints/services/store-initialization-checkpoints/node_modules/@walmartlabs/cookie-cutter-azure/dist/utils/QueueClient.js:219:57) at Generator.next (<anonymous>) at fulfilled (/Users/yug0003/Documents/store-initialization-checkpoints/services/store-initialization-checkpoints/node_modules/@walmartlabs/cookie-cutter-azure/dist/utils/QueueClient.js:11:58) at runMicrotasks (<anonymous>) at processTicksAndRejections (internal/process/task_queues.js:93:5)

Encoded message written with older CC azure queue sink: {&quot;payload&quot;:{&quot;context&quot;:{&quot;store&quot;:{&quot;tenantId&quot;:&quot;walmart-us&quot;,&quot;storeId&quot;:&quot;3014&quot;},&quot;time&quot;:{&quot;eventTime&quot;:{&quot;seconds&quot;:1645131671,&quot;nanos&quot;:770000000}},&quot;correlation&quot;:{&quot;parentEventIds&quot;:[{}],&quot;api&quot;:{}},&quot;location&quot;:{&quot;point&quot;:{}},&quot;camera&quot;:{}},&quot;laneNumber&quot;:3},&quot;headers&quot;:{&quot;event_type&quot;:&quot;kepler.store.checkpoints.initialization.events.InferenceRunningCompleted&quot;}}