search

elasticio / csv-component

CSV Reading and Writing component for elastic.io platform
Apache License 2.0
3 stars 7 forks source link

How about to emit "error" in case of issue with attachments API? #19

Closed yarikos closed 4 years ago

yarikos commented 5 years ago

There is a flow: CSV(read)-CSV(write)-SFTP We had a situation with csv(write), when step looks like successful executed - with green number of messages, but an error was in the logs:

[2018-11-16 14:10:11.644]: Starting slug download ...
[2018-11-16 14:10:13.342]: Successfully downloaded and extracted slug file
[2018-11-16 14:10:13.386]: Writing config vars to .profile.d/config_vars
[2018-11-16 14:10:13.413]: Detected 256 MB available memory
[2018-11-16 14:10:13.413]: Detected 0.1 CPU available
[2018-11-16 14:10:13.418]: Starting the application with node ./node_modules/elasticio-sailor-nodejs/run.js
[2018-11-16 14:10:15.049]: {"name":"sailor","hostname":"eio.6bb4afcc8f03d487efd68b80.5a32882919294f00197552d2.stepz7r48","pid":7,"ELASTICIO_TASK_ID":"5a32882919294f00197552d2","ELASTICIO_EXEC_ID":"17e591acf21e4eb09a1ac0b448715ec0","ELASTICIO_STEP_ID":"step_6","ELASTICIO_COMP_ID":"563887ddf2287f060000004c","ELASTICIO_FUNCTION":"write_attachment","tag":"5a32882919294f00197552d2","level":30,"msg":"About to initialize component for execution","time":"2018-11-16T12:10:15.047Z","v":0}
[2018-11-16 14:10:15.201]: Using delimiter: ','
[2018-11-16 14:10:15.201]: Configured column names: [ 'message', 'time' ]
[2018-11-16 14:10:15.339]: CSV file to be uploaded file to uri=http://steward-service.platform.svc.cluster.local:8200/files/57356501-c76a-4a1c-baf4-6e26de74a4ac
[2018-11-16 14:10:15.341]: {"name":"sailor","hostname":"eio.6bb4afcc8f03d487efd68b80.5a32882919294f00197552d2.stepz7r48","pid":7,"ELASTICIO_TASK_ID":"5a32882919294f00197552d2","ELASTICIO_EXEC_ID":"17e591acf21e4eb09a1ac0b448715ec0","ELASTICIO_STEP_ID":"step_6","ELASTICIO_COMP_ID":"563887ddf2287f060000004c","ELASTICIO_FUNCTION":"write_attachment","tag":"5a32882919294f00197552d2","level":30,"msg":"Component execution initialized successfully","time":"2018-11-16T12:10:15.341Z","v":0}
[2018-11-16 14:10:15.351]: {"name":"sailor","hostname":"eio.6bb4afcc8f03d487efd68b80.5a32882919294f00197552d2.stepz7r48","pid":7,"ELASTICIO_TASK_ID":"5a32882919294f00197552d2","ELASTICIO_EXEC_ID":"17e591acf21e4eb09a1ac0b448715ec0","ELASTICIO_STEP_ID":"step_6","ELASTICIO_COMP_ID":"563887ddf2287f060000004c","ELASTICIO_FUNCTION":"write_attachment","tag":"5a32882919294f00197552d2","level":30,"deliveryTag":1,"messageId":"ed233375-2c2d-4b2d-a717-0a2d72e2389e","parentMessageId":"ed233375-2c2d-4b2d-a717-0a2d72e2389e","traceId":"aa12bba0-5cb1-48e8-83f4-7428dc802a30","messagesCount":1,"messageProcessingTime":0,"msg":"processMessage received","time":"2018-11-16T12:10:15.350Z","v":0}
[2018-11-16 14:10:15.353]: Writing Row: {"message":"hello world60","time":"2018-11-16T12:08:36.477Z"}
[2018-11-16 14:10:15.353]: Incoming data: {"time":"2018-11-16T12:08:36.477Z","message":"hello world60"}
[2018-11-16 14:10:15.356]: {"name":"sailor","hostname":"eio.6bb4afcc8f03d487efd68b80.5a32882919294f00197552d2.stepz7r48","pid":7,"ELASTICIO_TASK_ID":"5a32882919294f00197552d2","ELASTICIO_EXEC_ID":"17e591acf21e4eb09a1ac0b448715ec0","ELASTICIO_STEP_ID":"step_6","ELASTICIO_COMP_ID":"563887ddf2287f060000004c","ELASTICIO_FUNCTION":"write_attachment","tag":"5a32882919294f00197552d2","level":30,"deliveryTag":1,"messageId":"ed233375-2c2d-4b2d-a717-0a2d72e2389e","parentMessageId":"ed233375-2c2d-4b2d-a717-0a2d72e2389e","traceId":"aa12bba0-5cb1-48e8-83f4-7428dc802a30","messagesCount":0,"errorCount":0,"promises":0,"messageProcessingTime":5,"msg":"processMessage emit end","time":"2018-11-16T12:10:15.355Z","v":0}
[2018-11-16 14:10:23.491]: at IncomingMessage.<anonymous> (/app/node_modules/request/request.js:1085:12)
[2018-11-16 14:10:23.491]: at Object.onceWrapper (events.js:314:30)
[2018-11-16 14:10:23.491]: at endReadableNT (_stream_readable.js:1045:12)
[2018-11-16 14:10:23.491]: at Request.self.callback (/app/node_modules/request/request.js:186:22)
[2018-11-16 14:10:23.491]: at Request.emit (events.js:213:7)
[2018-11-16 14:10:23.491]: at Request.<anonymous> (/app/node_modules/request/request.js:1163:10)
[2018-11-16 14:10:23.491]: at emitOne (events.js:115:13)
[2018-11-16 14:10:23.491]: at _combinedTickCallback (internal/process/next_tick.js:102:11)
[2018-11-16 14:10:23.491]: at process._tickCallback (internal/process/next_tick.js:161:9)
**[2018-11-16 14:10:23.491]: Unhandled rejection StatusCodeError: 500 - "Internal Server Error"**
[2018-11-16 14:10:23.491]: at new StatusCodeError (/app/node_modules/request-promise-core/lib/errors.js:32:15)
[2018-11-16 14:10:23.491]: at Request.plumbing.callback (/app/node_modules/request-promise-core/lib/plumbing.js:104:33)
[2018-11-16 14:10:23.491]: at Request.RP$callback [as _callback] (/app/node_modules/request-promise-core/lib/plumbing.js:46:31)
[2018-11-16 14:10:23.491]: at emitTwo (events.js:125:13)
[2018-11-16 14:10:23.491]: at Request.emit (events.js:210:7)
[2018-11-16 14:10:23.491]: at emitNone (events.js:110:20)
[2018-11-16 14:10:23.491]: at IncomingMessage.emit (events.js:207:7)
[2018-11-16 14:10:25.357]: Closing the stream due to inactivity
[2018-11-16 14:10:25.357]: The resulting CSV file contains 1 rows
[2018-11-16 14:10:25.358]: Emitting message {"id":"997347d0-e998-11e8-a704-21ae36f35c64","attachments":{"997347d0-e998-11e8-a704-21ae36f35c64.csv":{"content-type":"text/csv","url":"http://steward-service.platform.svc.cluster.local:8200/files/57356501-c76a-4a1c-baf4-6e26de74a4ac"}},"body":{"rowCount":1},"headers":{},"metadata":{}}
[2018-11-16 14:10:25.358]: {"name":"sailor","hostname":"eio.6bb4afcc8f03d487efd68b80.5a32882919294f00197552d2.stepz7r48","pid":7,"ELASTICIO_TASK_ID":"5a32882919294f00197552d2","ELASTICIO_EXEC_ID":"17e591acf21e4eb09a1ac0b448715ec0","ELASTICIO_STEP_ID":"step_6","ELASTICIO_COMP_ID":"563887ddf2287f060000004c","ELASTICIO_FUNCTION":"write_attachment","tag":"5a32882919294f00197552d2","level":30,"deliveryTag":1,"messageId":"ed233375-2c2d-4b2d-a717-0a2d72e2389e","parentMessageId":"ed233375-2c2d-4b2d-a717-0a2d72e2389e","traceId":"aa12bba0-5cb1-48e8-83f4-7428dc802a30","messagesCount":0,"messageProcessingTime":10007,"msg":"processMessage emit data","time":"2018-11-16T12:10:25.357Z","v":0}
[2018-11-16 14:10:25.362]: Configured column names: [ 'message', 'time' ]
[2018-11-16 14:10:25.362]: Using delimiter: ','
[2018-11-16 14:10:25.515]: CSV file to be uploaded file to uri=http://steward-service.platform.svc.cluster.local:8200/files/5c5b17c3-2a97-4d71-afe9-e1724e631d59
[2018-11-16 14:11:12.375]: Received SIGTERM
[2018-11-16 14:11:12.376]: Disconnecting
[2018-11-16 14:11:12.378]: Successfully disconnected
[2018-11-16 14:11:12.386]: [INFO  tini (1)] Main child exited normally (with status '0')
[2018-11-16 14:11:12.386]: [INFO  tini (1)] Spawned child process '/runner/init' with pid '7'

Component must report errors

A3a3e1 commented 4 years ago

This task helps with the issue: https://github.com/elasticio/csv-component/issues/41