search

terascope / teraslice

Scalable data processing pipelines in JavaScript
https://terascope.github.io/teraslice/
Apache License 2.0
50 stars 13 forks source link

Bug Fix: kubernetesV2 backend doesn't exit properly in the event of a parsing error #3754

Closed sotojn closed 2 weeks ago

sotojn commented 2 months ago

When running in kubernetesV2 with a slicer that throws an error, the pod resource will be marked as completed instead of of an error state.

Here is the execution pod logs in kubernetesV2:

[2024-09-16T21:14:23.251Z] DEBUG: teraslice/10 on ts-exc-kafka-to-es-json-error-74930545-9b47-kt5h7: dispatched slice 1a996a7c-e463-4ad2-9da0-67afd4816b09 to worker 10.244.0.10__tCfeRXYB (assignment=execution_controller, module=execution_controller, worker_id=1sAhzHh2, ex_id=6e6641da-697e-4ec3-bf55-00a988512acd, job_id=74930545-9b47-4f9c-9473-8bd443fb4c34)
[2024-09-16T21:14:44.655Z] ERROR: teraslice/10 on ts-exc-kafka-to-es-json-error-74930545-9b47-kt5h7: execution_controller received an uncaughtException, exiting in 30s... (assignment=execution_controller, module=execution_controller:shutdown_handler, worker_id=1sAhzHh2, ex_id=6e6641da-697e-4ec3-bf55-00a988512acd, job_id=74930545-9b47-4f9c-9473-8bd443fb4c34)
    Error: Unable to parse JSON: Unexpected token n in JSON at position 2
        at Timeout._onTimeout (file:///app/assets/8a13b6aabe2c651a21008d37f487057a754c5648/index.js:22948:15)
        at listOnTimeout (node:internal/timers:569:17)
        at process.processTimers (node:internal/timers:512:7)
[2024-09-16T21:14:44.757Z] DEBUG: teraslice/10 on ts-exc-kafka-to-es-json-error-74930545-9b47-kt5h7: execution shutdown was called for ex 6e6641da-697e-4ec3-bf55-00a988512acd (assignment=execution_controller, module=execution_controller, worker_id=1sAhzHh2, ex_id=6e6641da-697e-4ec3-bf55-00a988512acd, job_id=74930545-9b47-4f9c-9473-8bd443fb4c34)
[2024-09-16T21:14:44.757Z] DEBUG: teraslice/10 on ts-exc-kafka-to-es-json-error-74930545-9b47-kt5h7: stopping scheduler... (assignment=execution_controller, module=execution_scheduler, worker_id=1sAhzHh2, ex_id=6e6641da-697e-4ec3-bf55-00a988512acd, job_id=74930545-9b47-4f9c-9473-8bd443fb4c34)
[2024-09-16T21:14:44.757Z] DEBUG: teraslice/10 on ts-exc-kafka-to-es-json-error-74930545-9b47-kt5h7: execution 6e6641da-697e-4ec3-bf55-00a988512acd is finished scheduling, 3 remaining slices in the queue (assignment=execution_controller, module=execution_scheduler, worker_id=1sAhzHh2, ex_id=6e6641da-697e-4ec3-bf55-00a988512acd, job_id=74930545-9b47-4f9c-9473-8bd443fb4c34)
[2024-09-16T21:14:44.758Z] DEBUG: teraslice/10 on ts-exc-kafka-to-es-json-error-74930545-9b47-kt5h7: shutdown is waiting for execution to finish... (assignment=execution_controller, module=execution_controller, worker_id=1sAhzHh2, ex_id=6e6641da-697e-4ec3-bf55-00a988512acd, job_id=74930545-9b47-4f9c-9473-8bd443fb4c34)
[2024-09-16T21:14:44.760Z] DEBUG: teraslice/10 on ts-exc-kafka-to-es-json-error-74930545-9b47-kt5h7: execution 6e6641da-697e-4ec3-bf55-00a988512acd did not finish (assignment=execution_controller, module=execution_controller, worker_id=1sAhzHh2, ex_id=6e6641da-697e-4ec3-bf55-00a988512acd, job_id=74930545-9b47-4f9c-9473-8bd443fb4c34)
[2024-09-16T21:14:44.760Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-74930545-9b47-kt5h7: [START] "elasticsearch_sender_api" operation shutdown (assignment=execution_controller, module=slicer_context, worker_id=1sAhzHh2, ex_id=6e6641da-697e-4ec3-bf55-00a988512acd, job_id=74930545-9b47-4f9c-9473-8bd443fb4c34)
[2024-09-16T21:14:44.760Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-74930545-9b47-kt5h7: [START] "kafka_reader_api:kafka_reader-0" operation shutdown (assignment=execution_controller, module=slicer_context, worker_id=1sAhzHh2, ex_id=6e6641da-697e-4ec3-bf55-00a988512acd, job_id=74930545-9b47-4f9c-9473-8bd443fb4c34)
[2024-09-16T21:14:44.760Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-74930545-9b47-kt5h7: [START] "kafka_reader" operation shutdown (assignment=execution_controller, module=slicer_context, worker_id=1sAhzHh2, ex_id=6e6641da-697e-4ec3-bf55-00a988512acd, job_id=74930545-9b47-4f9c-9473-8bd443fb4c34)
[2024-09-16T21:14:44.760Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-74930545-9b47-kt5h7: [FINISH] "elasticsearch_sender_api" operation shutdown, took 0ms (assignment=execution_controller, module=slicer_context, worker_id=1sAhzHh2, ex_id=6e6641da-697e-4ec3-bf55-00a988512acd, job_id=74930545-9b47-4f9c-9473-8bd443fb4c34)
[2024-09-16T21:14:44.760Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-74930545-9b47-kt5h7: [FINISH] "kafka_reader" operation shutdown, took 0ms (assignment=execution_controller, module=slicer_context, worker_id=1sAhzHh2, ex_id=6e6641da-697e-4ec3-bf55-00a988512acd, job_id=74930545-9b47-4f9c-9473-8bd443fb4c34)
[2024-09-16T21:14:44.760Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-74930545-9b47-kt5h7: [FINISH] "kafka_reader_api:kafka_reader-0" operation shutdown, took 0ms (assignment=execution_controller, module=slicer_context, worker_id=1sAhzHh2, ex_id=6e6641da-697e-4ec3-bf55-00a988512acd, job_id=74930545-9b47-4f9c-9473-8bd443fb4c34)
[2024-09-16T21:14:44.761Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-74930545-9b47-kt5h7: calculating statistics (assignment=execution_controller, module=slice_analytics, worker_id=1sAhzHh2, ex_id=6e6641da-697e-4ec3-bf55-00a988512acd, job_id=74930545-9b47-4f9c-9473-8bd443fb4c34)
[2024-09-16T21:14:44.761Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-74930545-9b47-kt5h7: (assignment=execution_controller, module=slice_analytics, worker_id=1sAhzHh2, ex_id=6e6641da-697e-4ec3-bf55-00a988512acd, job_id=74930545-9b47-4f9c-9473-8bd443fb4c34)

    operation kafka_reader
    average completion time of: 30053 ms, min: 30053 ms, and max: 30053 ms
    average size: 0, min: 0, and max: 0
    average memory: -11063472, min: -11063472, and max: -11063472

[2024-09-16T21:14:44.761Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-74930545-9b47-kt5h7: (assignment=execution_controller, module=slice_analytics, worker_id=1sAhzHh2, ex_id=6e6641da-697e-4ec3-bf55-00a988512acd, job_id=74930545-9b47-4f9c-9473-8bd443fb4c34)

    operation elasticsearch_bulk
    average completion time of: 0 ms, min: 0 ms, and max: 0 ms
    average size: 0, min: 0, and max: 0
    average memory: 1688, min: 1688, and max: 1688

[2024-09-16T21:14:44.761Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-74930545-9b47-kt5h7: execution 6e6641da-697e-4ec3-bf55-00a988512acd has finished in 60 seconds (assignment=execution_controller, module=execution_controller, worker_id=1sAhzHh2, ex_id=6e6641da-697e-4ec3-bf55-00a988512acd, job_id=74930545-9b47-4f9c-9473-8bd443fb4c34)
[2024-09-16T21:14:44.783Z] ERROR: teraslice/10 on ts-exc-kafka-to-es-json-error-74930545-9b47-kt5h7: execution 6e6641da-697e-4ec3-bf55-00a988512acd received shutdown before the slicer could complete, setting status to "terminated" (assignment=execution_controller, module=execution_controller, worker_id=1sAhzHh2, ex_id=6e6641da-697e-4ec3-bf55-00a988512acd, job_id=74930545-9b47-4f9c-9473-8bd443fb4c34)
[2024-09-16T21:14:44.859Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-74930545-9b47-kt5h7: shutting down (assignment=execution_controller, module=state_storage, worker_id=1sAhzHh2, ex_id=6e6641da-697e-4ec3-bf55-00a988512acd, job_id=74930545-9b47-4f9c-9473-8bd443fb4c34)
[2024-09-16T21:14:44.859Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-74930545-9b47-kt5h7: shutting down. (assignment=execution_controller, module=ex_storage, worker_id=1sAhzHh2, ex_id=6e6641da-697e-4ec3-bf55-00a988512acd, job_id=74930545-9b47-4f9c-9473-8bd443fb4c34)
[2024-09-16T21:14:44.861Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-74930545-9b47-kt5h7: client 10.244.0.10__tCfeRXYB disconnected { reason: 'client namespace disconnect' } (assignment=execution_controller, module=messaging:server, worker_id=1sAhzHh2, ex_id=6e6641da-697e-4ec3-bf55-00a988512acd, job_id=74930545-9b47-4f9c-9473-8bd443fb4c34)
[2024-09-16T21:14:44.862Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-74930545-9b47-kt5h7: client 6e6641da-697e-4ec3-bf55-00a988512acd disconnected { reason: 'io client disconnect' } (assignment=execution_controller, module=messaging:client, worker_id=1sAhzHh2, ex_id=6e6641da-697e-4ec3-bf55-00a988512acd, job_id=74930545-9b47-4f9c-9473-8bd443fb4c34)
[2024-09-16T21:14:44.905Z] DEBUG: teraslice/10 on ts-exc-kafka-to-es-json-error-74930545-9b47-kt5h7: execution 6e6641da-697e-4ec3-bf55-00a988512acd is done (assignment=execution_controller, module=execution_controller, worker_id=1sAhzHh2, ex_id=6e6641da-697e-4ec3-bf55-00a988512acd, job_id=74930545-9b47-4f9c-9473-8bd443fb4c34)
[2024-09-16T21:14:49.866Z]  WARN: teraslice/10 on ts-exc-kafka-to-es-json-error-74930545-9b47-kt5h7: execution controller 6e6641da-697e-4ec3-bf55-00a988512acd is shutdown (assignment=execution_controller, module=execution_controller, worker_id=1sAhzHh2, ex_id=6e6641da-697e-4ec3-bf55-00a988512acd, job_id=74930545-9b47-4f9c-9473-8bd443fb4c34)
[2024-09-16T21:14:49.867Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-74930545-9b47-kt5h7: execution_controller shutdown, already shutting down, remaining 25s (assignment=execution_controller, module=execution_controller:shutdown_handler, worker_id=1sAhzHh2, ex_id=6e6641da-697e-4ec3-bf55-00a988512acd, job_id=74930545-9b47-4f9c-9473-8bd443fb4c34)
[2024-09-16T21:14:50.870Z] DEBUG: teraslice/10 on ts-exc-kafka-to-es-json-error-74930545-9b47-kt5h7: flushed logs successfully, will exit with code 1 (assignment=execution_controller, module=execution_controller:shutdown_handler, worker_id=1sAhzHh2, ex_id=6e6641da-697e-4ec3-bf55-00a988512acd, job_id=74930545-9b47-4f9c-9473-8bd443fb4c34)
[2024-09-16T21:14:50.871Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-74930545-9b47-kt5h7: execution_controller shutdown took 6s, exit with zero status code (assignment=execution_controller, module=execution_controller:shutdown_handler, worker_id=1sAhzHh2, ex_id=6e6641da-697e-4ec3-bf55-00a988512acd, job_id=74930545-9b47-4f9c-9473-8bd443fb4c34)

Here is the execution pod logs in kubernetes mode (same job):

[2024-09-16T21:28:49.686Z] ERROR: teraslice/10 on ts-exc-kafka-to-es-json-error-35ca55ff-8ead-b4ljx: execution_controller received an uncaughtException, exiting in 30s... (assignment=execution_controller, module=execution_controller:shutdown_handler, worker_id=tJj80yMh, ex_id=37750760-932c-4c2f-9a66-61b17f80f9f3, job_id=35ca55ff-8ead-4f2d-a205-08b4c18cd4de)
    Error: Unable to parse JSON: Unexpected token n in JSON at position 2
        at Timeout._onTimeout (file:///app/assets/8a13b6aabe2c651a21008d37f487057a754c5648/index.js:22948:15)
        at listOnTimeout (node:internal/timers:569:17)
        at process.processTimers (node:internal/timers:512:7)
[2024-09-16T21:28:49.787Z] DEBUG: teraslice/10 on ts-exc-kafka-to-es-json-error-35ca55ff-8ead-b4ljx: execution shutdown was called for ex 37750760-932c-4c2f-9a66-61b17f80f9f3 (assignment=execution_controller, module=execution_controller, worker_id=tJj80yMh, ex_id=37750760-932c-4c2f-9a66-61b17f80f9f3, job_id=35ca55ff-8ead-4f2d-a205-08b4c18cd4de)
[2024-09-16T21:28:49.787Z] DEBUG: teraslice/10 on ts-exc-kafka-to-es-json-error-35ca55ff-8ead-b4ljx: stopping scheduler... (assignment=execution_controller, module=execution_scheduler, worker_id=tJj80yMh, ex_id=37750760-932c-4c2f-9a66-61b17f80f9f3, job_id=35ca55ff-8ead-4f2d-a205-08b4c18cd4de)
[2024-09-16T21:28:49.787Z] DEBUG: teraslice/10 on ts-exc-kafka-to-es-json-error-35ca55ff-8ead-b4ljx: execution 37750760-932c-4c2f-9a66-61b17f80f9f3 is finished scheduling, 3 remaining slices in the queue (assignment=execution_controller, module=execution_scheduler, worker_id=tJj80yMh, ex_id=37750760-932c-4c2f-9a66-61b17f80f9f3, job_id=35ca55ff-8ead-4f2d-a205-08b4c18cd4de)
[2024-09-16T21:28:49.788Z] DEBUG: teraslice/10 on ts-exc-kafka-to-es-json-error-35ca55ff-8ead-b4ljx: shutdown is waiting for execution to finish... (assignment=execution_controller, module=execution_controller, worker_id=tJj80yMh, ex_id=37750760-932c-4c2f-9a66-61b17f80f9f3, job_id=35ca55ff-8ead-4f2d-a205-08b4c18cd4de)
[2024-09-16T21:28:49.791Z] DEBUG: teraslice/10 on ts-exc-kafka-to-es-json-error-35ca55ff-8ead-b4ljx: execution 37750760-932c-4c2f-9a66-61b17f80f9f3 did not finish (assignment=execution_controller, module=execution_controller, worker_id=tJj80yMh, ex_id=37750760-932c-4c2f-9a66-61b17f80f9f3, job_id=35ca55ff-8ead-4f2d-a205-08b4c18cd4de)
[2024-09-16T21:28:49.791Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-35ca55ff-8ead-b4ljx: [START] "elasticsearch_sender_api" operation shutdown (assignment=execution_controller, module=slicer_context, worker_id=tJj80yMh, ex_id=37750760-932c-4c2f-9a66-61b17f80f9f3, job_id=35ca55ff-8ead-4f2d-a205-08b4c18cd4de)
[2024-09-16T21:28:49.792Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-35ca55ff-8ead-b4ljx: [START] "kafka_reader_api:kafka_reader-0" operation shutdown (assignment=execution_controller, module=slicer_context, worker_id=tJj80yMh, ex_id=37750760-932c-4c2f-9a66-61b17f80f9f3, job_id=35ca55ff-8ead-4f2d-a205-08b4c18cd4de)
[2024-09-16T21:28:49.792Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-35ca55ff-8ead-b4ljx: [START] "kafka_reader" operation shutdown (assignment=execution_controller, module=slicer_context, worker_id=tJj80yMh, ex_id=37750760-932c-4c2f-9a66-61b17f80f9f3, job_id=35ca55ff-8ead-4f2d-a205-08b4c18cd4de)
[2024-09-16T21:28:49.792Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-35ca55ff-8ead-b4ljx: [FINISH] "elasticsearch_sender_api" operation shutdown, took 1ms (assignment=execution_controller, module=slicer_context, worker_id=tJj80yMh, ex_id=37750760-932c-4c2f-9a66-61b17f80f9f3, job_id=35ca55ff-8ead-4f2d-a205-08b4c18cd4de)
[2024-09-16T21:28:49.792Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-35ca55ff-8ead-b4ljx: [FINISH] "kafka_reader" operation shutdown, took 0ms (assignment=execution_controller, module=slicer_context, worker_id=tJj80yMh, ex_id=37750760-932c-4c2f-9a66-61b17f80f9f3, job_id=35ca55ff-8ead-4f2d-a205-08b4c18cd4de)
[2024-09-16T21:28:49.792Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-35ca55ff-8ead-b4ljx: [FINISH] "kafka_reader_api:kafka_reader-0" operation shutdown, took 0ms (assignment=execution_controller, module=slicer_context, worker_id=tJj80yMh, ex_id=37750760-932c-4c2f-9a66-61b17f80f9f3, job_id=35ca55ff-8ead-4f2d-a205-08b4c18cd4de)
[2024-09-16T21:28:49.792Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-35ca55ff-8ead-b4ljx: calculating statistics (assignment=execution_controller, module=slice_analytics, worker_id=tJj80yMh, ex_id=37750760-932c-4c2f-9a66-61b17f80f9f3, job_id=35ca55ff-8ead-4f2d-a205-08b4c18cd4de)
[2024-09-16T21:28:49.792Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-35ca55ff-8ead-b4ljx: (assignment=execution_controller, module=slice_analytics, worker_id=tJj80yMh, ex_id=37750760-932c-4c2f-9a66-61b17f80f9f3, job_id=35ca55ff-8ead-4f2d-a205-08b4c18cd4de)

    operation kafka_reader
    average completion time of: 30058 ms, min: 30058 ms, and max: 30058 ms
    average size: 0, min: 0, and max: 0
    average memory: -4701232, min: -4701232, and max: -4701232

[2024-09-16T21:28:49.792Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-35ca55ff-8ead-b4ljx: (assignment=execution_controller, module=slice_analytics, worker_id=tJj80yMh, ex_id=37750760-932c-4c2f-9a66-61b17f80f9f3, job_id=35ca55ff-8ead-4f2d-a205-08b4c18cd4de)

    operation elasticsearch_bulk
    average completion time of: 0 ms, min: 0 ms, and max: 0 ms
    average size: 0, min: 0, and max: 0
    average memory: 1688, min: 1688, and max: 1688

[2024-09-16T21:28:49.792Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-35ca55ff-8ead-b4ljx: execution 37750760-932c-4c2f-9a66-61b17f80f9f3 has finished in 60 seconds (assignment=execution_controller, module=execution_controller, worker_id=tJj80yMh, ex_id=37750760-932c-4c2f-9a66-61b17f80f9f3, job_id=35ca55ff-8ead-4f2d-a205-08b4c18cd4de)
[2024-09-16T21:28:49.811Z] ERROR: teraslice/10 on ts-exc-kafka-to-es-json-error-35ca55ff-8ead-b4ljx: execution 37750760-932c-4c2f-9a66-61b17f80f9f3 received shutdown before the slicer could complete, setting status to "terminated" (assignment=execution_controller, module=execution_controller, worker_id=tJj80yMh, ex_id=37750760-932c-4c2f-9a66-61b17f80f9f3, job_id=35ca55ff-8ead-4f2d-a205-08b4c18cd4de)
[2024-09-16T21:28:49.890Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-35ca55ff-8ead-b4ljx: shutting down (assignment=execution_controller, module=state_storage, worker_id=tJj80yMh, ex_id=37750760-932c-4c2f-9a66-61b17f80f9f3, job_id=35ca55ff-8ead-4f2d-a205-08b4c18cd4de)
[2024-09-16T21:28:49.890Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-35ca55ff-8ead-b4ljx: shutting down. (assignment=execution_controller, module=ex_storage, worker_id=tJj80yMh, ex_id=37750760-932c-4c2f-9a66-61b17f80f9f3, job_id=35ca55ff-8ead-4f2d-a205-08b4c18cd4de)
[2024-09-16T21:28:49.892Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-35ca55ff-8ead-b4ljx: client 10.244.0.10__5Vb12gBA disconnected { reason: 'client namespace disconnect' } (assignment=execution_controller, module=messaging:server, worker_id=tJj80yMh, ex_id=37750760-932c-4c2f-9a66-61b17f80f9f3, job_id=35ca55ff-8ead-4f2d-a205-08b4c18cd4de)
[2024-09-16T21:28:49.893Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-35ca55ff-8ead-b4ljx: client 37750760-932c-4c2f-9a66-61b17f80f9f3 disconnected { reason: 'io client disconnect' } (assignment=execution_controller, module=messaging:client, worker_id=tJj80yMh, ex_id=37750760-932c-4c2f-9a66-61b17f80f9f3, job_id=35ca55ff-8ead-4f2d-a205-08b4c18cd4de)
[2024-09-16T21:28:49.932Z] DEBUG: teraslice/10 on ts-exc-kafka-to-es-json-error-35ca55ff-8ead-b4ljx: execution 37750760-932c-4c2f-9a66-61b17f80f9f3 is done (assignment=execution_controller, module=execution_controller, worker_id=tJj80yMh, ex_id=37750760-932c-4c2f-9a66-61b17f80f9f3, job_id=35ca55ff-8ead-4f2d-a205-08b4c18cd4de)
[2024-09-16T21:28:54.892Z]  WARN: teraslice/10 on ts-exc-kafka-to-es-json-error-35ca55ff-8ead-b4ljx: execution controller 37750760-932c-4c2f-9a66-61b17f80f9f3 is shutdown (assignment=execution_controller, module=execution_controller, worker_id=tJj80yMh, ex_id=37750760-932c-4c2f-9a66-61b17f80f9f3, job_id=35ca55ff-8ead-4f2d-a205-08b4c18cd4de)
[2024-09-16T21:28:54.892Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-35ca55ff-8ead-b4ljx: execution_controller shutdown, already shutting down, remaining 25s (assignment=execution_controller, module=execution_controller:shutdown_handler, worker_id=tJj80yMh, ex_id=37750760-932c-4c2f-9a66-61b17f80f9f3, job_id=35ca55ff-8ead-4f2d-a205-08b4c18cd4de)
[2024-09-16T21:28:55.894Z] DEBUG: teraslice/10 on ts-exc-kafka-to-es-json-error-35ca55ff-8ead-b4ljx: flushed logs successfully, will exit with code 1 (assignment=execution_controller, module=execution_controller:shutdown_handler, worker_id=tJj80yMh, ex_id=37750760-932c-4c2f-9a66-61b17f80f9f3, job_id=35ca55ff-8ead-4f2d-a205-08b4c18cd4de)
[2024-09-16T21:28:55.894Z]  INFO: teraslice/10 on ts-exc-kafka-to-es-json-error-35ca55ff-8ead-b4ljx: execution_controller shutdown took 6s, exit with 1 status code (assignment=execution_controller, module=execution_controller:shutdown_handler, worker_id=tJj80yMh, ex_id=37750760-932c-4c2f-9a66-61b17f80f9f3, job_id=35ca55ff-8ead-4f2d-a205-08b4c18cd4de)

I need to find out why it's exiting with a zero status code when this happens and ensure I fail with a status code of 1.

godber commented 1 month ago

What are the steps to reproduce?

What is the status of the Teraslice job after one of these failures?

I think this is probably pretty important to resolve.

sotojn commented 2 weeks ago

This may have been resolved in a prior PR. #3756 I'll confirm this by running a job with an intentional parsing error. I'm expecting the process to close with an exit code of 1 and be an error.

sotojn commented 2 weeks ago

I also found this, I believe these issues are the same: https://github.com/terascope/teraslice/issues/3755

sotojn commented 2 weeks ago

This is resolved