Informatievlaanderen / VSDS-Linked-Data-Interactions

https://informatievlaanderen.github.io/VSDS-Linked-Data-Interactions/
European Union Public License 1.2
4 stars 7 forks source link

When resume the pipeline halted by 404 error, the client doesn't as expected. #630

Closed xdxxxdx closed 2 days ago

xdxxxdx commented 6 months ago

Versions ldes/ldi-orchestrator:2.6.0-SNAPSHOT

Describe the bug Thanks to https://github.com/Informatievlaanderen/VSDS-Linked-Data-Interactions/issues/605, when the client now faces 404 error, the pipeline is halted as expected. But when the source LDES stream is recovered, manually resume the pipeline by API call, the pipeline doesn't continue consuming the LDES stream.

To Reproduce

  1. ldes/ldi-orchestrator:2.6.0-SNAPSHOT is up and running with the following configuration
    
    server:
    port: 8080

orchestrator: pipelines:

spring: codec: max-in-memory-size: 10MB

logging: level: root: INFO Ldio.Application: INFO Ldio:HttpIn: DEBUG Ldio:HttpOut: DEBUG


`http://ldes-server:8080/bearb/timebased-50` is not available for moment => **UPDATED status for pipeline 'bear-b-pipeline-client' to HALTED** as expected
3.`http://ldes-server:8080/bearb/timebased-50` is back online
4. Call API `http://localhost:8082/admin/api/v1/pipeline/bear-b-pipeline-client/resume` to resume the pipeline

**Current behavior**
In the trace of LDIO we can see the pipeline is recovered, but it doesn't restart to consume the LDES stream.
![image](https://github.com/Informatievlaanderen/VSDS-Linked-Data-Interactions/assets/15314807/ac1de5e6-0135-4487-a5b5-d6ade77c7212)

**Expected behavior**
The pipeline is recovered and continue consuming the stream `http://ldes-server:8080/bearb/timebased-50`. 
If resuming the pipeline doesn't' resume the function, there is no need to resume it..
jobulcke commented 6 months ago

A little investigation shows that the task that is responsible for everything in the ExecutorService is considered finished after an exception is thrown, which is not the case when the client is paused manually. So a way to "restart" the process(es) responsible in the client in a clean way needs to be found and implemented