CLI times out on long Ingestor transactions

[tangobravo62]

Ingesting products may take a long time. After some time the CLI returns with a time out message:

prosEO (S2)> ingest --file=../deployment/processing-facility/testfiles/ingest_l1c_products.json telekom-otc
(E2806) Command failed (cause: (E2810) HTTP request failed (cause: 504 Gateway Time-out); nested exception is org.springframework.web.client.HttpServerErrorException$GatewayTimeout: 504 Gateway Time-out)

The Ingestor process is still running correctly and is ingesting products as requested (note that the request entailed coying of 6.8 GB of data at the processing facility):

2020-06-23 15:58:42.442 TRACE 1 --- [nio-8080-exec-2] d.d.p.i.rest.IngestControllerImpl        : >>> ingestProducts(telekom-otc, IngestorProduct[21])
[...]
2020-06-23 16:02:34.517  INFO 1 --- [nio-8080-exec-2] d.d.p.i.rest.IngestControllerImpl        : (I2058) 21 products ingested in processing facility telekom-otc

Improved behaviour: Do not time out, but report on progress.

[tangobravo62]

Timeouts may also arise in the whole chain CLI -> Ingestor -> Storage Manager -> Object Storage, e. g. when ingesting large product files (several 100 MB to several GB).

A protocol needs to be defined to communicate between clients and servers along the chain. All potentially long-running HTTP requests shall be changed to using the reactive Spring WebClient instead of the old (synchronous) RestTemplate.

[tangobravo62]

Solved: All timeouts now configurable (including CLI), note that in all nginx instances the parameter proxy_read_timeout needs to be set.

Transcript of long-running transactions: (1) CLI commands (ingest --file=... in file cli_commands.txt):

[root@esaitt-qualification transfer]# java -jar proseo-ui-cli.jar -uproseo -psb24ZxoBxR6y5bVm -mS1 <cli_commands.txt
[...]
(I2821) User proseo logged in
prosEO (S1)> (I2959) 1 products ingested to processing facility telekom-otc
prosEO (S1)> (I2959) 1 products ingested to processing facility telekom-otc
prosEO (S1)> (I2959) 1 products ingested to processing facility telekom-otc
prosEO (S1)> (I2959) 1 products ingested to processing facility telekom-otc

Ingestor log excerpt showing 3-4 minutes needed for requests without timeouts occurring:

2020-07-07 14:11:39.460 DEBUG 1 --- [nio-8080-exec-3] org.apache.http.wire                     : http-outgoing-28 >> "{"targetStorageType":"S3","sourceFilePaths":["/mnt/S1_out/S1B_IW_RAW__0SDV_20200318T053427_20200318T053459_020742_027539_B799.ISIP.zip"],"productId":"5066","sourceStorageType":"POSIX"}"
2020-07-07 14:15:57.427 DEBUG 1 --- [nio-8080-exec-3] org.apache.http.wire                     : http-outgoing-28 << "HTTP/1.1 201 Created[\r][\n]"
[...]
2020-07-07 14:15:57.780 DEBUG 1 --- [nio-8080-exec-4] org.apache.http.wire                     : http-outgoing-29 >> "{"targetStorageType":"S3","sourceFilePaths":["/mnt/S1_out/S1B_IW_RAW__0SDV_20200318T053452_20200318T053524_020742_027539_19CD.ISIP.zip"],"productId":"5068","sourceStorageType":"POSIX"}"
2020-07-07 14:18:50.455 DEBUG 1 --- [nio-8080-exec-4] org.apache.http.wire                     : http-outgoing-29 << "HTTP/1.1 201 Created[\r][\n]"

Request to create protocol for asynchronous communication dropped for the moment (may be future improvement).