Submission UI fails to recognize compilation failure and enters endless 'processing underway' loop.

[DavidLFielding]

Jim originally reported a problem with a malformed compilation summary for a know bad submission. The compilation summary was basically blank (no indication of TeX runs).

I have not been able to reproduce Jim's results. When I submit the problem source the compilation step enters an endless loop and never returns or times out.

I'm able to see a compilation error in the logs but this does not seem to propagate back to the submission UI properly.

At a minimum, the submission UI should time out on the compilation process. My compilation at develpopment.arxiv.org has been running for 15 minutes.

In my opinion, there should be a reasonable timeout on all calls to external services. When a timeout occurs appropriate action should be taken (retry, report error and give up).

Something else is going on here, likely related to exception handling in the compilation service. More testing should be performed on exception cases.

The issue demonstrating failed exception handling may be repeated in other services that follow the same communications model.

The two files are cesar.tex and cesar.bbl (uploaded to Slack by Jim)

[erickpeirson]

Getting close to a final pass on this. Two more PRs open:

• https://github.com/arXiv/arxiv-compiler/pull/25
• https://github.com/arXiv/arxiv-compiler/pull/26

Final steps to resolve:

• Add ability to force-recompile to the UI (already supported in compiler service)
• Add timeouts to compilation: https://github.com/arXiv/arxiv-compiler/issues/27

[erickpeirson]

Briefly looked into task cancellation. This is not a terribly straightforward thing to do. Celery does have revocation, which will drop the task before it is picked up by a worker. Revocation can also generate a kill/term signal, but this will take the whole worker down (and may impact other tasks). Let's start with timeouts and recompilation for now.