jenkinsci / jenkinsfile-runner

A command line tool to run Jenkinsfile as a function
MIT License
1.14k stars 296 forks source link

Add --write-log-init-timeout option to allow specifying a timeout to retry writing logs to stdout #698

Closed romanisb closed 1 year ago

romanisb commented 1 year ago

This contribution proposes a new PipelineRunOptions which provides an option to set a specific timeout within the runner is periodically trying to initialize the log output stream.

This option will replace the hardcoded retrycount with a more dynamical approach.

Without increasing the duration of this retry, we occasionally ran into issues as shown below. Especially in a Kubernetes environment in which the nodes is under high utilization.

Cloning into '.'...
Checking out pipeline from revision master
Your branch is up to date with 'origin/master'.
Already on 'master'
Delete pipeline git clone credentials
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.thoughtworks.xstream.core.util.Fields (file:/app/repo/org/jvnet/hudson/xstream/1.4.7-jenkins-1/xstream-1.4.7-jenkins-1.jar) to field java.util.TreeMap.comparator
WARNING: Please consider reporting this to the maintainers of com.thoughtworks.xstream.core.util.Fields
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2021-01-15 07:50:03.002+0000 [id=1] SEVERE  h.i.i.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler#uncaughtException: A thread (main/1) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code.
java.io.FileNotFoundException: /jenkins_home/jobs/job/builds/1/log (No such file or directory)
    at java.base/java.io.RandomAccessFile.open0(Native Method)
    at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:345)
    at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:259)
    at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:214)
    at org.kohsuke.stapler.framework.io.LargeText$FileSession.<init>(LargeText.java:399)
    at org.kohsuke.stapler.framework.io.LargeText$2.open(LargeText.java:121)
    at org.kohsuke.stapler.framework.io.LargeText.writeLogTo(LargeText.java:212)
    at hudson.console.AnnotatedLargeText.writeLogTo(AnnotatedLargeText.java:165)
    at hudson.model.Run.writeWholeLogTo(Run.java:1556)
    at io.jenkins.jenkinsfile.runner.Runner.writeLogTo(Runner.java:134)
    at io.jenkins.jenkinsfile.runner.Runner.run(Runner.java:94)
Caused: java.lang.reflect.InvocationTargetException
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at io.jenkins.jenkinsfile.runner.JenkinsfileRunnerLauncher.doLaunch(JenkinsfileRunnerLauncher.java:32)
    at io.jenkins.jenkinsfile.runner.JenkinsLauncher.launch(JenkinsLauncher.java:83)
    at io.jenkins.jenkinsfile.runner.App.run(App.java:13)
    at io.jenkins.jenkinsfile.runner.bootstrap.Bootstrap.run(Bootstrap.java:312)
    at io.jenkins.jenkinsfile.runner.bootstrap.Bootstrap.main(Bootstrap.java:135)

Testing done

### Submitter checklist
- [x] Make sure you are opening from a **topic/feature/bugfix branch** (right side) and not your main branch!
- [x] Ensure that the pull request title represents the desired changelog entry
- [x]  Please describe what you did
- [ ] Link to relevant issues in GitHub or Jira
- [ ] Link to relevant pull requests, esp. upstream and downstream changes
- [ ] Ensure you have provided tests - that demonstrates feature works or fixes the issue