Capability to manually create spans

[xBis7]

opentelemetry-plugin issue: https://github.com/jenkinsci/opentelemetry-plugin/issues/90

This PR is based on the work done on PR https://github.com/jenkinsci/opentelemetry-plugin/pull/84.

The patch is adding a new step withNewSpan which allows users to create their own spans.

The option accepts 3 parameters

• label
  • name of the span, string value
• attributes
  • list of attributes, same as withSpanAttributes
• setAttributesOnlyOnParent
  • whether to inherit the defined attributes to children spans or not, boolean value
  • all defined attributes are inherited by default

From the above 3 parameters, only the label is required and everything else is optional.

Example usage

withNewSpan(label: 'custom-build-span', attributes: ([
  spanAttribute(key: 'modules', value: '2'),
  spanAttribute(key: 'command', value: 'mvn clean install')
]), setAttributesOnlyOnParent: true) {
  sh './build-module1.sh'
  sh './build-module2.sh'
}

or

withNewSpan(label: 'custom-build-span', attributes: ([
  spanAttribute(key: 'modules', value: '2'),
  spanAttribute(key: 'command', value: 'mvn clean install')
])) {
  sh './build-module1.sh'
  sh './build-module2.sh'
}

or

withNewSpan(label: 'custom-build-span') {
  sh './build-module1.sh'
  sh './build-module2.sh'
}

• The new step was named withNewSpan to be consistent with the naming of other steps like withSpanAttributes
• For the step execution, SpanAttributeStepExecution was reused
• Added a parameter so that the user can control whether the defined attributes should be inherited on children spans of not

Testing done

• An integration test was added for testing the overall usage with different pipeline scripts
• A unit test was added for testing the changes in MonitorPipelineListener
  • It's also testing the code logic for not inheriting the new span attributes to children spans
• The patch was also tested manually with jenkins and jaeger

Screenshot from manual testing

Submitter checklist

• [X] Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
• [X] Ensure that the pull request title represents the desired changelog entry
• [X] Please describe what you did
• [X] Link to relevant issues in GitHub or Jira
• [X] Link to relevant pull requests, esp. upstream and downstream changes
• [X] Ensure you have provided tests - that demonstrates feature works or fixes the issue

[xBis7]

@kuisathaverat @cyrille-leclerc Can you take a look at this PR?

It's related to this issue https://github.com/jenkinsci/opentelemetry-plugin/issues/90

[xBis7]

@kuisathaverat Did you get a chance to take a look?

[kuisathaverat]

Sorry for not getting back to you sooner, but I did not find time to perform the manual test needed to verify that there are no regressions. Also, I want to make a huge pipeline test to verify there are no leaks or performance issues.

[xBis7]

@kuisathaverat Is there something I can do to help? Run some test? I've run manual tests but my use-cases might not match yours.

[kuisathaverat]

I need more time. I have compared two "similar" pipelines to see if the memory usage grows, and it does.

A pipeline with 500 steps uses about 450MB of RAM and does not increase memory usage after it is finished. I executed it 10 times.

pipeline {
  agent any
  stages {
    stage('prepare') {
      steps {
        bunchOfSteps(0)
      }
    }
  }
}

def bunchOfSteps(level) {
    if (level < 100) {
        sh "echo level ${level}"
        bunchOfSteps(level + 1)
    }
}

A pipeline with 500 steps and the new step to create spans uses about 850MB of RAM and increases the memory usage after it finishes about 350MB. I executed it one time.

pipeline {
  agent any
  stages {
    stage('prepare') {
      steps {
        bunchOfSteps(0)
      }
    }
  }
}

def bunchOfSteps(level) {
    if (level < 500) {
        withNewSpan(label: 'custom-build-span-' + level, attributes: ([
            spanAttribute(key: 'level', value: level),
            ]), setAttributesOnlyOnParent: true) {
                sh "echo level ${level}"
                bunchOfSteps(level + 1)
            }
    }
}

[xBis7]

It would be interesting to compare the RAM usage between

pipeline {
  agent any
  stages {
    stage('prepare') {
      steps {
        bunchOfSteps(0)
      }
    }
  }
}

def bunchOfSteps(level) {
    if (level < 500) {
        withSpanAttributes([
            spanAttribute(key: 'level', value: level),
            ]) {
                sh "echo level ${level}"
                bunchOfSteps(level + 1)
            }
    }
}

pipeline {
  agent any
  stages {
    stage('prepare') {
      steps {
        bunchOfSteps(0)
      }
    }
  }
}

def bunchOfSteps(level) {
    if (level < 500) {
        withNewSpan(label: 'custom-build-span-' + level, attributes: ([
            spanAttribute(key: 'level', value: level),
            ]), setAttributesOnlyOnParent: true) {
                sh "echo level ${level}"
                bunchOfSteps(level + 1)
            }
    }
}

withSpanAttributes might also increase the RAM usage afterwards.

[kuisathaverat]

this one directly crash after 250 steps, but I not see the memory change after several executions

def bunchOfSteps(level) {
    if (level < 500) {
        withSpanAttributes([
            spanAttribute(key: 'level', value: level),
            ]) {
                sh "echo level ${level}"
                bunchOfSteps(level + 1)
            }
    }
}

[kuisathaverat]

I am going to test this one too

def bunchOfSteps(level) {
    if (level < 500) {
        withNewSpan(label: 'custom-build-span-' + level, setAttributesOnlyOnParent: true) {
                sh "echo level ${level}"
                bunchOfSteps(level + 1)
            }
    }
}

[kuisathaverat]

I am going to change the approach recursivity 500 it is a lite too much for a pipeline

[kuisathaverat]

pipeline {
  agent any
  stages {
    stage('prepare') {
      steps {
        bunchOfSteps(0)
      }
    }
  }
}

def bunchOfSteps(level) {
    if (level < 10) {
        recursiveSteps(0)
        bunchOfSteps(level + 1)
    }
}

def recursiveSteps(level){
    if (level < 50) {
        withNewSpan(label: 'custom-build-span-' + level, attributes: ([
            spanAttribute(key: 'level', value: level),
            ]), setAttributesOnlyOnParent: true) {
                sh "echo level ${level}"
                recursiveSteps(level + 1)
            }
    }
}

[kuisathaverat]

I am also testing

pipeline {
  agent any
  stages {
    stage('prepare') {
      steps {
      withNewSpan(label: 'custom-build-span', attributes: ([
        spanAttribute(key: 'level', value: 'foo'),
        ]), setAttributesOnlyOnParent: false) {
            bunchOfSteps(0)
        }
      }
    }
  }
}

def bunchOfSteps(level) {
    if (level < 500) {
        sh "echo level ${level}"
        bunchOfSteps(level + 1)
    }
}

[kuisathaverat]

I need to grab some head dumps and verify there is nothing weird.

[kuisathaverat]

The memory issue was a snowflake. I cannot replicate it anymore, but I do not see any issue, so we can merge.

[xBis7]

@kuisathaverat Thanks for the review and the help!

[xBis7]

@kuisathaverat Thanks for merging it. Regarding using the change, should we wait for a new release, or is there some other way?

[kuisathaverat]

When we merge a change with labels bug or enhancement a new release is trigger, IIRC it takes 6 hours or so to be in the Jenkins update center https://github.com/jenkinsci/opentelemetry-plugin/releases/tag/3.1419.v3b_27ca_911066

[xBis7]

@kuisathaverat Thanks!