Egeria Sonar build failing

[planetf1]

odpi-egeria-sonar jobfailing from 13 April Example: https://jenkins.odpi.org/view/odpi-egeria/job/odpi-egeria-sonar/195/console

[planetf1]

 Looking at the log the error seems to occur around here:

[INFO] 251/1226 files analyzed, current file: /w/workspace/odpi-egeria-sonar/open-metadata-test/open-metadata-fvt/open-types-fvt/open-types-test/src/main/java/org/odpi/openmetadata/fvt/opentypes/entities/DeployedSoftwareComponent/DeployedSoftwareComponentReferences.java
[INFO] 251/1226 files analyzed, current file: /w/workspace/odpi-egeria-sonar/open-metadata-test/open-metadata-fvt/open-types-fvt/open-types-test/src/main/java/org/odpi/openmetadata/fvt/opentypes/entities/DeployedSoftwareComponent/DeployedSoftwareComponentReferences.java
FATAL: Unable to delete script file /tmp/jenkins4742838776629938586.sh
java.io.EOFException
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2680)
at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3155)
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:861)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:357)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
at hudson.remoting.Command.readFrom(Command.java:140)
at hudson.remoting.Command.readFrom(Command.java:126)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:36)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Caused: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
Caused: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on prd-centos7-builder-4c-4g-31294 failed. The channel is closing down or has closed down

Which suggests it is something related to the infrastructure in some way -- as if the filesystem perhaps has disappeared?

Could it relate to load? some kind of timeout? other resource exhaustion? Any logs from jenkins or your infrastructure scripts that may help?

I ran my own sonar scan off master using:

cd ~/src/egeria
mvn sonar:sonar \
  -Dsonar.projectKey=planetf1_egeria \
  -Dsonar.organization=planetf1-github \
  -Dsonar.host.url=https://sonarcloud.io \
  -Dsonar.login=XX-SECRET-XX

Whilst in the past it took 3-5 mins (nov), 4-7 mins (jan), today it failed aftger  Looking at the log the error seems to ocur around here:

[INFO] 251/1226 files analyzed, current file: /w/workspace/odpi-egeria-sonar/open-metadata-test/open-metadata-fvt/open-types-fvt/open-types-test/src/main/java/org/odpi/openmetadata/fvt/opentypes/entities/DeployedSoftwareComponent/DeployedSoftwareComponentReferences.java [INFO] 251/1226 files analyzed, current file: /w/workspace/odpi-egeria-sonar/open-metadata-test/open-metadata-fvt/open-types-fvt/open-types-test/src/main/java/org/odpi/openmetadata/fvt/opentypes/entities/DeployedSoftwareComponent/DeployedSoftwareComponentReferences.java FATAL: Unable to delete script file /tmp/jenkins4742838776629938586.sh java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2680) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3155) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:861) at java.io.ObjectInputStream.(ObjectInputStream.java:357) at hudson.remoting.ObjectInputStreamEx.(ObjectInputStreamEx.java:49) at hudson.remoting.Command.readFrom(Command.java:140) at hudson.remoting.Command.readFrom(Command.java:126) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:36) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63) Caused: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77) Caused: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on prd-centos7-builder-4c-4g-31294 failed. The channel is closing down or has closed down

Which suggests it is something related to the infrastructure in some way -- as if the filesystem perhaps has disappeared?

Could it relate to load? some kind of timeout? other resource exhaustion? Any logs from jenkins or your infrastructure scripts that may help?

I ran my own sonar scan off master using:

cd ~/src/egeria mvn sonar:sonar \   -Dsonar.projectKey=planetf1_egeria \   -Dsonar.organization=planetf1-github \   -Dsonar.host.url=https://sonarcloud.io \   -Dsonar.login=XX-SECRET-XX

It completed succesfully, but whist in the past it took 3-5 mins (nov), 4-7 mins (jan), today it failed after 35 mins with: org.sonar.ce.task.CeTaskTimeoutException: Execution of task timed out after 2113080 ms

One run isn't enough for a definitive answer, but resource exhaustion/timeout looks a possibility here...

A significant change was made in this code area -- we had some generated code that has been removed from our main code build, but a similar approach is now being used for test generation

It may be we need to exclude some code from the sonar scan. Any generated scan should be annotated with @generated - though we did hit a problem here with Java 8 vs Java 11 ( see https://github.com/odpi/egeria/issues/916 )

I'll check with the author (David Radley)

[schannamallu]

@planetf1 , I checked with infrastructure team, as per them - it looks like the builder terminates before the job finishes, may be its not any infra issues there.

[planetf1]

Could we add this to Administration / General / Analysis Scope / Files / Source File Exclusions :

**/open-metadata-fvt/**

I have tested this on sonarcloud.io and it fixes the sonar build failing with a timeout

Perhaps it would be useful if we had access to modifying these settings as it's an iterative process... cc: @davidradl

[schannamallu]

@planetf1 , Updated the 'Source file Exclusion'. Will test it once..

[schannamallu]

[schannamallu]

@planetf1 , After the settings, still the job is failing..

[planetf1]

How can we get further information? I did try on sonarcloud.io (the public site) and with the old settings sonar was failing with a timeout after many minutes so I guessed perhaps the cause was similar. We don't need to include this code in our scans anyway, so exclusion seemed a reasonable approach. (ref https://sonarcloud.io/dashboard?id=planetf1_egeria )

The log on odpi jenkins is showing

[INFO] 380/1199 files analyzed, current file: /w/workspace/odpi-egeria-sonar/open-metadata-test/open-metadata-fvt/open-types-fvt/open-types-test/src/main/java/org/odpi/openmetadata/fvt/opentypes/entities/ITProfile/ITProfileReferences.java
[INFO] 383/1199 files analyzed, current file: /w/workspace/odpi-egeria-sonar/open-metadata-test/open-metadata-fvt/open-types-fvt/open-types-test/src/main/java/org/odpi/openmetadata/fvt/opentypes/entities/ImplementationSnippet/ImplementationSnippetReferences.java
FATAL: Unable to delete script file /tmp/jenkins1451910748490605752.sh
java.io.EOFException
    at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2680)
    at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3155)
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:861)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:357)
    at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
    at hudson.remoting.Command.readFrom(Command.java:140)
    at hudson.remoting.Command.readFrom(Command.java:126)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:36)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Caused: java.io.IOException: Unexpected termination of the channel
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
Caused: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on prd-centos7-builder-4c-4g-168 failed. The channel is closing down or has closed down

which is the same as before. I wonder if that 'channel is closing down' means that there is some timeout coming into play on the infrastructure - can you check to see how long the scripts will wait for sonar to complete.?

The log also seems to indicate that the source exclusion made no difference, which seems a little odd - is the right sonar config definately updated? I don't have access to sonarcloud.io logs to compare but it does seem as if my config change made the difference.

Also I note our sonar build isnt get a gate (fail) on our daily/commit builds, since we know we don't meet the current targets. Is the maven publish #43 failing due to the job failing? is that a necessary check? Is it just that the job should run rather than the metrics pass?

[schannamallu]

@planetf1 Good News ! It was success Egeria job. https://logs.odpi.org/production/vex-yul-odpi-jenkins-prod-1/odpi-egeria-sonar/229/

[schannamallu]

@planetf1 , Update: Looks like still getting some more issue. I am looking into it.

[planetf1]

Looking at the build logs I can clearly see the fvt exclusions ARE being picked up

[INFO]   sonar.exclusions=open-metadata-fvt/**/*
[INFO]   sonar.exclusions= **/open-metadata-fvt/**
[INFO]   sonar.exclusions=src/main/java/org/odpi/openmetadata/fvt/**
[INFO]   sonar.exclusions=**/open-metadata-test/open-metadata-fvt/**/*
[INFO] Excluded tests: 
[INFO]   **/open-metadata-fvt/**
[INFO]   open-metadata-fvt/**/*
[INFO]   sonar.exclusions=open-metadata-fvt/**/*.java
[INFO]   open-metadata-fvt/**/*
[INFO]   open-metadata-fvt/**/*.java
[INFO]   sonar.test.exclusions=**/open-metadata-test/open-metadata-fvt/**/*

So perhaps the fact that later in the log I see:

[INFO] 173/1199 files analyzed, current file: /w/workspace/odpi-egeria-sonar/open-metadata-test/open-metadata-fvt/open-types-fvt/open-types-test/src/main/java/org/odpi/openmetadata/fvt/opentypes/entities/Collection/CollectionReferences.java

is normal A little confusing though since that file seems to match the pattern Yet it is still failing

Do we have anyone with sonar skills who could advise if this is correct? Can we ask sonar support if this is ok.

That being said the build then seems to fail at

[INFO] 284/1199 files analyzed, current file: /w/workspace/odpi-egeria-sonar/open-metadata-test/open-metadata-fvt/open-types-fvt/open-types-test/src/main/java/org/odpi/openmetadata/fvt/opentypes/entities/EventType/EventTypeReferences.java
FATAL: Unable to delete script file /tmp/jenkins4691599265212173408.sh
java.io.EOFException
    at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2680)

That sounds as if something is getting closed or ending prematurely. Whether it is the sonar scan crashing, or the filesystem being removed underneath the process, or the container being terminated I'm not sure.

Are you able to add more debug to the actual build scripts so we can be sure exactly what the command that is ending is? Can you somehow 'freeze' the build after it hits a failure so that the vm/containers 'hang around' and can be logged into to investigate? Could it still be some kind of timeout? What applies for this build process? Or other resource limits?

[schannamallu]

@planetf1, Actually the scan of open-metadata-fvt is taking long time and the jenkins to node connection is getting timed out because of that.

And some how adding the exclusions in the sonarqube settings is not working and the build is getting failed.

So can we add the exclusions in the pom file or any config file in the code directly. Checked with the internal team as well. They suggested the same option.

I see some exclusion in this pom file - https://github.com/odpi/egeria/blob/master/pom.xml#L398 Can you add the exclusion for open-metadata-fvt the same way.

Other projects excluding using pom file eg: https://github.com/onap/ccsdk-platform-plugins/blob/master/pom.xml#L44 https://github.com/onap/ccsdk-platform-plugins/blob/master/dnsdesig/pom.xml#L52

[planetf1]

I've added a PR to add the sonar exclusions locally. I hadn't remembered we could do it there. Probably makes sense as it is easier for dev team to control. I'll try and get the issue pushed later today assuming no objections.

[schannamallu]

@planetf1, Still getting the same issue. exclusion is identified but not excluding.

[INFO] Source encoding: UTF-8, default locale: en_US [INFO] Index files [INFO] Excluded sources: [INFO] /open-metadata-fvt/ [INFO] 1201 files indexed [INFO] 0 files ignored because of inclusion/exclusion patterns [INFO] Quality profile for java: Sonar way [INFO] Quality profile for xml: Sonar way [INFO] Excluded sources for coverage: [INFO] /generated/ [INFO] Sensor JavaSquidSensor [java] [INFO] Configured Java source version (sonar.java.source): 8 [INFO] JavaClasspath initialization [INFO] JavaClasspath initialization (done) | time=1ms [INFO] JavaTestClasspath initialization [INFO] JavaTestClasspath initialization (done) | time=2ms [INFO] Java Main Files AST scan [INFO] 1199 source files to be analyzed [INFO] 36/1199 files analyzed, current file: /w/workspace/odpi-egeria-sonar/open-metadata-test/open-metadata-fvt/open-types-fvt/open-types-test/src/main/java/org/odpi/openmetadata/fvt/opentypes/classifications/ControlPoint/ControlPointMapper.java [INFO] 76/1199 files analyzed, current file: /w/workspace/odpi-egeria-sonar/open-metadata-test/open-metadata-fvt/open-types-fvt/open-types-test/src/main/java/org/odpi/openmetadata/fvt/opentypes/classifications/MeteringLog/MeteringLogMapper.java [INFO] 124/1199 files analyzed, current file: /w/workspace/odpi-egeria-sonar/open-metadata-test/open-metadata-fvt/open-types-fvt/open-types-test/src/main/java/org/odpi/openmetadata/fvt/opentypes/entities/APIOperation/APIOperationMapper.java

[schannamallu]

@planetf1 we tried with multiple options


1. Giving exclude in SonarUI as /open-metadata-fvt/ -- same issue


2. Change the folder patterns in -Dsonar.exclusions= /open-metadata-fvt//* ,/open-metadata-fvt//** --same issue


3. Change the mvn goals as -DskippedModules=open-metadata-test -- same issue


In my local system i didn't face such issues . MVN V3.6 ,JDK 1.8 two pretty commands 1> mvn clean install 2> mvn sonar:sonar-Dsonar.projectKey=XXXX -Dsonar.organization=XXX -Dsonar.host.url=https://sonarcloud.io -Dsonar.login=

[planetf1]

If one works, and one doesn't ..How do the jobs at LF differ from sonarcloud.io

• What version of sonar do we have? Is it backlevel & in need of upgrading?
• How does it's configuration differ (this could be tricky.. but can you share more complete config?)
• I can guess java/maven might differ, though I'd be inclined not to change ... YET...

Also what 'support' do we have with the company offering sonar? Can we report the issue to them for help? Are there any issues with other projects?

Can we change the timeout/resource limits for the job - even if the exclusion doesn't work, can we at least get the job to run to completion?

[schannamallu]

we modified the pom.xml at open-metadata-test with the following sonar property

<properties>
       <sonar.skip>true</sonar.skip>
    </properties>

that will do the trick . The sonar skip the analyzing the open-metadata-test.

Please find more logs : https://jenkins.odpi.org/sandbox/view/All/job/odpi-egeria-sonar/3/console

[schannamallu]

If one works, and one doesn't ..How do the jobs at LF differ from sonarcloud.io

• What version of sonar do we have? Is it backlevel & in need of upgrading?
• How does it's configuration differ (this could be tricky.. but can you share more complete config?)
• I can guess java/maven might differ, though I'd be inclined not to change ... YET...

Also what 'support' do we have with the company offering sonar? Can we report the issue to them for help? Are there any issues with other projects?

Can we change the timeout/resource limits for the job - even if the exclusion doesn't work, can we at least get the job to run to completion?

I tested with sonarcloud, it works perfect seems the problem with https://sonar.odpi.org . for more logs checkout here: https://jenkins.odpi.org/sandbox/view/All/job/odpi-egeria-sonar/5/console

checkout the sonarcloud analyze url : https://sonarcloud.io/organizations/odpi/projects

[planetf1]

The latest report was good. The previous one gave us all-green scores - 0 bugs, so something went a bit odd. I suggest we keep this open for a few days to see if the build is now stable.

I can see your sonarcloud.io run. That is a viable alternative if we still have the flexibility to run against multiple releases and scan automatically. That may be preferable to maintaining our own sonar install, and could be more up to date. Do you have any recommendation?

[schannamallu]

@planetf1

[schannamallu]

@planetf1

[schannamallu]

The latest report was good. The previous one gave us all-green scores - 0 bugs, so something went a bit odd. I suggest we keep this open for a few days to see if the build is now stable.

I can see your sonarcloud.io run. That is a viable alternative if we still have the flexibility to run against multiple releases and scan automatically. That may be preferable to maintaining our own sonar install, and could be more up to date. Do you have any recommendation?

the recommendations are

1. Setup a new one with the latest release and try the analysis
2. if its good we will update existing to the latest release

[planetf1]

Ok, sounds good. thanks

[schannamallu]

@

The latest report was good. The previous one gave us all-green scores - 0 bugs, so something went a bit odd. I suggest we keep this open for a few days to see if the build is now stable. I can see your sonarcloud.io run. That is a viable alternative if we still have the flexibility to run against multiple releases and scan automatically. That may be preferable to maintaining our own sonar install, and could be more up to date. Do you have any recommendation?

the recommendations are

1. Setup a new one with the latest release and try the analysis
2. if its good we will update existing to the latest release

@planetf1 we tested the the code analasys with sonarqube 7.7. It is working fine, i mean theres no need to add exclusion conditions in the SonarUI.

if you are ok with 7.7 . we will upgrade the old to latest . please let me know. I need to inform infra team

[planetf1]

Yes, that's ok to go with. thanks

[schannamallu]

@planetf1 successfully build the sonarqube job using sonrcloud. Please find the logs below https://jenkins.odpi.org/view/odpi-egeria/job/odpi-egeria-sonar/

[schannamallu]

@planetf1 successfully build the sonarqube job using sonrcloud. Please find the logs below https://jenkins.odpi.org/view/odpi-egeria/job/odpi-egeria-sonar/272/console