Closed planetf1 closed 5 years ago
Looking at the log the error seems to occur around here:
[INFO] 251/1226 files analyzed, current file: /w/workspace/odpi-egeria-sonar/open-metadata-test/open-metadata-fvt/open-types-fvt/open-types-test/src/main/java/org/odpi/openmetadata/fvt/opentypes/entities/DeployedSoftwareComponent/DeployedSoftwareComponentReferences.java
[INFO] 251/1226 files analyzed, current file: /w/workspace/odpi-egeria-sonar/open-metadata-test/open-metadata-fvt/open-types-fvt/open-types-test/src/main/java/org/odpi/openmetadata/fvt/opentypes/entities/DeployedSoftwareComponent/DeployedSoftwareComponentReferences.java
FATAL: Unable to delete script file /tmp/jenkins4742838776629938586.sh
java.io.EOFException
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2680)
at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3155)
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:861)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:357)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
at hudson.remoting.Command.readFrom(Command.java:140)
at hudson.remoting.Command.readFrom(Command.java:126)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:36)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Caused: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
Caused: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on prd-centos7-builder-4c-4g-31294 failed. The channel is closing down or has closed down
Which suggests it is something related to the infrastructure in some way -- as if the filesystem perhaps has disappeared?
Could it relate to load? some kind of timeout? other resource exhaustion? Any logs from jenkins or your infrastructure scripts that may help?
I ran my own sonar scan off master using:
cd ~/src/egeria
mvn sonar:sonar \
-Dsonar.projectKey=planetf1_egeria \
-Dsonar.organization=planetf1-github \
-Dsonar.host.url=https://sonarcloud.io \
-Dsonar.login=XX-SECRET-XX
Whilst in the past it took 3-5 mins (nov), 4-7 mins (jan), today it failed aftger Looking at the log the error seems to ocur around here:
[INFO] 251/1226 files analyzed, current file: /w/workspace/odpi-egeria-sonar/open-metadata-test/open-metadata-fvt/open-types-fvt/open-types-test/src/main/java/org/odpi/openmetadata/fvt/opentypes/entities/DeployedSoftwareComponent/DeployedSoftwareComponentReferences.java
[INFO] 251/1226 files analyzed, current file: /w/workspace/odpi-egeria-sonar/open-metadata-test/open-metadata-fvt/open-types-fvt/open-types-test/src/main/java/org/odpi/openmetadata/fvt/opentypes/entities/DeployedSoftwareComponent/DeployedSoftwareComponentReferences.java
FATAL: Unable to delete script file /tmp/jenkins4742838776629938586.sh
java.io.EOFException
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2680)
at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3155)
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:861)
at java.io.ObjectInputStream.
Which suggests it is something related to the infrastructure in some way -- as if the filesystem perhaps has disappeared?
Could it relate to load? some kind of timeout? other resource exhaustion? Any logs from jenkins or your infrastructure scripts that may help?
I ran my own sonar scan off master using:
cd ~/src/egeria mvn sonar:sonar \ -Dsonar.projectKey=planetf1_egeria \ -Dsonar.organization=planetf1-github \ -Dsonar.host.url=https://sonarcloud.io \ -Dsonar.login=XX-SECRET-XX
It completed succesfully, but whist in the past it took 3-5 mins (nov), 4-7 mins (jan), today it failed after 35 mins with:
org.sonar.ce.task.CeTaskTimeoutException: Execution of task timed out after 2113080 ms
One run isn't enough for a definitive answer, but resource exhaustion/timeout looks a possibility here...
A significant change was made in this code area -- we had some generated code that has been removed from our main code build, but a similar approach is now being used for test generation
It may be we need to exclude some code from the sonar scan. Any generated scan should be annotated with @generated - though we did hit a problem here with Java 8 vs Java 11 ( see https://github.com/odpi/egeria/issues/916 )
I'll check with the author (David Radley)
@planetf1 , I checked with infrastructure team, as per them - it looks like the builder terminates before the job finishes, may be its not any infra issues there.
Could we add this to Administration / General / Analysis Scope / Files / Source File Exclusions :
**/open-metadata-fvt/**
I have tested this on sonarcloud.io and it fixes the sonar build failing with a timeout
Perhaps it would be useful if we had access to modifying these settings as it's an iterative process... cc: @davidradl
@planetf1 , Updated the 'Source file Exclusion'. Will test it once..
@planetf1 , After the settings, still the job is failing..
How can we get further information? I did try on sonarcloud.io (the public site) and with the old settings sonar was failing with a timeout after many minutes so I guessed perhaps the cause was similar. We don't need to include this code in our scans anyway, so exclusion seemed a reasonable approach. (ref https://sonarcloud.io/dashboard?id=planetf1_egeria )
The log on odpi jenkins is showing
[INFO] 380/1199 files analyzed, current file: /w/workspace/odpi-egeria-sonar/open-metadata-test/open-metadata-fvt/open-types-fvt/open-types-test/src/main/java/org/odpi/openmetadata/fvt/opentypes/entities/ITProfile/ITProfileReferences.java
[INFO] 383/1199 files analyzed, current file: /w/workspace/odpi-egeria-sonar/open-metadata-test/open-metadata-fvt/open-types-fvt/open-types-test/src/main/java/org/odpi/openmetadata/fvt/opentypes/entities/ImplementationSnippet/ImplementationSnippetReferences.java
FATAL: Unable to delete script file /tmp/jenkins1451910748490605752.sh
java.io.EOFException
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2680)
at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3155)
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:861)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:357)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
at hudson.remoting.Command.readFrom(Command.java:140)
at hudson.remoting.Command.readFrom(Command.java:126)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:36)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Caused: java.io.IOException: Unexpected termination of the channel
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
Caused: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on prd-centos7-builder-4c-4g-168 failed. The channel is closing down or has closed down
which is the same as before. I wonder if that 'channel is closing down' means that there is some timeout coming into play on the infrastructure - can you check to see how long the scripts will wait for sonar to complete.?
The log also seems to indicate that the source exclusion made no difference, which seems a little odd - is the right sonar config definately updated? I don't have access to sonarcloud.io logs to compare but it does seem as if my config change made the difference.
Also I note our sonar build isnt get a gate (fail) on our daily/commit builds, since we know we don't meet the current targets. Is the maven publish #43 failing due to the job failing? is that a necessary check? Is it just that the job should run rather than the metrics pass?
@planetf1 Good News ! It was success Egeria job. https://logs.odpi.org/production/vex-yul-odpi-jenkins-prod-1/odpi-egeria-sonar/229/
@planetf1 , Update: Looks like still getting some more issue. I am looking into it.
Looking at the build logs I can clearly see the fvt exclusions ARE being picked up
[INFO] sonar.exclusions=open-metadata-fvt/**/*
[INFO] sonar.exclusions= **/open-metadata-fvt/**
[INFO] sonar.exclusions=src/main/java/org/odpi/openmetadata/fvt/**
[INFO] sonar.exclusions=**/open-metadata-test/open-metadata-fvt/**/*
[INFO] Excluded tests:
[INFO] **/open-metadata-fvt/**
[INFO] open-metadata-fvt/**/*
[INFO] sonar.exclusions=open-metadata-fvt/**/*.java
[INFO] open-metadata-fvt/**/*
[INFO] open-metadata-fvt/**/*.java
[INFO] sonar.test.exclusions=**/open-metadata-test/open-metadata-fvt/**/*
So perhaps the fact that later in the log I see:
[INFO] 173/1199 files analyzed, current file: /w/workspace/odpi-egeria-sonar/open-metadata-test/open-metadata-fvt/open-types-fvt/open-types-test/src/main/java/org/odpi/openmetadata/fvt/opentypes/entities/Collection/CollectionReferences.java
is normal A little confusing though since that file seems to match the pattern Yet it is still failing
Do we have anyone with sonar skills who could advise if this is correct? Can we ask sonar support if this is ok.
That being said the build then seems to fail at
[INFO] 284/1199 files analyzed, current file: /w/workspace/odpi-egeria-sonar/open-metadata-test/open-metadata-fvt/open-types-fvt/open-types-test/src/main/java/org/odpi/openmetadata/fvt/opentypes/entities/EventType/EventTypeReferences.java
FATAL: Unable to delete script file /tmp/jenkins4691599265212173408.sh
java.io.EOFException
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2680)
That sounds as if something is getting closed or ending prematurely. Whether it is the sonar scan crashing, or the filesystem being removed underneath the process, or the container being terminated I'm not sure.
Are you able to add more debug to the actual build scripts so we can be sure exactly what the command that is ending is? Can you somehow 'freeze' the build after it hits a failure so that the vm/containers 'hang around' and can be logged into to investigate? Could it still be some kind of timeout? What applies for this build process? Or other resource limits?
@planetf1, Actually the scan of open-metadata-fvt is taking long time and the jenkins to node connection is getting timed out because of that.
And some how adding the exclusions in the sonarqube settings is not working and the build is getting failed.
So can we add the exclusions in the pom file or any config file in the code directly. Checked with the internal team as well. They suggested the same option.
I see some exclusion in this pom file - https://github.com/odpi/egeria/blob/master/pom.xml#L398 Can you add the exclusion for open-metadata-fvt the same way.
Other projects excluding using pom file eg: https://github.com/onap/ccsdk-platform-plugins/blob/master/pom.xml#L44 https://github.com/onap/ccsdk-platform-plugins/blob/master/dnsdesig/pom.xml#L52
I've added a PR to add the sonar exclusions locally. I hadn't remembered we could do it there. Probably makes sense as it is easier for dev team to control. I'll try and get the issue pushed later today assuming no objections.
@planetf1, Still getting the same issue. exclusion is identified but not excluding.
[INFO] Source encoding: UTF-8, default locale: en_US [INFO] Index files [INFO] Excluded sources: [INFO] /open-metadata-fvt/ [INFO] 1201 files indexed [INFO] 0 files ignored because of inclusion/exclusion patterns [INFO] Quality profile for java: Sonar way [INFO] Quality profile for xml: Sonar way [INFO] Excluded sources for coverage: [INFO] /generated/ [INFO] Sensor JavaSquidSensor [java] [INFO] Configured Java source version (sonar.java.source): 8 [INFO] JavaClasspath initialization [INFO] JavaClasspath initialization (done) | time=1ms [INFO] JavaTestClasspath initialization [INFO] JavaTestClasspath initialization (done) | time=2ms [INFO] Java Main Files AST scan [INFO] 1199 source files to be analyzed [INFO] 36/1199 files analyzed, current file: /w/workspace/odpi-egeria-sonar/open-metadata-test/open-metadata-fvt/open-types-fvt/open-types-test/src/main/java/org/odpi/openmetadata/fvt/opentypes/classifications/ControlPoint/ControlPointMapper.java [INFO] 76/1199 files analyzed, current file: /w/workspace/odpi-egeria-sonar/open-metadata-test/open-metadata-fvt/open-types-fvt/open-types-test/src/main/java/org/odpi/openmetadata/fvt/opentypes/classifications/MeteringLog/MeteringLogMapper.java [INFO] 124/1199 files analyzed, current file: /w/workspace/odpi-egeria-sonar/open-metadata-test/open-metadata-fvt/open-types-fvt/open-types-test/src/main/java/org/odpi/openmetadata/fvt/opentypes/entities/APIOperation/APIOperationMapper.java
@planetf1 we tried with multiple options
Giving exclude in SonarUI as /open-metadata-fvt/ -- same issue
Change the folder patterns in -Dsonar.exclusions= /open-metadata-fvt//* ,/open-metadata-fvt//** --same issue
Change the mvn goals as -DskippedModules=open-metadata-test -- same issue
In my local system i didn't face such issues .
MVN V3.6 ,JDK 1.8
two pretty commands
1> mvn clean install
2> mvn sonar:sonar-Dsonar.projectKey=XXXX -Dsonar.organization=XXX -Dsonar.host.url=https://sonarcloud.io -Dsonar.login=
odpi-egeria-sonar jobfailing from 13 April Example: https://jenkins.odpi.org/view/odpi-egeria/job/odpi-egeria-sonar/195/console