Tasks vanish, projects have no status or are in error

[davidediruscio]

The ci4 instance seems to be in limbo. Projects don't show any status in the main page, they're marked as error when clicking on a specific project, and I can't see the tasks usually queued in the workers page.

When clicking on the project the task is in error.

None of the projects are displayed in the dashboard, but it may be because of another issue (https://github.com/crossminer/scava-deployment/issues/107).

[davidediruscio]

Now I realise I cannot add new tasks (or re-run existing tasks). Blocker.

[davidediruscio]

Hi @borisbaldassari , can you check the logs of the oss-app. We are struggling with a similar problem and the getting http server errors on the oss-app. Something starting with:

WARNING: Exception or error caught in server resource
Internal Server Error (500) - The server encountered an unexpected condition which prevented it from fulfilling the request

Maybe related with https://github.com/crossminer/scava/issues/341

[davidediruscio]

Hi @borisbaldassari

Projects don't show any status in the main page,

The commit (6c9a81f) should fix that !

they're marked as error when clicking on a specific project,

It's expected in case that the task is failed to create the Delta. I took a snapshot from the oss-app_1 container logs(where the task was failed ) and I got a No space left on device error.

INFO [ProjectDelta (Eclipse Trace Compass,20180101)] (13:03:00): Creating Delta ERROR [ProjectDelta (Eclipse Trace Compass,20180101)] (13:03:09): Delta creation failed. org.eclipse.jgit.api.errors.TransportException: No space left on device at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:255) at org.eclipse.jgit.api.CloneCommand.fetch(CloneCommand.java:302) at org.eclipse.jgit.api.CloneCommand.call(CloneCommand.java:200) at org.eclipse.scava.platform.vcs.git.GitManager.getGit(GitManager.java:347) at org.eclipse.scava.platform.vcs.git.GitManager.getRevisionsForDate(GitManager.java:283) at org.eclipse.scava.platform.delta.vcs.PlatformVcsManager.getRevisionsForDate(PlatformVcsManager.java:77) at org.eclipse.scava.platform.delta.vcs.VcsProjectDelta.<init>(VcsProjectDelta.java:25) at org.eclipse.scava.platform.delta.ProjectDelta.create(ProjectDelta.java:64) at org.eclipse.scava.platform.osgi.analysis.ProjectAnalyser.executeAnalyse(ProjectAnalyser.java:104) at org.eclipse.scava.platform.osgi.services.TaskExecutor.run(TaskExecutor.java:26) at java.lang.Thread.run(Thread.java:748) Caused by: org.eclipse.jgit.errors.TransportException: No space left on device at org.eclipse.jgit.transport.BasePackFetchConnection.doFetch(BasePackFetchConnection.java:400) at org.eclipse.jgit.transport.BasePackFetchConnection.fetch(BasePackFetchConnection.java:323) at org.eclipse.jgit.transport.BasePackFetchConnection.fetch(BasePackFetchConnection.java:314) at org.eclipse.jgit.transport.FetchProcess.fetchObjects(FetchProcess.java:265) at org.eclipse.jgit.transport.FetchProcess.executeImp(FetchProcess.java:163) at org.eclipse.jgit.transport.FetchProcess.execute(FetchProcess.java:124) at org.eclipse.jgit.transport.Transport.fetch(Transport.java:1271) at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:244)
... 10 more Caused by: java.io.IOException: No space left on device at java.io.RandomAccessFile.writeBytes(Native Method) at java.io.RandomAccessFile.write(RandomAccessFile.java:525) at org.eclipse.jgit.internal.storage.file.ObjectDirectoryPackParser.onStoreStream(ObjectDirectoryPackParser.java:289) at org.eclipse.jgit.transport.PackParser.sync(PackParser.java:1278) at org.eclipse.jgit.transport.PackParser.fill(PackParser.java:1248) at org.eclipse.jgit.transport.PackParser$InflaterStream.read(PackParser.java:1817) at org.eclipse.jgit.util.IO.readFully(IO.java:234) at org.eclipse.jgit.transport.PackParser.inflateAndReturn(PackParser.java:1650) at org.eclipse.jgit.transport.PackParser.whole(PackParser.java:1101) at org.eclipse.jgit.transport.PackParser.indexOneObject(PackParser.java:1013) at org.eclipse.jgit.transport.PackParser.parse(PackParser.java:568) at org.eclipse.jgit.internal.storage.file.ObjectDirectoryPackParser.parse(ObjectDirectoryPackParser.java:197) at org.eclipse.jgit.transport.PackParser.parse(PackParser.java:529) at org.eclipse.jgit.transport.BasePackFetchConnection.receivePack(BasePackFetchConnection.java:823) at org.eclipse.jgit.transport.BasePackFetchConnection.doFetch(BasePackFetchConnection.java:393) ... 17 more Suppressed: java.io.IOException: No space left on device at java.io.RandomAccessFile.writeBytes(Native Method) at java.io.RandomAccessFile.write(RandomAccessFile.java:525) at org.eclipse.jgit.internal.storage.file.ObjectDirectoryPackParser.onStoreStream(ObjectDirectoryPackParser.java:289) at org.eclipse.jgit.transport.PackParser.sync(PackParser.java:1278) at org.eclipse.jgit.transport.PackParser.fill(PackParser.java:1248) at org.eclipse.jgit.transport.PackParser$InflaterStream.read(PackParser.java:1817) at java.io.InputStream.read(InputStream.java:101) at org.eclipse.jgit.transport.PackParser$InflaterStream.close(PackParser.java:1839) at org.eclipse.jgit.transport.PackParser.inflateAndReturn(PackParser.java:1651) ... 24 more ERROR [ProjectExecutor (w1:toolstracecompass_task)] (13:03:09): Project delta creation failed. Aborting.

and I can't see the tasks usually queued in the workers page.

The tasks queued in the workers view when they 're on PENDING_EXECUTION status and there is no worker available to process them.