search

crossminer / scava

https://eclipse.org/scava/
Eclipse Public License 2.0
18 stars 13 forks source link

[eclipse forums] Failing to generate the Oauth token #428

Closed valeriocos closed 4 years ago

valeriocos commented 4 years ago

I'm analysing the project https://projects.eclipse.org/projects/iot.mosquitto, and the Oauth token to mine the eclipse forum cannot be generated. This causes the task execution to end up with an error. If I remove the forum, the execution runs smooth (http://crossminer.bitergia.net:5601/#/project/configure/iotmosquitto).

Should I configure something on my machine @tdegueul @creat89 ? (cc @borisbaldassari)

INFO  [ProjectExecutor (w1:iotmosquitto:mosquitto)] (07:44:31): Project iotmosquittoTask execution ( iotmosquitto:mosquitto : Date 20180101 )
INFO  [AnalysisSchedulingService] (07:44:31): Starting new daily execution AnalysisTask 'iotmosquitto:mosquitto'
INFO  [ProjectExecutor (w1:iotmosquitto:mosquitto)] (07:44:31): Date: 20180101, project: Eclipse Mosquitto
INFO  [ProjectDelta (Eclipse Mosquitto,20180101)] (07:44:31): Creating Delta
INFO  [platform.bugtrackingsystem.eclipseforums] (07:44:34): Getting Client
INFO  [eclipse_forums.token.manager] (07:44:34): Attempting to generate OAuth Token
ERROR [ProjectDelta (Eclipse Mosquitto,20180101)] (07:44:35): Delta creation failed.
java.lang.NullPointerException
    at org.eclipse.scava.platform.communicationchannel.eclipseforums.client.manager.ClientManager.generateOAuthToken(ClientManager.java:143)
    at org.eclipse.scava.platform.communicationchannel.eclipseforums.client.manager.ClientManager.addNewClient(ClientManager.java:64)
    at org.eclipse.scava.platform.communicationchannel.eclipseforums.client.manager.ClientManager.clientDataExists(ClientManager.java:189)
    at org.eclipse.scava.platform.communicationchannel.eclipseforums.EclipseForumsManager.getClient(EclipseForumsManager.java:447)
    at org.eclipse.scava.platform.communicationchannel.eclipseforums.EclipseForumsManager.getDelta(EclipseForumsManager.java:123)
    at org.eclipse.scava.platform.communicationchannel.eclipseforums.EclipseForumsManager.getDelta(EclipseForumsManager.java:1)
    at org.eclipse.scava.platform.delta.communicationchannel.PlatformCommunicationChannelManager.getDelta(PlatformCommunicationChannelManager.java:103)
    at org.eclipse.scava.platform.delta.communicationchannel.CommunicationChannelProjectDelta.<init>(CommunicationChannelProjectDelta.java:28)
    at org.eclipse.scava.platform.delta.ProjectDelta.create(ProjectDelta.java:69)
    at org.eclipse.scava.platform.osgi.analysis.ProjectAnalyser.executeAnalyse(ProjectAnalyser.java:103)
    at org.eclipse.scava.platform.osgi.services.TaskExecutor.run(TaskExecutor.java:26)
    at java.lang.Thread.run(Thread.java:748)
ERROR [ProjectExecutor (w1:iotmosquitto:mosquitto)] (07:44:35): Project delta creation failed. Aborting.
creat89 commented 4 years ago

That's an issue from our side. I'll fix asap. There must be a wrong comparison.

valeriocos commented 4 years ago

Thank you @creat89 for the quick reply!

creat89 commented 4 years ago

This should be fixed with commit https://github.com/crossminer/scava/commit/1bfa5fedf60b1358cf1f0805aff98700c43cd4ab

valeriocos commented 4 years ago

The error has changed @creat89, maybe I'm doing something wrong. Can you check if the communication channels of the projects are correctly set (http://crossminer.bitergia.net:5601/#/project/configure/iotmosquitto)? Thanks

INFO  [ProjectExecutor (w1:iotmosquitto:aaamosqu)] (14:06:23): Extracting Factoid metrics of all metricProviders
INFO  [ProjectExecutor (w1:iotmosquitto:aaamosqu)] (14:06:23): Extracting Factoid metricProviders is done.
INFO  [ProjectExecutor (w1:iotmosquitto:aaamosqu)] (14:06:23): Creating metric branches.
INFO  [ProjectExecutor (w1:iotmosquitto:aaamosqu)] (14:06:23): Creating metric branches is done in 8 ms
INFO  [ProjectExecutor (w1:iotmosquitto:aaamosqu)] (14:06:23): Dates: 685
INFO  [ProjectExecutor (w1:iotmosquitto:aaamosqu)] (14:06:23): Project iotmosquittoTask execution ( iotmosquitto:aaamosqu : Date 20180101 )
INFO  [AnalysisSchedulingService] (14:06:23): Starting new daily execution AnalysisTask 'iotmosquitto:aaamosqu'
INFO  [ProjectExecutor (w1:iotmosquitto:aaamosqu)] (14:06:23): Date: 20180101, project: Eclipse Mosquitto
INFO  [ProjectDelta (Eclipse Mosquitto,20180101)] (14:06:23): Creating Delta
INFO  [eclipse_forums.token.manager] (14:06:24): Client map has been initialised
INFO  [platform.bugtrackingsystem.eclipseforums] (14:06:24): Getting Client
INFO  [eclipse_forums.token.manager] (14:06:24): Unauthenticated client has been added for Mosquitto discussion forum/270:EclipseForums
INFO  [eclipse_forums.token.manager] (14:06:24): Client data has been added to Client Map Mosquitto discussion forum/270:EclipseForums
ERROR [ProjectDelta (Eclipse Mosquitto,20180101)] (14:06:24): Delta creation failed.
java.lang.NullPointerException
    at org.eclipse.scava.platform.communicationchannel.eclipseforums.client.manager.ClientManager.checkForRefresh(ClientManager.java:98)
    at org.eclipse.scava.platform.communicationchannel.eclipseforums.EclipseForumsManager.getClient(EclipseForumsManager.java:449)
    at org.eclipse.scava.platform.communicationchannel.eclipseforums.EclipseForumsManager.getDelta(EclipseForumsManager.java:123)
    at org.eclipse.scava.platform.communicationchannel.eclipseforums.EclipseForumsManager.getDelta(EclipseForumsManager.java:1)
    at org.eclipse.scava.platform.delta.communicationchannel.PlatformCommunicationChannelManager.getDelta(PlatformCommunicationChannelManager.java:103)
    at org.eclipse.scava.platform.delta.communicationchannel.CommunicationChannelProjectDelta.<init>(CommunicationChannelProjectDelta.java:28)
    at org.eclipse.scava.platform.delta.ProjectDelta.create(ProjectDelta.java:69)
    at org.eclipse.scava.platform.osgi.analysis.ProjectAnalyser.executeAnalyse(ProjectAnalyser.java:103)
    at org.eclipse.scava.platform.osgi.services.TaskExecutor.run(TaskExecutor.java:26)
    at java.lang.Thread.run(Thread.java:748)
ERROR [ProjectExecutor (w1:iotmosquitto:aaamosqu)] (14:06:24): Project delta creation failed. Aborting.
INFO  [WorkerService] (14:06:24): Completing AnalysisTask on Worker 'w1'
INFO  [WorkerService] (14:06:24): Interrupting  AnalysisTask 'iotmosquitto:aaamosqu' on Worker 'w1
INFO  [WorkerExecutor] (14:06:33): Worker 'w1' Waiting new Tasks
INFO  [WorkerExecutor] (14:06:38): Worker 'w1' Waiting new Tasks
INFO  [WorkerExecutor] (14:06:43): Worker 'w1' Waiting new Tasks
creat89 commented 4 years ago

I'll fix the fue asap.

creat89 commented 4 years ago

The issue again is a bad comparison but on another part of the code.

valeriocos commented 4 years ago

ok, thank you for the quick reply!

creat89 commented 4 years ago

Commit https://github.com/crossminer/scava/commit/15901538bf7ef7b0c0c6402d10152acf35c6e0f6 should fix this. The problem was a comparison between null and "null" :/

valeriocos commented 4 years ago

Almost :)

ERROR [RascalMetricProvider (Java cohesion)] (10:12:31): Trying to use results from trans.rascal.OO.java.LCOM4-Java but no result was found. Ignoring parameter lcom4
INFO  [ProjectExecutor (w1:iotmosquitto:aaamosqu)] (10:12:31): Updating last executed date.
INFO  [ProjectExecutor (w1:iotmosquitto:aaamosqu)] (10:12:31): Date 20180110 Task Execution ( iotmosquitto:aaamosqu completed in 58478 ms )
INFO  [ProjectExecutor (w1:iotmosquitto:aaamosqu)] (10:12:31): Project iotmosquittoTask execution ( iotmosquitto:aaamosqu : Date 20180111 )
INFO  [AnalysisSchedulingService] (10:12:31): Starting new daily execution AnalysisTask 'iotmosquitto:aaamosqu'
INFO  [ProjectExecutor (w1:iotmosquitto:aaamosqu)] (10:12:31): Date: 20180111, project: Eclipse Mosquitto
INFO  [ProjectDelta (Eclipse Mosquitto,20180111)] (10:12:31): Creating Delta
                -page 1
ERROR [ProjectDelta (Eclipse Mosquitto,20180111)] (11:01:46): Delta creation failed.
java.lang.NullPointerException
        at org.eclipse.scava.platform.communicationchannel.eclipseforums.EclipseForumsManager.processPosts(EclipseForumsManager.java:310)
        at org.eclipse.scava.platform.communicationchannel.eclipseforums.EclipseForumsManager.getPosts(EclipseForumsManager.java:247)
        at org.eclipse.scava.platform.communicationchannel.eclipseforums.EclipseForumsManager.getDelta(EclipseForumsManager.java:133)
        at org.eclipse.scava.platform.communicationchannel.eclipseforums.EclipseForumsManager.getDelta(EclipseForumsManager.java:1)
        at org.eclipse.scava.platform.delta.communicationchannel.PlatformCommunicationChannelManager.getDelta(PlatformCommunicationChannelManager.java:103)
        at org.eclipse.scava.platform.delta.communicationchannel.CommunicationChannelProjectDelta.<init>(CommunicationChannelProjectDelta.java:28)
        at org.eclipse.scava.platform.delta.ProjectDelta.create(ProjectDelta.java:69)
        at org.eclipse.scava.platform.osgi.analysis.ProjectAnalyser.executeAnalyse(ProjectAnalyser.java:103)
        at org.eclipse.scava.platform.osgi.services.TaskExecutor.run(TaskExecutor.java:26)
        at java.lang.Thread.run(Thread.java:748)
ERROR [ProjectExecutor (w1:iotmosquitto:aaamosqu)] (11:01:46): Project delta creation failed. Aborting.
INFO  [WorkerService] (11:01:46): Completing AnalysisTask on Worker 'w1'
INFO  [WorkerService] (11:01:46): Interrupting  AnalysisTask 'iotmosquitto:aaamosqu' on Worker 'w1

Could you test the next fix with https://eclipse.org/mosquitto (details about the task are at: http://crossminer.bitergia.net:5601/#/project/configure/iotmosquitto)? thanks

creat89 commented 4 years ago

So, I have found the reason and it is related to the number of calls allowed per hour. I will try to improve the current implementation of checking the validity of calls.

valeriocos commented 4 years ago

ok, ping me when I can check it again, thanks

creat89 commented 4 years ago

I have modified the program that was charged of dealing with the number of calls supported by Eclipse Forums. Hopefully with the current version, we will not have to deal with the previous errors, as the code will retry the call if something went wrong and we need to wait for more calls.

creat89 commented 4 years ago

I have modified again the code and now I add two more minutes to the time that we need to wait for getting new calls. I have tested it as far but it was complex, as I have to wait one clock hour, rather than one hour since execution. I have modified the code as well to check if the call is negative numbers. Hopefully the two extra minutes will solve the issue and if we need more time, the negative checker will catch the issue and force an extra hour of waiting.

valeriocos commented 4 years ago

The fix seems to work, the fetching of iotmosquitto eclipse forum is advancing. In the logs there are some NPE (see below), but shouldn't be a problem, right @creat89 ?

ERROR [MetricListExecutor (iotmosquitto, 20180317)] (09:06:56): Exception thrown during metric provider execution (trans.newsgroups.sentiment).
java.lang.NullPointerException
        at org.eclipse.scava.metricprovider.trans.newsgroups.sentiment.SentimentTransMetricProvider.measure(SentimentTransMetricProvider.java:142)
        at org.eclipse.scava.metricprovider.trans.newsgroups.sentiment.SentimentTransMetricProvider.measure(SentimentTransMetricProvider.java:1)
        at org.eclipse.scava.platform.osgi.analysis.MetricListExecutor.run(MetricListExecutor.java:103)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
INFO  [ProjectExecutor (w1:iotmosquitto:teststs)] (09:06:56): Executing factoids.

ERROR [MetricListExecutor (iotmosquitto, 20180318)] (09:07:56): Exception thrown during metric provider execution (factoid.newsgroups.size).
java.lang.NullPointerException
        at org.eclipse.scava.factoid.newsgroups.size.NewsgroupsChannelSizeFactoid.measureImpl(NewsgroupsChannelSizeFactoid.java:131)
        at org.eclipse.scava.platform.AbstractFactoidMetricProvider.measure(AbstractFactoidMetricProvider.java:68)
        at org.eclipse.scava.platform.AbstractFactoidMetricProvider.measure(AbstractFactoidMetricProvider.java:1)
        at org.eclipse.scava.platform.osgi.analysis.MetricListExecutor.run(MetricListExecutor.java:103)
        at org.eclipse.scava.platform.osgi.analysis.ProjectAnalyser.executeAnalyse(ProjectAnalyser.java:148)
        at org.eclipse.scava.platform.osgi.services.TaskExecutor.run(TaskExecutor.java:26)
        at java.lang.Thread.run(Thread.java:748)
WARN  [ProjectExecutor (w1:iotmosquitto:teststs)] (09:07:56): Project in error state. Resuming execution.
creat89 commented 4 years ago

I'm glad that the reader is now working. But I am not glad that there are some npe. I need to check some issues that Martin asked me to check. Then I'll fix those issues in the cc.

creat89 commented 4 years ago

Well, some metrics and factoids might not have the correct values.

valeriocos commented 4 years ago

thanks @creat89

creat89 commented 4 years ago

I will close the issue because the reader works now and the I couldn't replicate the issues regarding the metrics. Feel free to open new issues.

valeriocos commented 4 years ago

thanks @creat89 ! @borisbaldassari feel free to test it on your side