search

q-rapids / qrapids-dashboard

Strategic Dashboard supporting decision-makers managing Quality Requirements, the RESTful API documentation is available at the next URL
https://q-rapids.github.io/qrapids-dashboard/
Apache License 2.0
6 stars 4 forks source link

Strategic Indicators are not calculated #38

Closed aabherve closed 5 years ago

aabherve commented 5 years ago

Component : qr-dashboard Version : v0.3.2

The strategic indicators are not calculated since the deployment of the last version of qr-dashboard.

An error is return by the REST Service used to trig the calculation

http://ip:8080/qr-dashboard/api/assessStrategicIndicators?prj=modelio_ng&from=25-03-2019

URL /error Status: 400 I/O error on GET request for "http://localhost:8081/api/Train": Connexion refusée (Connection refused); nested exception is java.net.ConnectException: Connexion refusée (Connection refused)

http://ip:8080/qr-dashboard/api/assessStrategicIndicators URL /error Status: 400 I/O error on GET request for "http://localhost:8081/api/Train": Connexion refusée (Connection refused); nested exception is java.net.ConnectException: Connexion refusée (Connection refused)

lidix91 commented 5 years ago

It is related to training the prediction models.

aabherve commented 5 years ago

i confirm that i have deployment issue with the qr-forecast service. I will try to redeploy it.

lidix91 commented 5 years ago

Remember that you need a new version of the R script. I will send the script by mail in order to check if you have the last version.

lidix91 commented 5 years ago

Can you check the forecast.url in the application.properties. The endpoint you included in your first comment is http://localhost:8081/api/Train (Train is one of the qr-forecast services).

It seems that something is missing in this endpoint http://localhost:8081/**qr-forecast/**api/Train

aabherve commented 5 years ago

After have :

forecast.url=http://localhost:8080/qr-forecast-rest-0.2
assessSI.url=http://localhost:8080/qr-si_assessment-rest-0.1

I have the following exception in tomcat logs when i try to :

org.rosuda.REngine.REngineException: eval failed at org.rosuda.REngine.Rserve.RConnection.parseAndEval(RConnection.java:454) at org.rosuda.REngine.REngine.parseAndEval(REngine.java:108) at Forecast.Common.evaluateR(Common.java:35) at Forecast.Elastic_RForecast.sourceTimeSeriesFunctions(Elastic_RForecast.java:145) at Forecast.Elastic_RForecast.initializeR(Elastic_RForecast.java:125) at Forecast.Elastic_RForecast.(Elastic_RForecast.java:71) at com.qrapids.forecast_rest.configuration.Connection.getConnection(Connection.java:77) at com.qrapids.forecast_rest.services.ForecastService.train(ForecastService.java:33) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498)

AleixBalletbo commented 5 years ago

Since the error is related with the training for the forecast, we need to make sure that the strategic indicators are not being calculated and saved before the error occurs. Please, could you check your ElasticSearch index and confirm that there are not strategic indicators saved for the dates when you performed the assessment through the dashboard API?

Thank you.

martimanzano commented 5 years ago

The error seems to be on R side, when loading the new scripts file. As there is a new package used in the script, can you try to manually install the new 'prophet' required R package to see if that's the cause of the error?

The R command to install it would be: install.packages("prophet")

aabherve commented 5 years ago

@AleixBalletbo : The last index in kibana is date of 24-03-2019 ip:8080/qr-dashboard/api/assessStrategicIndicators?prj=modelio_ng&from=26-03-2019 tri the exception presented below

@martimanzano :

I have reinitialize the RServer and install manually the list of dependencies presented in the wiki :

install.packages("Rserve") install.packages("forecast") install.packages("elastic") install.packages("prophet") install.packages("forecastHybrid")

I have now a new exception when i try to :

EVAL OK: source("/home/logadmine/QRapids/apache-tomcat-8.5.23/data/Rscript/NewTimeSeriesFunctions.R")
TIME SERIES FUNCTIONS SOURCED
ERROR: R Serve Eval Exception : Error in inherits(x, "Elasticsearch") : 
  l'argument "conn" est manquant, avec aucune valeur par défaut

org.rosuda.REngine.REXPMismatchException: attempt to access org.rosuda.REngine.REXPString as Error in inherits(x, "Elasticsearch") : 
  l'argument "conn" est manquant, avec aucune valeur par défaut

    at Forecast.Common.evaluateR(Common.java:43)
    at Forecast.Elastic_RForecast.initESConnection(Elastic_RForecast.java:156)
    at Forecast.Elastic_RForecast.initializeR(Elastic_RForecast.java:126)
martimanzano commented 5 years ago

Unfortunately, the R elastic package has been updated with a breaking change (https://cran.r-project.org/web/packages/elastic/news/news.html). For the moment, the easiest solution would be installing the 0.8.4 version of the package and check if everything works.

The R commands to achieve it are: remove.packages("elastic") install.packages("devtools") require(devtools) install_version("elastic", version="0.8.4")

aabherve commented 5 years ago

After the installation of this new packages in R,i have no more exception in tomcat log and i can see some entry related to predictions

Key: mitigationtaskcompletion, Doc Count: 38
ID: modelio_ng-mitigationtaskcompletion-2019-04-25, Source: {"evaluationDate":"2019-04-25","metric":"mitigationtaskcompletion","name":"Mitigation Task Completion","project":"modelio_ng","description":"\"Pecentage of mitigation action derviated from quality requirement which has bean completed\"","source":null,"type":"metrics","weights":[1.0],"value":0.75,"factors":["qualityrequirementcompletion"],"info":"parameters: {totalCommitsRelatedToIssue=0, lastSnapshotDate=2019-04-24, totalSpecificationTask=3, totalMantisCommonlIssue=166, totalDevelomenetTask=15, filesInSnapshot=3042, totalMantisCriticalIssue=97}\nquery-properties: {projectName=ModelioNG, periode=now-1d/d}\nexecutionResults: {derived_qualityrequirement=3.0, closed_qualityrequirement=2.0}\nformula: (closed_qualityrequirement + 1) / (derived_qualityrequirement + 1)\nvalue: 0.75"}

...

FITTING ARIMA MODEL, FORCING SEASONALITY
EVAL OK: trainArimaModel(name = "activitycompletion", index = "factors.modelio_ng", forceSeasonality = TRUE, frequencyts = 7)
FITTING ARIMA MODEL, FORCING SEASONALITY
EVAL OK: trainArimaModel(name = "softwarestability", index = "factors.modelio_ng", forceSeasonality = TRUE, frequencyts = 7)
FITTING ARIMA MODEL, FORCING SEASONALITY

But but strategic indicators history is still not calculated ... :(

lidix91 commented 5 years ago

"But but strategic indicators history is still not calculated ... :(" Have you call directly from the browers using the parameter "from"?

aabherve commented 5 years ago

Yes : http://10.78.1.18:8080/qr-dashboard/api/assessStrategicIndicators?prj=modelio_ng&from=26-03-2019

I noticed something strange : for one SI on "Detailled Strategic Indicator" view, the last value display for the factor is still blocked as the same date of last calculated SI (24-03).

However ,data related to this factors its still available until today (and visible in another SI)

Sélection_003

I will try to delete this SI, it may be it who fails the calculation of others SI

lidix91 commented 5 years ago

We will check it, the problem could be with Tasks' Velocity (in Management Quality SI) value for day March 26th.

Could you check the content of that factor? Maybe there is a NaN and the chart is failling.

We should check if the 0z are real 0z. Could you check if there is a 0, a NaN, of if there is no entry for any of the days the chart is showing a 0?

aabherve commented 5 years ago

There some 0 values (27-03 to 30-03: collectors were stop during the deployment of the new version of the platform) but it's also the case for the others factors ...

One specificity of this SI : it was created by Alessandra and use factors which are involve in calculation of others SI

aabherve commented 5 years ago

I have solve my issue ... it was all my fault. I have delete the SI form postgres database during the last migration ... The SI were still visible because of old calculated value in elastic but new values were not computed ...

Sorry for the time lost on this issue ...