Sonar gitlab not working with Python 3.6 ? #82

Open laurentnguyenwalnut opened 7 years ago

Hello Gabriel,

It seems that when I install the SonarGitLab plugin, it messes up with Python 3.6 syntax parsing (See console output below). We are just starting with Python 3.6 in my team, and the only new feature of the language we use as of now is the "type hint". This type of code is is correctly analyzed by Sonar when the GitLab sonar plugin is uninstalled.

Is this something that has been reported to you before? Do you have any advice on the matter?

Thanks.

ERROR: Unable to parse file: /walnut-project/walnut-finance/backtest/wa_backtest/environnement/live_backtest_comparison_session_builder.py ERROR: Parse error at line 18 column 9:

13: str, Any]): 14: self.configurationManager = configurationManager 15: self.overwriteArgs = overwriteArgs 16: 17: def buildSession(self): --> s: Session = Session() 19: e: Experiment = Experiment()
20: Session().experiment.generalParameters.update() 21: models: List

Hi, Do you get me full log ? Thanks

sonar-logs.tar.gz Please find all the logs I have attached. If you need me to increase the log level, please tell me. Thanks for the help.

Laurent

Sorry, I wanted the logs of the compilation

snapshot log

Running with gitlab-runner 10.0.2 (a9a76a50) on gitlab-runner-gitlab-runner-189447667-5cqzn (2c076538) Using Kubernetes namespace: default Using Kubernetes executor with image newtmitch/sonar-scanner:3.0.3-alpine ... Waiting for pod default/runner-2c076538-project-132-concurrent-1jznzc to be running, status is Pending Running on runner-2c076538-project-132-concurrent-1jznzc via gitlab-runner-gitlab-runner-189447667-5cqzn... Cloning repository... Cloning into '/walnut-project/walnut-finance'... Checking out f9ccf085 as typechecking_sonar... Skipping Git submodules setup $ echo ${CI_COMMIT_REF_SLUG} typechecking-sonar $ echo ${CONTAINER_REGISTRY_PATH}

$ echo ${CI_COMMIT_SHA:0:8} f9ccf085 $ echo ${CI_JOB_ID} 27523 $ sonar-scanner -Dsonar.projectKey=walnut-finance -Dsonar.projectBaseDir="$PWD" -Dsonar.sources=. -Dsonar.host.url=http://13.80.23.56 -Dsonar.login=6a677b588872955ce223ca651a0625e7ad05e464 -Dsonar.projectName=walnut-finance -Dsonar.projectVersion="$CI_JOB_ID" INFO: Scanner configuration file: /root/sonar-scanner-3.0.3.778-linux/conf/sonar-scanner.properties INFO: Project root configuration file: NONE INFO: SonarQube Scanner 3.0.3.778 INFO: Java 1.8.0_131 Oracle Corporation (64-bit) INFO: Linux 4.4.64+ amd64 INFO: User cache: /root/.sonar/cache INFO: Load global settings INFO: Load global settings (done) | time=129ms WARN: Property 'sonar.jdbc.url' is not supported any more. It will be ignored. There is no longer any DB connection to the SQ database. INFO: User cache: /root/.sonar/cache INFO: Load plugins index INFO: Load plugins index (done) | time=29ms INFO: Download sonar-python-plugin-1.8.0.1496.jar INFO: Download sonar-csharp-plugin-6.4.1.3596.jar INFO: Download sonar-java-plugin-4.14.0.11784.jar INFO: Download sonar-scm-git-plugin-1.2.jar INFO: Download sonar-xml-plugin-1.4.3.1027.jar INFO: Download sonar-flex-plugin-2.3.jar INFO: Download sonar-php-plugin-2.10.0.2087.jar INFO: Download sonar-scm-svn-plugin-1.5.0.715.jar INFO: Download sonar-javascript-plugin-3.2.0.5506.jar INFO: SonarQube server 6.5.0 INFO: Default locale: "en_US", source code encoding: "UTF-8" (analysis is platform dependent) INFO: Process project properties INFO: Load project repositories INFO: Load project repositories (done) | time=203ms INFO: Load quality profiles INFO: Load quality profiles (done) | time=56ms INFO: Load active rules INFO: Load active rules (done) | time=748ms INFO: Load metrics repository INFO: Load metrics repository (done) | time=88ms INFO: Publish mode INFO: Project key: walnut-finance INFO: ------------- Scan walnut-finance INFO: Load server rules INFO: Load server rules (done) | time=78ms INFO: Base dir: /walnut-project/walnut-finance INFO: Working dir: /walnut-project/walnut-finance/.scannerwork INFO: Source paths: . INFO: Source encoding: UTF-8, default locale: en_US INFO: Index files INFO: Excluded sources: INFO: /*.h5 INFO: */.sh INFO: /*.xlsx INFO: */.jar INFO: 1233 files indexed INFO: 53 files ignored because of inclusion/exclusion patterns INFO: Quality profile for py: Sonar way INFO: Quality profile for xml: Sonar way INFO: Sensor PythonXUnitSensor [python] INFO: Sensor PythonXUnitSensor [python] (done) | time=60ms INFO: Sensor Python Squid Sensor [python] ERROR: Unable to parse file: /walnut-project/walnut-finance/backtest/wa_backtest/environnement/live_backtest_comparison_session_builder.py ERROR: Parse error at line 18 column 9:

13: str, Any]): 14: self.configurationManager = configurationManager 15: self.overwriteArgs = overwriteArgs 16: 17: def buildSession(self): --> s: Session = Session() 19: e: Experiment = Experiment()
20: Session().experiment.generalParameters.update() 21: models: List

ERROR: Unable to parse file: /walnut-project/walnut-finance/backtest/wa_backtest/tests/test_experiment_parameters.py ERROR: Parse error at line 30 column 9:

28: app_settings()['live-config']['model-config']['models'] 29: allocator = ConfigurationManager().app_settings()['live-config']['allocator'] --> p: ParsedData = ParsedData({'models': models, 'allocator': allocator}) 31: Session().experiment.backtesterParameters.update(p) 32: 33: self

ERROR: Unable to parse file: /walnut-project/walnut-finance/backtest/wa_backtest/tests/test_live_backtest_comparison_session_builder.py ERROR: Parse error at line 18 column 22:

8: Session
9: from wa_trading.allocators.cashalot import Cashalot

10: 11: 12: class TestLiveBacktestComparisonSessionBuilder(TestCase): 13: """ 14: Tests for the backtester session builder 15: """ 16: 17: def testBuildSession(self): --> envTypeManager: EnvTypeManager = EnvTypeManager() 19: envTypeManager.get_env_type = MagicMock(return_value='local') 20: 21: configurationManager: ConfigurationManager = ConfigurationManager(envTypeManager) 22: overwriteDict: Dict

ERROR: Unable to parse file: /walnut-project/walnut-finance/common/wa_common/tests/data_io/test_arctic_connector.py ERROR: Parse error at line 17 column 18:

9: wa_common.data_io import ArcticConnector

10: from wa_common.configuration_manager import ConfigurationManager 11: 12: 13: 14: class TestDatabaseConnector(TestCase):
15: def setUp(self): 16: --> arcticHost: str = ConfigurationManager().app_settings()['service-config']['arctic']['host'] 18: arcticPort: int = ConfigurationManager().app_settings(

ERROR: Unable to parse file: /walnut-project/walnut-finance/live/historical_candle_farmer.py ERROR: Parse error at line 24 column 21:

18: : 19: try: 20: cmdLineArgs_l = FarmerCmdLineParser().cmdLineArgs 21: sleep_time = cmdLineArgs_l['sleep'] 22: time.sleep(sleep_time) 23: --> emailSettings: Dict[str, str] = ConfigurationManager().app_settings()['service-config']['email'] 25: LoggingHandler.configureLogging(os.path.

ERROR: Unable to parse file: /walnut-project/walnut-finance/live/live.py ERROR: Parse error at line 61 column 14:

49: 50: """ 51: Verify that Candle-Farmer Historical-Data is up-to-date and consistent. 52: """ 53: return HistoricalDataValidator(self.resultFolder).run() 54: 55: def runContractValidator(self, parameters_p): 56: """ 57: Verify that the IB_SYMBOL tables is well-formed, ie that all contracts defined in this table are valid 58: and that IB returns a valid instance for each contract 59: """ 60: --> ibHost: str = ConfigurationManager().app_settings()['service-config']['ib']['host'] 62: ibPort: int = ConfigurationManager().app_settings(

ERROR: Unable to parse file: /walnut-project/walnut-finance/live/wa_live/ib/ib_contract_validator.py ERROR: Parse error at line 39 column 21:

26: )) 27: 28: def stop(self): 29: self.disconnect() 30: 31: def validateContracts(self): 32: """ 33: Logs all information fetched from IB for every existing Contract in database 34: It returns True on success, False if at least one request failed when querying IB for information 35: 36: This method might be a blocking-call, depending on the TIMEOUT_ON_RECEIVE value 37: """ 38: --> dbConnector_l: DatabaseConnector = DatabaseConnector() 40: try: 41: dbConnector_l.select("IB_SYMBOL", fields_p=["ID"]) 42: except Exception as ex_l:

ERROR: Unable to parse file: /walnut-project/walnut-finance/live/wa_live/tests/backtest_comparison/test_live_backtest_comparison.py ERROR: Parse error at line 33 column 25:

24: 25: 26: def tearDown(self): 27: Session().reset() 28: 29: def testLiveBacktestComparison1(self): 30: """ 31: Run a backtest with the models and execute a comparison 32: """ --> mockConfiguration: Dict[str, Any] = ConfigurationManager().app_settings() 34: mockConfiguration["live-config"]['model-config']['models']: List[str

ERROR: Unable to parse file: /walnut-project/walnut-finance/mkdata/wa_mkdata/tests/pipeline/test_arctic_candle_and_piecewise_exporter.py ERROR: Parse error at line 21 column 18:

13: from .test_utils import generatePiecewiseCandleCurve 14: from wa_common.configuration_manager import ConfigurationManager 15: 16: 17: class TestArcticCanldeAndPiecewiseExporter(TestCase): 18: 19: def setUp(self): 20: --> arcticHost: str = ConfigurationManager().app_settings()['service-config']['arctic']['host'] 22: arcticPort: int = ConfigurationManager().app_settings(

ERROR: Unable to parse file: /walnut-project/walnut-finance/mkdata/wa_mkdata/tests/pipeline/test_hdf_piecewise_exporter.py ERROR: Parse error at line 20 column 35:

12: .session import Session 13: from .test_utils import generatePiecewiseCandleCurve 14: 15: 16: class TestHDFPiecewiseExporter(TestCase): 17: 18: def setUp(self): 19: --> self.configurationManager: ConfigurationManager = ConfigurationManager() 21: self.fake_app_settings = app_settings = self.__configurationManager.app_settings() 22: self.__fake_app_settings["service-config"]["raw-data-folder"]

INFO: Python unit test coverage INFO: Python integration test coverage INFO: Python overall test coverage INFO: Sensor Python Squid Sensor [python] (done) | time=13508ms INFO: Sensor SonarJavaXmlFileSensor [java] INFO: 2 source files to be analyzed INFO: 2/2 source files have been analyzed INFO: Sensor SonarJavaXmlFileSensor [java] (done) | time=411ms INFO: Sensor XML Sensor [xml] INFO: Sensor XML Sensor [xml] (done) | time=139ms INFO: Sensor Analyzer for "php.ini" files [php] INFO: Sensor Analyzer for "php.ini" files [php] (done) | time=7ms INFO: Sensor Zero Coverage Sensor INFO: Sensor Zero Coverage Sensor (done) | time=302ms INFO: Sensor CPD Block Indexer INFO: Sensor CPD Block Indexer (done) | time=1296ms INFO: SCM provider for this project is: git INFO: 3 files to be analyzed WARN: Missing blame information for the following files: WARN: /walnut-project/walnut-finance/backtest/wa_test_models/stop_order_model/simple_trading_model_with_stop_order.py WARN: /walnut-project/walnut-finance/backtest/wa_test_models/dummy_model/dummy_trading_model.py WARN: * /walnut-project/walnut-finance/backtest/wa_test_models/limit_order_model/simple_trading_model_with_limit_order.py WARN: This may lead to missing/broken features in SonarQube INFO: 0/3 files analyzed INFO: 245 files had no CPD blocks INFO: Calculating CPD for 770 files INFO: CPD calculation finished INFO: Analysis report generated in 934ms, dir size=5 MB INFO: Analysis reports compressed in 1351ms, zip size=2 MB INFO: Analysis report uploaded in 388ms INFO: ANALYSIS SUCCESSFUL, you can browse http://13.80.23.56/dashboard/index/walnut-finance INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report INFO: More about the report processing at http://13.80.23.56/api/ce/task?id=AV-BanuTg1yiH5V_o5t9 INFO: Task total time: 24.136 s INFO: ------------------------------------------------------------------------ INFO: EXECUTION SUCCESS INFO: ------------------------------------------------------------------------ INFO: Total time: 28.548s INFO: Final Memory: 65M/973M INFO: ------------------------------------------------------------------------ Job succeeded

Ok, and log with sonar-gitlab-plugin ?

It appears that event when I uninstall the Gitlab-sonar plugin, I keep having the parsing error, but before I installed it, I didn't get those errors. I guess something was installed when I installed the plugin (with sonar interface, in the available plugins) but not removed when I uninstalled.

Reinstalling right now.

When you restarted sonar, did you have to update the python plugin?

Not that I know of. I didn't do it manually at least, and as I recall, the SonarPython version didn't change. I could reinstall everything from scratch if you need me to really confirm this. But I will only be able to do it this evening.

I cannot reproduce the issue in my test environmentfor the sonar preview right now, but here is the original log that made me drill into the issue

Running with gitlab-runner 10.0.2 (a9a76a50) on gitlab-runner-gitlab-runner-189447667-9ztw5 (d399fd16) Using Kubernetes namespace: default Using Kubernetes executor with image maven:3.5.0-jdk-8-alpine ... Waiting for pod default/runner-d399fd16-project-132-concurrent-3c3m1q to be running, status is Pending Waiting for pod default/runner-d399fd16-project-132-concurrent-3c3m1q to be running, status is Pending Running on runner-d399fd16-project-132-concurrent-3c3m1q via gitlab-runner-gitlab-runner-189447667-9ztw5... Cloning repository... Cloning into '/walnut-project/walnut-finance'... Checking out 5ec34a1f as typechecking_sonar... Skipping Git submodules setup $ echo ${CI_COMMIT_REF_SLUG} typechecking-sonar $ echo ${CONTAINER_REGISTRY_PATH}

13: str, Any]): 14: self.configurationManager = configurationManager 15: self.overwriteArgs = overwriteArgs 16: 17: def buildSession(self): --> s: Session = Session() 19: e: Experiment = Experiment()
20: Session().experiment.generalParameters.update() 21: models: List

[ERROR] Unable to parse file: /walnut-project/walnut-finance/backtest/wa_backtest/tests/test_experiment_parameters.py [ERROR] Parse error at line 30 column 9:

28: app_settings()['live-config']['model-config']['models'] 29: allocator = ConfigurationManager().app_settings()['live-config']['allocator'] --> p: ParsedData = ParsedData({'models': models, 'allocator': allocator}) 31: Session().experiment.backtesterParameters.update(p) 32: 33: self

[ERROR] Unable to parse file: /walnut-project/walnut-finance/backtest/wa_backtest/tests/test_live_backtest_comparison_session_builder.py [ERROR] Parse error at line 18 column 22:

8: Session
9: from wa_trading.allocators.cashalot import Cashalot

10: 11: 12: class TestLiveBacktestComparisonSessionBuilder(TestCase): 13: """ 14: Tests for the backtester session builder 15: """ 16: 17: def testBuildSession(self): --> envTypeManager: EnvTypeManager = EnvTypeManager() 19: envTypeManager.get_env_type = MagicMock(return_value='local') 20: 21: configurationManager: ConfigurationManager = ConfigurationManager(envTypeManager) 22: overwriteDict: Dict

[ERROR] Unable to parse file: /walnut-project/walnut-finance/common/wa_common/tests/data_io/test_arctic_connector.py [ERROR] Parse error at line 17 column 18:

9: wa_common.data_io import ArcticConnector

10: from wa_common.configuration_manager import ConfigurationManager 11: 12: 13: 14: class TestDatabaseConnector(TestCase):
15: def setUp(self): 16: --> arcticHost: str = ConfigurationManager().app_settings()['service-config']['arctic']['host'] 18: arcticPort: int = ConfigurationManager().app_settings(

[ERROR] Unable to parse file: /walnut-project/walnut-finance/live/historical_candle_farmer.py [ERROR] Parse error at line 24 column 21:

18: : 19: try: 20: cmdLineArgs_l = FarmerCmdLineParser().cmdLineArgs 21: sleep_time = cmdLineArgs_l['sleep'] 22: time.sleep(sleep_time) 23: --> emailSettings: Dict[str, str] = ConfigurationManager().app_settings()['service-config']['email'] 25: LoggingHandler.configureLogging(os.path.

[ERROR] Unable to parse file: /walnut-project/walnut-finance/live/live.py [ERROR] Parse error at line 61 column 14:

49: 50: """ 51: Verify that Candle-Farmer Historical-Data is up-to-date and consistent. 52: """ 53: return HistoricalDataValidator(self.resultFolder).run() 54: 55: def runContractValidator(self, parameters_p): 56: """ 57: Verify that the IB_SYMBOL tables is well-formed, ie that all contracts defined in this table are valid 58: and that IB returns a valid instance for each contract 59: """ 60: --> ibHost: str = ConfigurationManager().app_settings()['service-config']['ib']['host'] 62: ibPort: int = ConfigurationManager().app_settings(

[ERROR] Unable to parse file: /walnut-project/walnut-finance/live/wa_live/ib/ib_contract_validator.py [ERROR] Parse error at line 39 column 21:

26: )) 27: 28: def stop(self): 29: self.disconnect() 30: 31: def validateContracts(self): 32: """ 33: Logs all information fetched from IB for every existing Contract in database 34: It returns True on success, False if at least one request failed when querying IB for information 35: 36: This method might be a blocking-call, depending on the TIMEOUT_ON_RECEIVE value 37: """ 38: --> dbConnector_l: DatabaseConnector = DatabaseConnector() 40: try: 41: dbConnector_l.select("IB_SYMBOL", fields_p=["ID"]) 42: except Exception as ex_l:

[ERROR] Unable to parse file: /walnut-project/walnut-finance/live/wa_live/tests/backtest_comparison/test_live_backtest_comparison.py [ERROR] Parse error at line 33 column 25:

24: 25: 26: def tearDown(self): 27: Session().reset() 28: 29: def testLiveBacktestComparison1(self): 30: """ 31: Run a backtest with the models and execute a comparison 32: """ --> mockConfiguration: Dict[str, Any] = ConfigurationManager().app_settings() 34: mockConfiguration["live-config"]['model-config']['models']: List[str

[ERROR] Unable to parse file: /walnut-project/walnut-finance/mkdata/wa_mkdata/tests/pipeline/test_arctic_candle_and_piecewise_exporter.py [ERROR] Parse error at line 21 column 18:

13: from .test_utils import generatePiecewiseCandleCurve 14: from wa_common.configuration_manager import ConfigurationManager 15: 16: 17: class TestArcticCanldeAndPiecewiseExporter(TestCase): 18: 19: def setUp(self): 20: --> arcticHost: str = ConfigurationManager().app_settings()['service-config']['arctic']['host'] 22: arcticPort: int = ConfigurationManager().app_settings(

[ERROR] Unable to parse file: /walnut-project/walnut-finance/mkdata/wa_mkdata/tests/pipeline/test_hdf_piecewise_exporter.py [ERROR] Parse error at line 20 column 35:

12: .session import Session 13: from .test_utils import generatePiecewiseCandleCurve 14: 15: 16: class TestHDFPiecewiseExporter(TestCase): 17: 18: def setUp(self): 19: --> self.configurationManager: ConfigurationManager = ConfigurationManager() 21: self.fake_app_settings = app_settings = self.__configurationManager.app_settings() 22: self.__fake_app_settings["service-config"]["raw-data-folder"]

[INFO] Python unit test coverage [INFO] Python integration test coverage [INFO] Python overall test coverage [INFO] Sensor Python Squid Sensor [python] (done) | time=3804ms [INFO] Sensor SonarJavaXmlFileSensor [java] [INFO] Sensor SonarJavaXmlFileSensor [java] (done) | time=15ms [INFO] Sensor Analyzer for "php.ini" files [php] [INFO] Sensor Analyzer for "php.ini" files [php] (done) | time=16ms [INFO] Load server issues [INFO] Load server issues (done) | time=998ms [INFO] Performing issue tracking [INFO] 204/1453 components tracked [INFO] 204/1453 components tracked [INFO] 1453/1453 components tracked [INFO] ANALYSIS SUCCESSFUL [INFO] Executing post-job GitLab Commit Issue Publisher [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 01:09 min [INFO] Finished at: 2017-11-02T10:02:42Z [INFO] Final Memory: 62M/1269M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.3.0.603:sonar (default-cli) on project walnut-finance: Report status=failed, desc=SonarQube reported 67 issues, with 2 critical (fail) and 3 major and 62 minor -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException ERROR: Job failed: error executing remote command: command terminated with non-zero exit code: Error executing in Docker Container: 1

Ok plugin fail because 2 critical issue

[ERROR] Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.3.0.603:sonar (default-cli) on project walnut-finance: Report status=failed, desc=SonarQube reported 67 issues, with 2 critical (fail) and 3 major and 62 minor -> [Help 1]

So I should not worry about the parse errors? These files are still going to be analyzed ?

The file problem is given by the python plugin. I don't know python.

gabrie-allaigre / sonar-gitlab-plugin

Sonar gitlab not working with Python 3.6 ? #82