search

leinardi / pylint-pycharm

A plugin providing both real-time and on-demand scanning of Python files with PyLint from within PyCharm/IDEA.
Apache License 2.0
197 stars 35 forks source link

Support linting inside current Virtualenv #7

Closed Routhinator closed 5 years ago

Routhinator commented 5 years ago

Is your feature request related to a problem? Please describe. Linter does not have a setting to support virtualenvs. PyLint ends up reporting valid imports as errors.

Describe the solution you'd like A setting to tell PyLint plugin to use the configured virtualenv, similar to https://github.com/perses76/pylint-pycharm.git so we can get proper reports from the linter.

Describe alternatives you've considered I can workaround this by not using this plugin and simply configuring an external tool with https://github.com/perses76/pylint-pycharm.git for now (setup is here for any curious: https://stackoverflow.com/a/48549144/2262288)

bivald commented 5 years ago

Amazing plugin!

I've worked around it locally by creating a pylint2 file which I point to and in it I add:

#!/path/to/bin/python3.4

# -*- coding: utf-8 -*-
import re
import sys
sys.path.append('/path/to/code')
from pylint import run_pylint

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(run_pylint())

But in reality I would love to see that it adds all content roots which are marked as sources to pylint, or ability to select which source roots to use.

bivald commented 5 years ago

That said, can't you simply install pylint in the virtualenv and point to this binary? This will still give you import errors for your own code, but for things installed in your venv it should work. Perhaps I'm missing something :)

Routhinator commented 5 years ago

I actully tried this and uninstalling pylint from the host system and installing pylint in the virtualenv does work so I am using this plugin now, however it feels a bit... dirty? to include a linter in my project requirements. However I suppose I can simply change my thinking here and have a requirements.txt and a requirements-prod.txt and maintain both files.

leinardi commented 5 years ago

Hey guys, I am trying to find a solution but I am a Java developer that started to mess with Python only recently, so I need help to fully understand the problem.

Currently I am able to get the path of the current environment used on the PyCharm project: ProjectRootManager.getInstance(project).getProjectSdk().getHomePath() will return the path to my venv, if I am using one. But how can I tell pylint to use this path?

The python tool you linked is sourcing the content of the venv path + "activate": https://github.com/tgamauf/pylint-pycharm/blob/ff16d80b894275623503bd0e6fe7a8a2903bb125/pylint_pycharm/converter.py#L132

Is this the only way?

leinardi commented 5 years ago

Ok, after some investigation it seems that the activate file is just changing 3 environmental variables:

  1. Adding VIRTUAL_ENV containing the virtualenv path
  2. unsetting PYTHONHOME if set
  3. PATH="$VIRTUAL_ENV/bin:$PATH"

I should be able to do those things directly from the plugin before running Pylint.

I will provide a test build soon and I would love is someone of you guys could test it.

bivald commented 5 years ago

I'm not sure if you need all that, or if you just can supply the binary to the virtualenv since that is the correct environment it (according to me) should run in. What would be nice would be if it can use your projects python interpreter (since that's often the virtualenv)

leinardi commented 5 years ago

OK, I think I fixed it.

@Routhinator @bivald @kwiersma, @switchtrue please test this build an let me know if is fixed also for you: pylint-plugin-0.9.0.zip

EDIT: I just find out that my fix only works if you specify the absolute path to pylint: if you test please go to the plugin settings and add the absolute path. EDIT2: Never mind, I have implemented some basic path auto-detection for *nix systems. Just download the plugin again to be sure to have the latest version.

switchtrue commented 5 years ago

This works perfectly!

leinardi commented 5 years ago

Hi all, unfortunately I just find out that the fix only works if you install Pylint in your virtualenv (@switchtrue can you please confirm this?).

I just noticed that, after setting up a venv from scratch, the plugin was not recognizing Pylint executable, even after providing the full path:

Traceback (most recent call last):
  File "/usr/local/bin/pylint", line 7, in <module>
    from pylint import run_pylint
ModuleNotFoundError: No module named 'pylint'

Any idea on how to fix this?

leinardi commented 5 years ago

Ok, my current plan is to make the plugin install Pylint inside the virtualenv if is not already installed. But I have no idea how to do it. I have asked for help here. Perhaps @vlasovskikh will have a look at it :)

switchtrue commented 5 years ago

Yes, I can confirm this!

Interesting. I always install pylint inside my virtualenvs as I use different versions for different project. As long as the auto install only installs if it's not already found then I should still be able to install whatever version I want outside of the plugin and have the plugin use it. Ideally, if its auto installed we should make it clear that it's happened and at which version.

switchtrue commented 5 years ago

Actually, looking at my config I've got things a little weird. I'm specifying the Path to Pylint executable as /usr/local/bin/pylint but its picking up the one inside my virtualenv at run time. Ideally I would just be able to specify pylint as my executable. If I do this though clicking "test" doesn't work and at run time it can't find it.

Also, I've pulled master and rebuilt and it doesn't seem to lint at all.

leinardi commented 5 years ago

I should still be able to install whatever version I want outside of the plugin and have the plugin use it I'm specifying the Path to Pylint executable as /usr/local/bin/pylint but its picking up the one inside my virtualenv at run time.

Yep, that also what I noticed yesterday: even if I invoke explicitly my global pylint executable, if I am using the venv interpreter, pylint will still run the venv version. And if there is no pylint installed I will get the error I posted in some comments above.

I think that the pylint executable is just using the pylint package installed in the current env and is not a standalone executable. So which version of pylint are you using depends entirely on which version is installed in the environment you are using.

@PCManticore if you have time, can you have a look at this issue?

Also, I've pulled master and rebuilt and it doesn't seem to lint at all.

I disabled the venv fix on master (see 1e4b300) because I wanted to release a new version with other urgent fixes.

leinardi commented 5 years ago

OK, I have a new fix for this issue. I implemented a better auto-detect for the Pylint executable and I can now install Pylint inside the current project interpreter if is missing.

Please test this new version and let me know if works fine (if possible test multiple scenarios like a fresh project with no venv set, then one with a venv and another with the system env).

pylint-plugin-0.9.0.zip

leinardi commented 5 years ago

Has anyone tested the new fix? Should I release it?

switchtrue commented 5 years ago

Yes, apologies! I did and I started drafting a comment here but got distracted.

It seems to work really well. The "Auto-detected: ..." message in the settings is a really nice touch.

One issue I've noticed though it that it is linting files I have ignored in my pylintrc file. In pylintrc you can add an ignore like like this:

# Add files or directories to the blacklist. They should be base names, not
# paths.
ignore=CVS,migrations,local_settings.py,manage.py,wsgi.py

If I run pylint manually these do not get linted, but if I run via this plugin they do get linted and in my setup they fail which is a bit of an issue. All other pylintrc settings seem to get respected so I'm guessing this is an artifact of how the files are passed to pylint. Do you want me to raise another issue for this?

leinardi commented 5 years ago

It seems to work really well.

Nice! Then I procede with a new release :+1:

One issue I've noticed though it that it is linting files I have ignored in my pylintrc file.

Mmm this could be hard to fix: the plugin is giving pylint a list of all the files it has to scan and I guess pylint is not respecting the pylintrc ignore if you ask to scan explicitly an ignored file...

I will open a separate bug to keep track of this issue but I don't see any easy solution...

cpchen commented 5 years ago

Hello, I have a issue with using a hardcoded pylint path that's in a virtualenv.

I'm using Intellij IDEA Ultimate 2018.2.4 on a Mac. The Pylint plugin is v0.10.2

I see:

Pylint Plugin (Unable to run Pylint) Pylint is installed inside the project environment but the plugin is not able to run it. If you just installed it try to restart your IDE, if the problem persists you may need to manually enter the path to the Pylint executable inside the Plugin settings. 

Pylint Plugin: Failure: executable "/Users/penc/virtualenvs/data-manager/bin/pylint" not found

in the log, even though the file is there and I can execute it directly. (I've also restarted the IDE many times.)

Any ideas?

leinardi commented 5 years ago

Hi @cpchen, could you please attach the idea.log file (https://intellij-support.jetbrains.com/hc/en-us/articles/207241085-Locating-IDE-log-files) after you get that error message?

cpchen commented 5 years ago 
2018-10-12 09:26:00,023 [  69537]  ERROR - harm.pylint.plapi.PylintRunner - Error while checking Pylint path 
com.intellij.execution.process.ProcessNotCreatedException: Cannot run program "/Library/Java/JavaVirtualMachines/jdk-10.0.2.jdk/Contents/Home": error=13, Permission denied
    at com.intellij.execution.configurations.GeneralCommandLine.createProcess(GeneralCommandLine.java:408)
    at com.leinardi.pycharm.pylint.plapi.PylintRunner.isPylintPathValid(PylintRunner.java:92)
    at com.leinardi.pycharm.pylint.plapi.PylintRunner.checkPylintAvailable(PylintRunner.java:176)
    at com.leinardi.pycharm.pylint.plapi.PylintRunner.checkPylintAvailable(PylintRunner.java:150)
    at com.leinardi.pycharm.pylint.PylintInspection.inspectFile(PylintInspection.java:76)
    at com.leinardi.pycharm.pylint.PylintInspection.lambda$checkFile$0(PylintInspection.java:65)
    at com.intellij.openapi.application.impl.ApplicationImpl$2.call(ApplicationImpl.java:337)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Cannot run program "/Library/Java/JavaVirtualMachines/jdk-10.0.2.jdk/Contents/Home": error=13, Permission denied
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
    at com.intellij.execution.configurations.GeneralCommandLine.startProcess(GeneralCommandLine.java:436)
    at com.intellij.execution.configurations.GeneralCommandLine.createProcess(GeneralCommandLine.java:404)
    ... 10 more
Caused by: java.io.IOException: error=13, Permission denied
    at java.lang.UNIXProcess.forkAndExec(Native Method)
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
    at java.lang.ProcessImpl.start(ProcessImpl.java:134)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
    ... 12 more
leinardi commented 5 years ago

Mmm this looks strange, can you check if the Python interpreter is properly set in your PyCharm project? Can you try to switch to another one (e.g. local venv) to test if you get the same error?

cpchen commented 5 years ago

The Python interpreter is set to the one inside the venv the Pylint bin is in. I can try another venv, if you think that would make a difference?

leinardi commented 5 years ago

If you can post the entire idea.log here (you can use the attach feature), there is a problem with the path "/Library/Java/JavaVirtualMachines/jdk-10.0.2.jdk/Contents/Home" and I don't understand from where is coming. I think the plugin is recognizing that as the python interpreter for some reason.

cpchen commented 5 years ago

idea.log

leinardi commented 5 years ago

Yep, for some reason you interpreter configuration is returning your Home as the path of the interpreter.

Please try to remove the interpreter configuration from your project and re-add it making sure that the base interpreter is pointing to a python bin:

image

cpchen commented 5 years ago

I'm using IntelliJ and the project is a Java project with a Python module. I don't see a global project interpreter I can set here, but this is my configuration (the 10 folder is the Java JDK):

image

leinardi commented 5 years ago

I am sorry but I fear this configuration is not currently supported. It looks like the python core plugin is giving back the wring path for the interpreter.

cpchen commented 5 years ago

Argh. So you think there's no configuration change I can make on my end? I thought the Python plugin needs to respect what bin I pass it.

On Fri, Oct 12, 2018, 10:08 AM Roberto Leinardi notifications@github.com wrote:

I am sorry but I fear this configuration is not currently supported. It looks like the python core plugin is giving back the wring path for the interpreter.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/leinardi/pylint-pycharm/issues/7#issuecomment-429395230, or mute the thread https://github.com/notifications/unsubscribe-auth/AAho2yFuL2xeiEmzlmEzaHoXjMdzcz3zks5ukMyZgaJpZM4WYFnY .

vlasovskikh commented 5 years ago

I guess the plugin tries the wrong SDK. In IntelliJ there may be many SDKs for different programming languages. On the screenshot I see that the project has a Java 10 SDK and a Python 2.7 virutalenv SDK set up.

@leinardi SDKs can be associated with a) the Project and b) with Modules of the project. If you run your code in the context of some file opened in the editor, you can use PythonSdkType#findPythonSdk(PsiElement): Sdk? to get the correct SDK. See PsiManager#findFile(VirtualFile): PsiFile? to get a PSI file from your VirtualFile and pass it to findPtyhonSdk.

vlasovskikh commented 5 years ago

@cpchen I'm not sure that your issue is related to the original issue. It's clearly about running the plugin in IntelliJ with several SDKs set up.

leinardi commented 5 years ago

@vlasovskikh thank you for your comment, that information might be very useful in the future, but for now I have to find the interpreter without the need of a VirtualFile (I need to be able to run pylint also before starting a scan of a file). Is there a way to get a list of all the environments that I can filter to pick the Python one? Currently this is how I am getting the SDK: Sdk projectSdk = ProjectRootManager.getInstance(project).getProjectSdk();

cpchen commented 5 years ago

Looking at a comment in the top post in https://stackoverflow.com/questions/24769117/how-do-i-configure-a-python-interpreter-in-intellij-idea-with-the-pycharm-plugin

it looks like I can create a Java project with Python facet for the same module. would you be able to grab the interpreter that way?

cpchen commented 5 years ago

let's move this discussion to https://github.com/leinardi/pylint-pycharm/issues/31