search

microsoft / python-language-server

Microsoft Language Server for Python
Apache License 2.0
913 stars 133 forks source link

When scraping compiled binaries why env is ignored? #1948

Open novoselov-ab opened 4 years ago

novoselov-ab commented 4 years ago

When scraping pyd files it runs python with -E flag, which ignores environment variables. For other modules I can set .env file which can be quite sophisticated. I have no control over binary bindings. They often can import other stuff or and do some additional logic in init.py file.

this is the line I am referring to: https://github.com/microsoft/python-language-server/blob/8ff80697a22b958100b9d12cec1a308b61dc7d10/src/Analysis/Ast/Impl/Modules/CompiledPythonModule.cs#L36

Could at least that be somehow an option? In practice it is really hard to setup autocompletion for binary modules working.

jakebailey commented 4 years ago

.env files aren't loaded by the language server, and the VS Code extension doesn't pass things from that file down other than PYTHONPATH.

IIRC we don't load the environment so that we don't open a vector for even more odd behavior; there are a slew of variables that may change the module resolution we expect and we'll end up scraping the wrong module. You may want to try removing that and see if it does fix your case.

Note that the scraper is intended for libraries, not user code, so I wouldn't try to use it for anything other than compiled code you aren't planning on editing.

novoselov-ab commented 4 years ago

Thank you for such a quick response!

yes PYTHONPATH is basically what is needed in that case. Removing -E fixes our issues.

I didn't fully understood last part sorry. We use scrapper implicitly to autocomplete compiled code. We have a lot of C++ bindings as .pyd files. Really cool part about MS python language server that it manages to generate stubs for those and show intellisense. But if say this .pyd expects other modules imported, it can't do so. Often .pyd file importing wrapped in init.py file which also could import additional stuff, like in usd library: https://github.com/PixarAnimationStudios/USD/blob/master/pxr/usd/usd/__init__.py

jakebailey commented 4 years ago

My comment on "libraries, not user code" was about if you had a project that was partially in C/C++ and partially in Python, and wanted to edit the C++ code and immediately see the changes in the same workspace in Python. The current scraping mechanism isn't set up to do that.

We can play with omitting -E, we'd just have to do some testing to see what the other impacts are.

novoselov-ab commented 4 years ago

What do you mean by immediately? I also noticed that those cached .pyi files are not updated that often. I had to clean cache by hand to test new changes. Is that what you imply?

Follow up question, is there a way to generate pyi files ourselves and let python server use those?

jakebailey commented 4 years ago

What do you mean by immediately? I also noticed that those cached .pyi files are not updated that often. I had to clean cache by hand to test new changes. Is that what you imply?

Yes, that is what I mean. We have no way to know that some change in some non-python file would mean that some other pyc/pyd file needs updating (nor would we know how to build it to update it, and so on). Essentially all users of the scraper are from builtin compiled files, or libraries like numpy with compiled components.

Follow up question, is there a way to generate pyi files ourselves and let python server use those?

There isn't an "automatic" way, but the recommended way would be to place pyi files next to your compiled modules when they're distributed such that tooling doesn't need to do any work to find them. You can run the scraper directly if you want (note that its output is not a "true" stub, it's much closer to python code and can't be processed by other tools), stubgen (if it works), hand write known APIs with the correct types, etc. If you had say foo/bar/compiled.pyd, you could have foo/bar/compiled.pyi and we'll be able to just go to the stub.