Automatic dependency management

[techtonik]

Hey @jaraco. I am reading about the project and for some reason I want to believe that rwt is able to figure out and fetch dependencies automatically, but so far I haven't found proof of that - we still need to create a custom "import" section just for rwt. In this respect rwt is still manual dependency management like http://docs.groovy-lang.org/latest/html/documentation/grape.html

[jaraco]

rwt is designed to provide "on-demand" dependency resolution, not automatic dependency resolution. So it creates a temporary environment to install and make available to the Python interpreter the packages that you or your script declare. It uses pip to make those dependencies available.

So yes, you must declare those dependencies somehow, which you can do in an __requires__ statement or through a requirements.txt file or just on the command line.

I want to believe that rwt is able to figure out and fetch dependencies automatically

Can you describe more about what you're expecting? Maybe an example?

[techtonik]

make available to the Python interpreter the packages that you or your script declare.

I would expect it to find out what my script imports and fetch those imports without explicit declaration (maybe recording requirements.txt on the way). I am pretty sure people tried it before.

[jaraco]

find out what my script imports and fetch those imports

Yeah, that seems tricky. Maybe I can imagine figuring out which imports are needed for a module or script, and maybe even keeping a database of libraries satisfying those imports, so if you, for example, import PIL, it will know to git pillow and not PIL.

Another thing that makes it tricky is the ways that imports can happen. Should it only support import statements at the top of a module? What about a package? What about __import__ statements or imports in a try block?

In any case, that's not the goal of rwt, though if someone built that engine, I'd be interested in applying it to rwt, whose role is about installing the specified dependencies and running an interpreter with those.

[techtonik]

Should it only support import statements at the top of a module?

Let's not forget that dependency detection can be made in two ways:

static. It should get as deep as implemented static analysis can go with ability for users to debug and enhance this analysis. So shipping with top imports only is okay. Looking at package imports manually is the same. Rarely anybody looks further than that main entrypoint and its direct dependencies. Python is not Java.

dynamic. Look at what package that is being run tries to import, notice failures and help user choose libraries while library awaits for the import to complete. Record that info. Supervised or guided "learning'. )

[techtonik]

The logical way for dynamic analysis is to use https://bitbucket.org/pypy/revdb/ for that.

[jaraco]

Sounds like there are some potential areas for improvement here. I welcome pull requests to this end, and if I find time, I may work on it, but I won't be dedicating time to it at this time.