search

Embarcadero / Lightweight-Python-Wrappers

Lightweight Wrappers based on Python4Delphi to make it easy to import Python modules into Delphi components.
MIT License
30 stars 11 forks source link

Allow redirect of IO for PIP #4

Open peardox opened 2 years ago

peardox commented 2 years ago

Looking at PyPackage.Manager.Pip.pas and cross-referencing with PythonEnvironments' PyTools.ExecCmd.pas there are methods defined to capture IO but these are unused as everything calls the Run(AOutput) version

It seems desirable to use the capture if a (unimplemented) PIP/Conda IO event were available. This would possibly allow the use of the Progress option to monitor what the install was doing. This would be especially useful for very large packages (e.g. Torch-GPU takes 4 mins to download with no way to inform the user what's going on)

lmbelo commented 2 years ago

@peardox it was in our plans to make it available on events. What do you have in mind?

peardox commented 2 years ago

From looking at what PIP does it shows a progress bar when installing a new module. This is displayed on one line being overwritten periodically looking like this...

Using pip 22.1.2 from

C:\Users\simon\AppData\Local\Programs\Python\Python39\lib\site-packages\pip (python 3.9) Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/cu116 Collecting torch Downloading https://download.pytorch.org/whl/cu116/torch-1.12.0%2Bcu116-cp39-cp39-win_amd64.whl (2388.0 MB) -- ------------------------------------- 0.2/2.4 GB 6.6 MB/s eta 0:05:36

The very last line is the interesting one as it changes showing the progress of the download. Try it yourself with this...

pip install -v torch --extra-index-url https://download.pytorch.org/whl/cu116

Yoiu might need to "pip uninstall torch" and possibly "pip cache purge" to get a proper idea of the issue.

What I'm hoping is that the last line can be parsed to return some useful info on install progress in the 5 mins 36 secs mine says it's taking. This could then be used to drive a Delphi progress bar etc.

While I'm using an extreme example here this also applies to smaller packages - it just isn't quite as bad as this case...

A lot of ML Python stuff relies on large quantities of data - I mean CoCo 2017 (117k training images) is 18Gb for example. Those I can handle as it's just files from the Internet but PIP has no mechanism for developer-friendly progress information.