Feature/improve package info handling

reorx commented 6 years ago

Goals:

Fix the problem of #124
Improve package_info related code (involved in #78), make the code more robust and easier to read, add comments to important functions, improve function consistency.

re-org the relationship of the functions below:

save_package_info: receives data from installation process, use make_package_info to create the dict, save it to file
make_package_info: in charge of creating the package_info dict from scratch
get_package_info: if the json file exists, get and parse its data; if not, call make_package_info to get the dict

Eventually, get_package_info's behavior becomes consistent and predictable, so now we can use it in list_everything easily and intuitively.

Other changes:

Add an uninstall-pipsi.py script
Add -e to pip when running get-pipsi.py .
Remove duplicated list_everything call

TODO:

[x] Define a class to handle read/write about package_info

smancill commented 6 years ago

I confirm this was the only way I got pipsi working.

The command I used was a combination of #162 and this branch:

$ curl https://raw.githubusercontent.com/reorx/pipsi/41ec097047ede9af0381d86080ca4b59cbe51062/get-pipsi.py | python3 - --src 'git+https://github.com/reorx/pipsi.git@41ec097047ede9af0381d86080ca4b59cbe51062#egg=pipsi'

reorx commented 6 years ago

@mitsuhiko @RonnyPfannschmidt guys, reviewing of this project has been very inactivate recently, and I'm sad to see that. Pipsi is a very great tool, I personally use it a lot, I think it can make better difference, and it still has a lot of improvement space. I'm wondering if I can be one of the maintainers, I have at lease one day free to do open source coding each week. I really want to push pipsi forward, to release a stable and mature version for all the people who care about it. How do you think? Looking forward to your reply :)

sahensley commented 6 years ago

After installing Pipsi from this branch, I now have a functioning Pipsi installation which is backed by Python 3.6.6. I can also verify that the virtual environments are being made via the venv module instead of virtualenv.

I'd love to see @reorx added as a maintainer and possibly a few others to help keep this project afloat.

sciyoshi commented 5 years ago

Thanks @reorx, the package_info.json issue causes problems when pipsi shares its venv directory with other tools like pew. :+1: to merging this as well as expanding the set of maintainers if that's something @mitsuhiko would support.

sciyoshi commented 5 years ago

@reorx I did notice an issue however:

Packages and scripts installed through pipsi:
  Package "awscli":
Traceback (most recent call last):
  File "/Users/sciyoshi/.local/bin/pipsi", line 11, in <module>
    load_entry_point('pipsi', 'console_scripts', 'pipsi')()
  File "/Users/sciyoshi/.local/venvs/pipsi/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/sciyoshi/.local/venvs/pipsi/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/sciyoshi/.local/venvs/pipsi/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/sciyoshi/.local/venvs/pipsi/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/sciyoshi/.local/venvs/pipsi/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/sciyoshi/.local/venvs/pipsi/lib/python3.6/site-packages/click/decorators.py", line 27, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/Users/sciyoshi/.local/venvs/pipsi/src/pipsi/pipsi/__init__.py", line 607, in list_cmd
    click.echo('    ' + script)
TypeError: must be str, not list

It seems like the file gets re-written with scripts that look like this: {"name": "httpie", "version": "0.9.9", "scripts": [["/Users/sciyoshi/.local/venvs/httpie/bin/http", "/Users/sciyoshi/.local/bin/http"]]}

reorx commented 5 years ago

@sciyoshi hmm, I find that too, thanks for reporting, to make pipsi list work temporarily, you can change line 607 to click.echo(' {}'.format(script)). I'm digging further into this problem to see what causes scripts to be a list in list.

reorx commented 5 years ago

@sciyoshi the new push should fix that issue, please check it out. For the infected package_info.json files, please manually fix them by removing the redundant [ and ] ([["/Users/sciyoshi/.local/venvs/httpie/bin/http", "/Users/sciyoshi/.local/bin/http"]] should become ["/Users/sciyoshi/.local/venvs/httpie/bin/http", "/Users/sciyoshi/.local/bin/http"]).

It was a bug in upgrade command that caused the json file to be wrong.

sciyoshi commented 5 years ago

  File "/Users/sciyoshi/.local/bin/pipsi", line 11, in <module>
    load_entry_point('pipsi', 'console_scripts', 'pipsi')()
  File "/Users/sciyoshi/.local/venvs/pipsi/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/sciyoshi/.local/venvs/pipsi/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/sciyoshi/.local/venvs/pipsi/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/sciyoshi/.local/venvs/pipsi/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/sciyoshi/.local/venvs/pipsi/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/sciyoshi/.local/venvs/pipsi/lib/python3.6/site-packages/click/decorators.py", line 27, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/Users/sciyoshi/.local/venvs/pipsi/src/pipsi/pipsi/__init__.py", line 570, in upgrade
    if repo.upgrade(package, editable):
  File "/Users/sciyoshi/.local/venvs/pipsi/src/pipsi/pipsi/__init__.py", line 466, in upgrade
    old_scripts = set(info.scripts)
TypeError: unhashable type: 'list'

I also got this issue when upgrading, which I was able to fix by removing and reinstalling.

reorx commented 5 years ago

@sciyoshi could you try upgrading using the the updated version of this PR (ensure scripts in package_info.json is not list of list first, see my previous comment)? The latest two commits should have fixed the upgrading issue.

aiguofer commented 5 years ago

I was having issues upgrading pipsi installed packages (using master). I can confirm that this branch fixes those issues.

cs01 commented 5 years ago

Just wanted to note that I have started pipx which combines the functionality of pipsi and npx. I have implemented several of the features requested in pipsi, including upgrading all packages with one command (pipx upgrade-all) and reinstalling all packages w/ a different Python executable with one command (pipx reinstall-all PYTHON). pipx works with python3.6+ only, and uses venvs only.

I am actively responding to issues and PRs. Would love to hear what you think -- https://github.com/cs01/pipx.

mitsuhiko / pipsi

Feature/improve package info handling #161