Allow capabilities to note their github links intead of assuming they're on pypi

[BryceStevenWilley]

I ran into a simple issue: our default Weaver configuration_capabilities.yml includes ILAO as an organization, and when selected adds docassemble.ILAO to the install_requires of the setup.py. However, docassemble.ILAO isn't on pypi, and when installing on a server that doesn't already have docassemble.ILAO installed, the installation will fail.

Python does let you directly install things from GitHub repos (more info in PEP-508). I manually changed the install_requires to docassemble.ILAO @ https://github.com/IllinoisLegalAidOnline/docassemble-ILAO/archive/main.zip, which should install directly from main on each install, and from what I can tell, DA will see it as a Github installation and properly show the update button next to it, which is nice.

IMO we should add either the direct github link or the github org and repo name of the branding package to configuration_capabilities.yml, which would allow us to add those as dependencies. I need to see if DA's playground behavior would mess up anything, but it's a good start to better plug and play IMO.

[BryceStevenWilley]

This doesn't seem like it actually needs a change in the Weaver; folks can just type the whole docassemble.ILAO @ https://github.com/IllinoisLegalAidOnline/docassemble-ILAO/archive/main.zip into organization_choices: ... dependency: in their configuration_capabilities.yml. I'll document this on our site, and close this issue once it's documented.

I did find a limitation of this approach: once the package has been installed for the first time from GitHub, docassemble forgets where it came from; if you try to click the "Update" button next to the package from the package management screen, it will try to install it directly from Pypi again. To fix this would require an upstream fix (with an alembic / database update, which always seem to be a bit unstable, I see a lot of errors from people's servers in the slack after they occur). The workaround would be to tell people to just re-install it from the GitHub URL if they run into errors.

I will still document it like this though; both options (the existing way and this proposal) should happen rarely, only if a branding package isn't installed server wide, and it's a choice between something breaking immediately when installing a package, or breaking when only someone attempts to directly update that package. I'll consider some upstream fixes to mitigate the issue, but will have to likely stay on the back burner until I have some time. Some bread crumbs for later:

• pip freeze gives how each package was installed, maintaining the full github URL used, but docassemble currently uses pip list --format=freeze which does not include this information. Should look into what a typical DA pip installation reports back for both of those, and how to turn the pip freeze into pip list --format=freeze, while keeping the Git URLs.

  • On my local docassemble, this is the diff between the two tools:

    www-data@0a1c27f8a847:/tmp$ pip list --format=freeze > tmp2.txt
    www-data@0a1c27f8a847:/tmp$ pip freeze > tmp.txt
    www-data@0a1c27f8a847:/tmp$ diff tmp.txt tmp2.txt 
    6c6
    < distro-info===1.1build1
    ---
    > distro-info==1.1build1
    10a11
    > pip==22.0.2
    14c15
    < python-debian===0.1.43ubuntu1
    ---
    > python-debian==0.1.43ubuntu1
    18a20
    > setuptools==59.6.0
    23a26
    > wheel==0.37.1

  So some extra packages like setuptools and wheel, and triple equals on ubuntu packages, but not much else. On my local machine, where things are wild, there are a few other cases to probably handle:

  • -e git+ssh://git@github.com/SuffolkLITLab/FormFyxer@873e4d1f721e2e3ca64f7a8f3ed80fc55051598b#egg=formfyxer: local editable installs. Shouldn't happen with DA
  • en-core-web-sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.2.0/en_core_web_sm-3.2.0-py3-none-any.whl: github direct installs: exactly what we want
  • sometimes there are comments, lines that start with # that we should be able to ignore easily; line.strip().startswith('#')

[BryceStevenWilley]

Did a bit more on this: I can install a package with a specific GitHub dependency, and docassemble will recognize it as a GitHub dependency. However, there's still some work to be done. Specifically, how do we determine what version of the dependency should we use when making a git commit or downloading the package from the playground? Do we need to keep track of the original dep, or should we always use what's on the server? Right now, using what's on the server is the default, I'm just not a big fan of it, and not sure if it needs to be better for this new feature to not be confusing.

Code at https://github.com/jhpyle/docassemble/compare/master...BryceStevenWilley:docassemble:github_requires?expand=1