Tasks required before moving to standard Python packages

[philipstarkey]

Original report (archived issue) by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

This is a list of the things we will need to do before we can move to having standard Python packages for labscript suite components. We can check things off as they are done

• Implement a 'user_devices' folder so that users can add devices outside of labscript_devices whilst hacking on them (done, labscript_utils PR #81)
• Create log files within a 'logs' directory under the labscript suite install directory (done, labscript_utils #PR 95)
• Finish the changes to BLACS that would allow it to import unit calibration classes from anywhere, and not just from within labscript_utils.
• Convert each project into the standard repo layout with a setup.py
• Brainstorm what the installer should be and do. Should there even be an installer? Perhaps the installer can simply be the setup.py of a 'labscript_suite' package that pulls in the latest versions of all the other packages and sets up the directories. Can we install the labscript suite components into the labscript suite install directory? Should we? We will still need to do path manipulation with a .pth file, since userlib and its subdirectories will still exist and need to be importable. Brainstorming done, see comment below. Yet to do: implement this
• Modify labscript_utils.winshell functions that refer to packages' __main__.py files and icon files to find them according to the import path of the package instead of assuming they are within the labscript suite install directory.(done, labscript_utils PR #96)
• Modify BLACS and possibly others to look up their file resources based on the __file__ attribute of their modules rather than assuming a fixed location within the labscript suite install directory (we actually switched BLACS to not use __file__ at one point because it was error-prone in the presence of os.chdir() calls. But in Python 3 this is less of an issue since __file__ is absolute unless the main script was run with a relative path. Even then, a single __file__ lookup in __init__.py should be unproblematic).

Phil mentioned implementing our planned backward-incompatible changes to communication between programs when we make this change. I don't think that is super important, since although the changes are backward incompatible, they are detectable at run time as being incompatible and users will be prompted to upgrade. So I don't think we need to roll this out at the same time necessarily.

[philipstarkey]

Original comment by fretchen (Bitbucket: fretchen).

---

Concerning your list there.

• Would number three not make number one and two simpler to test ?
• If we implement number three, it might not be very clear anymore what the installer is for no ?

I also wonder if one should first move to github and then start the disentanglement on the new platform ? So make issue #32 blocking for this one here ?

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

fretchen (fretchen), number 1 is already implemented, and was actually fairly unrelated to how labscript_devices itself is laid out. Changing the repositories to be normal packages should not interfere with this.

It may be possible to remove the intaller package altogether, and relegate the creation of shortcuts to the individual programs. However, the responsibility to create the default labconfig file, and choose its location, will still need to be delegated to some package. So perhaps a package playing that role will be created and shipped as 'labscript-suite' or similar, depending on the latest stable versions of the other packages. If installed with a 'dev' option it could basically do what it does now and clone the repositories (perhaps the development versions). Still thinking about it!

I am happy to move to github, yes. Once there, existing users will be without the ability to update unless they are enthusiastic and re-clone all the repositories as git ones. So enthused users can do that, and everyone else can just be without updates until we restructure the repositories. Messing with things sooner will create an additional breaking change, so might as well be avoided. Switching the two at exactly the same time would be ideal, so if things are looking good we will do that.

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

• Edited issue description

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

I thought about what the installer should be, and it's pretty obvious in hindsight!

All the repos labscript, labscript_devices, runviewer, runmanager, blacs, labscript_utils, will be their own PyPI/conda packages, that depend on each other with specific versions as required. Then there'll be a package which I think would best be called 'labscript-suite', that will depend on all the others - not on any particular versions except as required by the functionality of a script that will come with this package - such that conda install labscript-suite or pip install labscript-suite will install the latest releases of all the others.

The problem with pip/conda packages alone is that they have no state outside of of site-packages. We still need something that can prompt the user to ask them where they want to put the folder containing labconfig, log files, their userlib, and app saved configs, create start menu shortcuts, etc (i.e. everything that used to be considered part of a labscript suite 'installation' other then the actual applications/modules).

It is possible to make post-install scripts that will run after conda/pip install, and even be interactive, but I don't think we want this since we wouldn't want it to run on every upgrade, and there are other things we would want a manually-run script to do anyway, so I think the best solution is to just have the user run a script after doing pip/conda install

To that end, all the stuff that the installer currently creates other than the mercurial clones will be called a 'profile', and the labscript-suite package will come with a labscript-suite-profile script, with the following subcommands:

labscript-suite-profile new

This is what you will run after a pip/conda install labscript-suite. The end result will be the same as if you ran python setup.py install with the existing installer package. This command will:

• Ask user for a path to create a new profile (default C:\labscript_suite or ~/labscript_suite).
• Ask the user for a name for the profile. Defaults to nothing, or the conda env name if there is one.
• Creates all the same things as the current installer, except for cloning repos:
  • labconfig
  • log folder
  • userlib
  • "Registers" the profile with the currently running Python interpreter - see below

labscript-suite-profile register

This is run when creating a new profile, and is what you would run manually if you reinstalled Python (to recreate the .pth file and get the launchers working again), or you wanted to associate the same userlib and labconfig etc with multiple Python installations

• If being run manually, prompts for the path of an existing labscript suite profile (default C:\labscript_suite or ~/labscript_suite).
• Creates a .pth file in site-packages so that the userlib modules are in the python import path for that interpreter
• Creates the launchers and their start menu shortcuts.

labscript-suite-profile deregister

• Deletes .pth file
• Deletes launchers and start menu shortcuts. You can do this to intentionally disable a labscript suite profile and switch to another one with the same Python interpreter. You might do this before upgrading Python or similar, to avoid broken launchers and shortcuts in the meantime, though neglecting to do so will not stop the 'register' step from working.

labscript-suite-profile develop

Configure the python import path to import some modules directly from git repos instead of from site-packages, similar to python setup.py develop except not requiring uninstallation of the corresponding pip/conda package to work (you can put the repo first in the import path so that it takes precedent, unlike python setup.py develop which puts it last), and not requiring manual deletion of files from site-packages to reverse.

• With no arguments, or maybe an argument like 'list', lists which repos are currently in development mode.
• If adding repos to development mode, creates a development_repos directory within the profile directory if it doesn't exist.
• takes a list of repos that should be put into or out of development mode, maybe in a format like "+labscript, +runmanager, -lyse" or "+all" "-all" to add or remove repos from development mode
• If putting a repo into development mode, clones it into development_repos from the given URL prefix (defaulting to labscript suite mainline) if it's not already there. If turning off development mode for a repo, doesn't delete it or anything, just leaves it as is.
• prepends/removes each repo to the python import path by modifying the labscript suite .pth file.

I am pretty sure that if the package is made correctly, the script should end up in the bin directory where the Python interpreter lives, such that the user will be able to run it just as labscript-suite-profile, i.e. without the full path or anything. If that turns out not the case then it would be a little more verbose: python -m labscript-suite-profile.

[philipstarkey]

Original comment by fretchen (Bitbucket: fretchen).

---

Thanks so much for this.

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

• Edited issue description

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

• Edited issue description

[philipstarkey]

Original comment by Philip Starkey (Bitbucket: philipstarkey, GitHub: philipstarkey).

---

FYI: I figured out today how to automate the publishing of packages to PyPI using GitHub actions. You can check out the github workflow and setup.py files for the autoscrub project Russ and I made. It automatically publishes every version to Test PyPI, and tagged versions to real PyPI.

‌

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

Ah, that's great! We should definitely get that happening for labscript suite packages.

[philipstarkey]

Original comment by Philip Starkey (Bitbucket: philipstarkey, GitHub: philipstarkey).

---

Just leaving a note about this here so I don’t forget: I think there might be a bug in my “build and publish to PyPI” GitHub action code. I had attempted to make the git tag the authoritative version (overriding the version in setup.py in case we forgot to update it). However, while I think this works for wheels, I don’t think it works for installing from source builds (aka sdist builds) uploaded to PyPI. There are some solutions involving only writing out __version__.py if it doesn’t exist, but that probably doesn’t play nice with installing with the -e option. Given that installing with -e requires the version in setup.py to be accurate anyway, perhaps the answer is not to make the git tag authoritative, but have the build fail if the git tag does not match the version specified in the setup.py file in order to prevent publishing versions where we forgot to update the version number.

Anyway, something to resolve when it comes time to implement this.

‌