Deployment 2021-1.0 checklist

mrakitin commented 3 years ago

Current collection workstations

xf17id1-ws1  # interchangeable machine (can be used for DAMA tests if needed)
xf17id1-ws4  # interchangeable machine (can be used for DAMA tests if needed)
xf17id1-ws5  # interchangeable machine (can be used for DAMA tests if needed)
xf17id1-ca1  # collection server serving Jupyter notebooks, running via manage-ioc

Previsit

[x] Enable Azure-Pipelines CI
[x] Make sure that the current latest environments are pushed to all the bl machines
[x] Skim the IPython startup files in profile_collection
- Check if it's safe to start bsui remotely (does it touch any hardware?) - use git grep .put or git grep caput
[x] Check for outstanding PRs and issues on beamline repos
[x] Branch https://github.com/NSLS-II/playbooks for the purpose of updating the FMX current_env_tag in the production file (https://github.com/NSLS-II/playbooks/pull/430)

Housekeeping

[x] Update /opt/conda/.condarc and conda with ansible

$ ansible-playbook -i production install-base-conda.yml --limit=FMX -kK -vvvv

[x] Add BL staff to the BL GitHub organization as owners
[x] Work with BL staff to commit any un-committed changes to their profiles
[x] Tag the profile_collection repository as-found as 2020C2.1 and push to GitHub
```
$ git tag 2020C2.1 -m "Before 2021C1.0 deployment"
```
Note: tagged https://github.com/NSLS-II-FMX/profile_collection/releases/tag/2020C2.1 with the changes from #8 (that code in the PR branch was used during the previous cycle and was in fact their 'master'). The follow-up updates will be in a separate PR.
[x] Discuss with BL staff which conda envs they want to keep/delete and perform the cleanup
[x] Check/update the beamline's workstation inventory with the BL staff (in https://github.com/NSLS-II/playbooks/blob/master/production)
[ ] Ask beamline staff about the beamline unix group they want to use for bluesky logs. The members of the group can be checked with the following command:
```
$ getent group <beamline-group>
```
Add this group name to the collection vars in the inventory file (see this example).

[ ] In ~/.bashrc, if necessary, update the logging environment variables to use directory /var/log/bluesky/... and add umask 0002.

umask u=rwx,g=rwx,o=rx  # 0002
export BLUESKY_LOG_FILE=/var/log/bluesky/bluesky.log
export BLUESKY_IPYTHON_LOG_FILE=/var/log/bluesky/bluesky_ipython.log

and (if needed) create that directory with the following permissions and ownership:

$ sudo mkdir /var/log/bluesky
$ sudo chown -Rv xf17id1:<beamline-group> /var/log/bluesky
$ sudo chmod -Rv g+rws /var/log/bluesky

Remember to source ~/.bashrc.

[x] Remove explicit setting of ophyd logging level from the first startup script
[x] Double-check that if there is an open PR removing handlers, it is merged (and tested).
[x] Update vendored copy of PersistentDict to bug-fixed version (see the updated gist snippet: https://gist.github.com/jklynch/a4366b8900ec0c03883403455ae711b2).
[x] Make sure all repos have the BSD-3-Clause licenses (see https://github.com/NSLS-II-SRX/profile_collection/pull/16 as an example)

Test deployment

[x] Run profile against new environment
```
$ BS_ENV=collection-2021-1.0 bsui
```
It may be necessary to disable jedi in ipython. Add this to the ipython command line arguments inbsui:
```
--IPCompleter.use_jedi=False
```
[x] Update the profile as needed to run

[x] Test databroker v2

from databroker import catalog
cat = catalog['fmx']
cat[-1].primary.read()

[x] Create a Jupyter kernel for the new conda environment in xf17id1-ca1:/usr/share/jupyter/kernels:

$ cd /usr/share/jupyter/kernels-available/
$ sudo cp -r collection-2020-2.0rc8 collection-2021-1.0
# edit the kernel file to have the `collection-2021-1.0` conda environment in the path and update the description
$ vi collection-2021-1.0/kernel.json 
$ cd /usr/share/jupyter/kernels
$ sudo ln -s ../kernels-available/collection-2021-1.0 .

[x] Check the new kernel is available:
```
$ jupyter kernelspec list
```
[x] Update /epics/iocs/notebook/notebooks/services/st.cmd as needed (update the conda activate ... command with the new conda environment)

[x] Check the status of the Jupyter server (restart if anything was changed in the st.cmd file):

$ sudo systemctl status softioc-notebook
$ sudo systemctl restart softioc-notebook  # optional

[x] Commit and push the latest versions of st.cmd and kernel.json to:
- https://github.com/NSLS-II-FMX/services
- https://github.com/NSLS-II-FMX/kernels
[ ] Run acceptance tests (if any) and run a representative set of scans for the beamline against the new kernel in a Jupyter notebook.

Finish

[ ] Tag the updated profile as 2021C1.0

$ git tag 2021C1.0 -m "After 2021C1.0 deployment"

[ ] Open and merge a PR against https://github.com/NSLS-II/playbooks setting current_env_tag="2021-1.0" for the FMX beamline
[ ] Update bsui via ansible to use the new environment (http://nsls-ii.github.io/deployment/upgrading-beamlines.html#change-bsui-s-default-environment)

mrakitin commented 3 years ago

...@xf17id1-ca1:~$ cat /usr/share/jupyter/kernels/collection-2021-1.0/kernel.json
{
  "argv": [
    "/opt/conda_envs/collection-2021-1.0/bin/python",
    "-m",
    "ipykernel",
    "-f",
    "{connection_file}",
    "--profile=collection",
    "--log-level=DEBUG"
  ],
  "display_name": "Data Collection (2021-1.0)"
}

mrakitin commented 3 years ago

Syncs:

mrakitin commented 3 years ago

@mfuchsbnl, @danielballan, I do not know what was going on at the beamline, but I was able to run the scan with the latest kernel with the collection-2021-1.0 conda environment:

Please note, I created a separate kernel so that it does not interfere with the production kernels, and used a PR branch named deploy-2021-1.0 from #12 for my tests.

Is it something to do with the browser on the beamline? Can you try Google Chrome or a newer version of Firefox?

mrakitin commented 3 years ago

(/opt/conda_envs/collection-2021-1.0) mrakitin@xf17id1-ca1:~$ jupyter kernelspec list
Available kernels:
  python3                         /opt/conda_envs/collection-2021-1.0/share/jupyter/kernels/python3
  collection-17q2.0               /usr/share/jupyter/kernels/collection-17Q2.0
  collection-17q3.0               /usr/share/jupyter/kernels/collection-17Q3.0
  collection-2019-2.0             /usr/share/jupyter/kernels/collection-2019-2.0
  collection-2019-3.0             /usr/share/jupyter/kernels/collection-2019-3.0
  collection-2020-2.0rc8          /usr/share/jupyter/kernels/collection-2020-2.0rc8
  collection-2021-1.0             /usr/share/jupyter/kernels/collection-2021-1.0
  collection-2021-1.0-mrakitin    /usr/share/jupyter/kernels/collection-2021-1.0-mrakitin

mrakitin commented 3 years ago

That was successful with collection-2020-2.0rc8 too:

NSLS-II-FMX / profile_collection