search

NERC-CEH / rse_group

4 stars 1 forks source link

Writing bootstrap documentation #29

Open metazool opened 1 month ago

metazool commented 1 month ago

This is rather an unclosable issue, but I've got a backlog of "should write this" documentation about healthy development practises that I'm not making time for. Additions as well as contributions are really welcome.

mattjbr123 commented 3 weeks ago
  • Configuring a development environment on a private cloud VM with VSCode Remote SSH and miniconda (ideally the SciCom team maintain this - is there a documentation wiki or repo to contribute to?)

In the past I have created an environment yml or text file for people using miniconda with instructions on how to install it. Could be a starting point for this? I know 4-years-ago-me would have found this incredibly useful, instead of having to figure it all out on my own...

mattjbr123 commented 3 weeks ago

Simple git branch-merge workflow walkthrough with illustrative examples for both commandline and VSCode UI

@longr didn't you run some git courses/training sessions at one point (I think I went on one??)? Could some of the material from those be adapted into a tutorial of sorts? I could also dig out my notes from this too if needed.

longr commented 1 week ago

Simple git branch-merge workflow walkthrough with illustrative examples for both commandline and VSCode UI

@longr didn't you run some git courses/training sessions at one point (I think I went on one??)? Could some of the material from those be adapted into a tutorial of sorts? I could also dig out my notes from this too if needed.

Yes, I taught from the carpentries, not great for this, but https://docs.github.com/en/get-started/using-github/github-flow is quite good. Probably needs a few diagrams and instructions from VSCode, but good starting point. Perhaps needs re-wording for a different target audience. It focusses on branching, but a fork-branch-PR maybe a better way.

We also have the atlassian tutorials, again need adapting for different audience, but probably workable. https://www.atlassian.com/git/tutorials/comparing-workflows/feature-branch-workflow

Happy to look at this, writing users guides, depending on depth can be a happy place for me.

longr commented 1 week ago
  • Deploying applications on Posit Connect (it's such a common pattern and with caveats, useful infrastructure)

We will probably be doing this as part of the Research Data Cloud project, so perhaps a todo for me?

mattjbr123 commented 4 days ago
  • Configuring a development environment on a private cloud VM with VSCode Remote SSH and miniconda (ideally the SciCom team maintain this - is there a documentation wiki or repo to contribute to?)

In the past I have created an environment yml or text file for people using miniconda with instructions on how to install it. Could be a starting point for this? I know 4-years-ago-me would have found this incredibly useful, instead of having to figure it all out on my own...

Previously these are the instructions I've given people for creating and using a certain python environment using conda, assuming they have access to a linux machine/VM in some way. It would need some updating to avoid anaconda's new liscensing restrictions, and @jmarshrossney 's experimentation with conda-lock and lockfiles suggests there might be a better approach, but it did do the job to begin with and might at least be a good starting point that we can refine. (Background info: People used to have their own physical linux boxes which you could get using some project's funds, now they've virtualised/VMed(?) them all to the private cloud and you can request new VMs but as we've found out it takes ages to action. They are all also locked-down to some degree, no root access etc., so this was my way around that. Beforehand people were emailing IT to get various packages they needed installed which was very much not ideal...).

The code uses several python/R packages that are not installed by default on most systems.
The easiest way to install these is to use anaconda - a package management system for python and R modules/packages.
Anaconda installs packages in your home directory, and can be managed separately to any other python/R installations
you have, i.e. it won't interfere with these, and can be enabled only when you want to run this programme.
See below for instructions on installing and using anaconda.

Anaconda installation instructions
----------------------------------
To install anaconda and my python/R packages:

1. Download anaconda from https://www.anaconda.com
You want a python>=3.7, 64bit (x86) installer for linux.
As of 08/06/2020 this can be obtained by running: wget https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh
But note that the link to the most up to date version will likely change, so don't rely on this.

2. Install anaconda by running: bash the-script-you-just-downloaded.sh
By default it will install in your home directory, requiring no admin/root permissions.
At the end of the installation it will ask you whether you want to run conda-init.
Say no to this.

3. Add the following line to your .bashrc file in your home directory. If you don't have one, create one.
alias loadconda='export PATH="/path/to/homedir/anaconda3/bin:$PATH"'
Obtain the full path to your home directory by running pwd in your home directory.

4. Start a new bash shell by running bash, then load anaconda by running loadconda (the command we just created above).

5. Setup a 'DAenv' environment that contains all the modules you need to run my scripts,
using the conda_env_file_DAenv.txt. (You can change the name if you want).
In your home directory run 'bash' if you are not already using the bash shell (by default UKCEH uses the csh shell)
Then conda create --name DAenv --file conda_env_file_DAenv.txt (after copying the txt file to your home dir).

6. Run 'source activate DAenv' to load this environment, and you should be good to go.

Now, instead of sourcing the python and R modules from the default setup, or your own, it will source them
from ~/anaconda3/envs/DAenv/.
To get back to your own python/R setup, simply open a new terminal/shell.

Anytime you want to run these scripts, or use my python setup, run:
bash
loadconda
source activate DAenv

I also wrote one for windows, but as @jmarshrossney has pointed out this involves downloading anaconda's software which would now fall foul of their new liscensing terms:

ANACONDA INSTALLATION INSTRUCTIONS
----------------------------------
If you don't have access to python, you can install it on your own windows laptop/computer.
Anaconda is the recommended way to install python, as it manages all the different
modules and dependencies for you, and can be installed without needing admin rights.
The instructions here show you how to install anaconda, and additionally install
modules that you used during the python course on eu.pythonanywhere.com to analyse and
plot netcdf data, such as xarray and cartopy.

1. Download anaconda from https://www.anaconda.com/products/individual
   You want the python 3.9, 64bit installer for windows.

2. Run the installer you just downloaded. When prompted who to install for,
   select 'Just me' instead of 'Everyone'. All the other options you can
   just leave as they are. The installation will likely take a few minutes
   to an hour.

3. Now look in the start menu and open up Spyder. This is a 'interactive development
   environment' (IDE). Basically, it allows you to write scripts in one bit (the left side)
   and run them and any other commands in the terminal/console in another bit (the bottom right)
   The top right bit is where you can get help by default, but you can close it or have
   other windows, such as a file directory tree or variable tracker, here.

4. Currently we only have the basic python modules installed. This will likely not contain
   the ones you want to use. I have created an 'environment' (set of python modules) that
   will allow you to run all the examples and scripts you covered in the course.
   To install it, copy the environment file 'conda-env-file.txt' (or similar) to the
   directory shown in the top right of the Spyder programme you have open. Usually this is
   C:\Users\your_windows_username
   This is the directory that the python terminal in the bottom right is running in.
   Copy and run the following command in the python terminal:
   conda create --name testenv --file conda_env_file.txt
   You can change testenv to whatever you want to call this new environment.
   This again might take several minutes to an hour to run.

5. Now to access the new environment in Spyder, open up the 'Anaconda Navigator' from the start menu
   Select 'testenv' (or whatever you called your environment) from the 'Applications on' dropdown
   menu. The scroll down and find Spyder, and click 'Install'. Alternatively, if you'd rather just
   use a terminal/console without Spyder, you can install 'Qt Console' instead.

6. Once they are installed. Click 'launch' to open one or both of these in the new environment you
   created earlier. Now your console/Spyder will have access to all the python packages you're#
   likely to need

7. To add new python packages use the console (the QT console or the one in Spyder) and run
   conda install -c conda-forge name-of-package
   e.g. conda install -c conda-forge cftime
   (the '-c conda-forge' bit is telling anaconda to install all packages from a specific repository, which
   has far more of the packages you are likely to be using than the default repository)