search

cms-gem-daq-project / gem-plotting-tools

Repository for GEM commissioning plotting tools
GNU General Public License v3.0
1 stars 26 forks source link

Update setup instructions #109

Closed lmoureaux closed 6 years ago

lmoureaux commented 6 years ago

Description

User setup instructions are currently broken. This PR is about rewriting them.

I used Brian's script as a base, and modified it to run on every tested machine. I also added a few checks. The new script is here.

Types of changes

Motivation and Context

From most to least relevant:

How Has This Been Tested?

The instructions have been tested in the following environments:

Before starting, I removed all pip packages installed in .local. I used my favorite shell, bash.

The testing itself consisted of:

Planned work

Help needed [obsolete]

I briefly tested the setup script with zsh, and it's broken (doesn't call pip and virtualenv). I'm quite sure other shells are affected. I'd like to get:

Checklist:

bdorney commented 6 years ago

Test at P5, plus documentation (Can someone help me to create a proxy? http://cms-sw.github.io/tutorial-proxy.html doesn't seem to work with wget or curl)

You'll need to enable port forwarding for https. An example for ssh is:

# https://twiki.cern.ch/twiki/bin/view/CMS/DQMOnline#How_to_setup_GIT_to_work_from_in                                                                                                                                                    
Host github*
     User         <your username>
     Protocol     2
     # ProxyJump ssh cmsusr.cms ## requires OpenSSH 7.3 or higher
     ProxyCommand ssh cmsusr.cms -W %h:%p
     # ProxyCommand ssh cmsusr.cms nc %h %p

I don't know the ports that would be relevant here. Maybe @jsturdy or @mexanick can offer some advice. Otherwise manual checkout of required files needs to be added.

bdorney commented 6 years ago

I briefly tested the setup script with zsh, and it's broken (doesn't call pip and virtualenv). I'm quite sure other shells are affected. I'd like to get:

Which script are you referring to, and what is the calling syntax and terminal output? The script shown at my gist above works without issue in zsh as that's my default $SHELL.

lmoureaux commented 6 years ago

I briefly tested the setup script with zsh, and it's broken (doesn't call pip and virtualenv). I'm quite sure other shells are affected. I'd like to get:

Which script are you referring to, and what is the calling syntax and terminal output? The script shown at my gist above works without issue in zsh as that's my default $SHELL.

My modified script, setup.sh. The following lines may need to be updated:

lmoureaux commented 6 years ago

Point 5 update

Here are preliminary instructions. anaUltraScurve.py -h runs fine, but I haven't tested my updated script at lxplus and 904 (yet).

I keep the comments above in mind, just processing one issue at a time :snail:

@jsturdy Is installing PySocks system-wide an option?

Setup at Point 5

Due to the limited Internet access, the setup at Point 5 is more involved.

To be done once

Download a package to enable the use of SOCKS proxies in Python:

ssh cmsusr wget https://files.pythonhosted.org/packages/53/12/6bf1d764f128636cef7408e8156b7235b150ea31650d0260969215bb8e7d/PySocks-1.6.8.tar.gz

Install it:

pip install --user PySocks-1.6.8.tar.gz

To be done every time you create a new environment

Download the cmsgemos package manually:

scp lxplus.cern.ch:/afs/cern.ch/user/s/sturdy/public/cmsgemos_gempython-0.3.1.tar.gz .

Create a SOCKS proxy that will allow pip to reach the outer world:

PORT=5000
ssh -D *:$PORT lxplus.cern.ch -N -f

If you get an error saying bind: Address already in use, try with PORT=5001, 5002, ...

Note The proxy expires after some time. Just create it again if pip complains about the network being unreachable.

Define $ELOG_PATH:

export ELOG_PATH=/your/favorite/elog/path

Then execute:

source gem-plotting-tools/setup.sh -c 0.3.1 -g 1.0.0 -G 5 -v 2.0.0 -V 3 -P $PORT

Congratulations, you are done! You can use the usual commands to deactivate your virtualenv and activate it again.

bdorney commented 6 years ago

This part:

To be done once Download a package to enable the use of SOCKS proxies in Python: ssh cmsusr wget https://files.pythonhosted.org/packages/53/12/6bf1d764f128636cef7408e8156b7235b150ea31650d0260969215bb8e7d/PySocks-1.6.8.tar.gz Install it: pip install --user PySocks-1.6.8.tar.gz To be done every time you create a new environment

Could be scripted via something like:

PYSOCKS=$(pip list --user | grep "PySocks")
if [ ! -n "$PYSOCKS" ]
then

    # run the commands to make a user install of PySocks

fi

Of course a system wide installation could be done but this again depends on the sysadmin and if the package could be added at time of machine setup (e.g. through puppet)

This part:

Create a SOCKS proxy that will allow pip to reach the outer world: PORT=5000 ssh -D *:$PORT lxplus.cern.ch -N -f If you get an error saying bind: Address already in use, try with PORT=5001, 5002, ... Note The proxy expires after some time. Just create it again if pip complains about the network being unreachable.

Could also be scripted. The creation of the proxy should return some exit code which can be checked in a loop. For example perform this in a do...while(exit code not zero) and each iteration of the loop increment the port number until a successful port is found. Although maybe someone has a good argument on why automatic port assignment here is a bad idea (I could think of a few but perhaps in the 5000 range it's okay...).

lmoureaux commented 6 years ago

Force-pushed updates:

Caveats:

Next:

jsturdy commented 6 years ago

A workaround for differences between pure bash ($BASH_SOURCE) and zsh can be seen here

lmoureaux commented 6 years ago

A workaround for differences between pure bash ($BASH_SOURCE) and zsh can be seen here

Thanks @jsturdy! The other problem I have with zsh is:

CMD="echo A"
$CMD

Output: bash: A zsh: command not found: echo A

bdorney commented 6 years ago

The problems with zsh have got worse

Could you describe?

Thanks @jsturdy! The other problem I have with zsh is: CMD="echo A" $CMD Output: bash: A zsh: command not found: echo A

In zsh the correct way to do this is via the use of arrays, see for example:

% CMD=(echo A)
% $CMD
A

For more details see the FAQ, specifically section 3.1.

bdorney commented 6 years ago

Additionally can you push the changes you made to setup_gemdaq.sh here:

https://github.com/cms-gem-daq-project/sw_utils

Add a folder scripts and then have setup_gemdaq.sh located there.

I don't want USER to checkout a github repo but I think setup_gemdaq.sh is now sufficiently advanced where I would like to have some subversioning and history. I'll fork it and add my own changes on top of that.

lmoureaux commented 6 years ago

Additionally can you push the changes you made to setup_gemdaq.sh here:

I'll wget the script directly from the repo. Should I use master or another branch?

bdorney commented 6 years ago

I'll wget the script directly from the repo. Should I use master or another branch?

Push it to develop and we will go from there.

lmoureaux commented 6 years ago

I'll wget the script directly from the repo. Should I use master or another branch?

Push it to develop and we will go from there.

I meant in the README instructions.

Additionally I'm not allowed to push to the new repo.

bdorney commented 6 years ago

I'll wget the script directly from the repo

The script is already there, my interest is to incorporate the changes you made, and also some changes I've made based on some feedback from @jsturdy

lmoureaux commented 6 years ago
lmoureaux commented 6 years ago

Removing the WIP tag as testing is now complete. Dev docs update will be in another PR.

bdorney commented 6 years ago

Also upon further reflection we should probably remove all references to vfatqc here since there is no dependence of gem-plotting on vfatqc. e.g. if a USER wants to do only analysis they don't need vfatqc at all.

lmoureaux commented 6 years ago

@bdorney, I'm quite sure you'll have an opinion about how the -pre releases should be handled. Currently we only give instructions for stable and -dev releases.

bdorney commented 6 years ago

@bdorney, I'm quite sure you'll have an opinion about how the -pre releases should be handled. Currently we only give instructions for stable and -dev releases.

What's the use case? Naively I guess a USER is using a stable release and a developer is using either a stable or a dev release. Try to ask @jsturdy what a pre release would be doing and who should be using it. I assume only a sysadmin would be using a pre release and xcheck'ing it installs correctly but I could be mistaken.

lmoureaux commented 6 years ago

travis-ci build did not complete

Travis checks aren't reliable, they fail every now and then because they fail to fetch the Docker images. IIRC there's a "Retry" button somewhere (or is it on Gitlab?), pushing it might "fix" the issue.