search

dguest / pandamonium

Command line scripts to parse panda web api
BSD 3-Clause "New" or "Revised" License
28 stars 17 forks source link

SSL: CERTIFICATE_VERIFY_FAILED Error #24

Closed matthewfeickert closed 4 years ago

matthewfeickert commented 4 years ago

If a user clones pandamonium and then from the repo runs

python2 pandamon user.<username>

they will get the following error:

Traceback (most recent call last):
  File "pandamon", line 287, in <module>
    run()
  File "pandamon", line 247, in run
    reply = urllib2.urlopen(req).read().decode('utf-8')
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1241, in https_open
    context=self._context)
  File "/usr/lib/python2.7/urllib2.py", line 1198, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)>

This can be replicated in the python:2.7 Docker image

$ docker pull python:2.7
$ docker run --rm -it python:2.7 /bin/bash
root@806d2b21d2ab:/# export USER=feickert
root@806d2b21d2ab:/# git clone https://github.com/dguest/pandamonium.git
root@806d2b21d2ab:/# cd pandamonium
root@806d2b21d2ab:/pandamonium# python2 pandamon user.$USER
Traceback (most recent call last):
  File "pandamon", line 287, in <module>
    run()
  File "pandamon", line 247, in run
    reply = urllib2.urlopen(req).read().decode('utf-8')
  File "/usr/local/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/usr/local/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/usr/local/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python2.7/urllib2.py", line 1241, in https_open
    context=self._context)
  File "/usr/local/lib/python2.7/urllib2.py", line 1198, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)>

I haven't properly tried to debug this yet, so it maybe some silly dependency on openssl that just needs to be specified in the docs.

For reference, the python Docker images are built on Debian:

$ docker run --rm -it python:2.7 /bin/bash -c "cat /etc/os-release"
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/

This doesn't seem to be a Linux issue though, as it also happens with macOS VMs in CI.

dguest commented 4 years ago

I have some vague recollection of this. I thought that I saw this and then ran the commands that were listed on the bottom of the bigpanda page:

It is possible to retrieve task data in json format via curl so that it can be used programmatically. Do not use web page scraping; if you need info, ask how to get it. An example curl command is as follows; it takes the same url parameters as the browser version.

If you are using lxplus:

cern-get-sso-cookie -u https://bigpanda.cern.ch/ -o bigpanda.cookie.txt
curl -b bigpanda.cookie.txt -H 'Accept: application/json' -H 'Content-Type: application/json' "https://bigpanda.cern.ch/tasks/?taskname=user.nacken*&days=3&json"

If you are retrieving JSON outside of CERN:

ssh username@lxplus.cern.ch "cern-get-sso-cookie -u https://bigpanda.cern.ch/ -o bigpanda.cookie.txt;"
ssh username@lxplus.cern.ch 'curl -b ~/bigpanda.cookie.txt -H '"'"'Accept: application/json'"'"' -H '"'"'Content-Type: application/json'"'"' "https://bigpanda.cern.ch/tasks/?taskname=user.nacken*&days=3&json"'

The first command performs SSO authentication, the second delivers needed information. You may reuse single cookie until it get expired.

That said, if this is what fixed it for me, I have no idea why.

In anyone understands why this works we should add some documentation. If no one understands why it works, but it works, we should still add some documentation.

matthewfeickert commented 4 years ago

This didn't work right out of the box for me locally, but I can debug more.

In anyone understands why this works we should add some documentation. If no one understands why it works, but it works, we should still add some documentation.

I had naively assumed that as the README mentions

No login required

that it wouldn't matter if the user was on CERN servers or not. Given that it wasn't clear to me that this might be a requirement, I can still open up a PR that adds a snippet on where pandamonium is meant to be used. This would be a bit of a bummer though if I can't figure out how to get it to work remotely from CERN as I wanted to add CI as well.

dguest commented 4 years ago

It works fine on my laptop. I don't know what I ever did to make it work though.

matthewfeickert commented 4 years ago

It works fine on my laptop. I don't know what I ever did to make it work though.

This is great to know though as it is a solid point for me to start debugging! :)

@dguest This may be totally irrelevant, but can you tell me what version of openssl you're running on your Mac?

kratsg commented 4 years ago

If you want to bundle up your own ROOT CERT to figure this out temporarily: https://github.com/kratsg/stare#ssl ... however, I'd suggest first running pip install -U certifi (https://requests.readthedocs.io/en/latest/user/advanced/#ca-certificates) to see if you're not pulling some of the latest ROOT CERTs to trust.

dguest commented 4 years ago

On my mac:

> openssl version
LibreSSL 2.8.3

on our institute's cluster:

> openssl version
OpenSSL 1.0.2k-fips  26 Jan 2017

I have no issues running pandamon on either of them.

@kratsg didn't you get this working in a docker image at some point?

kratsg commented 4 years ago

https://hub.docker.com/r/kratsg/pandamonium

dguest commented 4 years ago

I guess it's a bit unfortunate if anyone ends up installing a docker image (which is 10,000 times larger than the pandamon script) just to run pandamon. @kratsg do you have any idea which of the files in your repository were necessary to get pandamon running?

The weird thing is that pandamon also works fine (with no authentication etc) when inside an image I built to run atlas analysis software. There's no panda or LCG middleware in there.

kratsg commented 4 years ago

This line is needed -- and there is technically LCG middleware (https://github.com/kratsg/dockerimages/blob/master/pandamonium/Dockerfile#L12) -- since you need to download the CA certs in order to "trust CERN".

dguest commented 4 years ago

Interesting, so I guess the reason it works on my laptop is that I (at some point) installed CERN's root certificate?

kratsg commented 4 years ago

Interesting, so I guess the reason it works on my laptop is that I (at some point) installed CERN's root certificate?

yes

matthewfeickert commented 4 years ago

There's probably enough information to figure out what is the minimum amount of software to get this to work, but as some self notes for later. I can see that emi (which after some googling I've learned is European Middleware Initiative) is needed for authentication, but printenvdiffs are showing it is setting quite a bit of environmental variables so I'm not clear on exactly what is actually required.

(pandamon-dev) [feickert@login pandamonium]$ hostname
login.usatlas.org
(pandamon-dev) [feickert@login pandamonium]$ voms-proxy-init -voms atlas
(pandamon-dev) [feickert@login pandamonium]$ kinit feickert@CERN.CH
(pandamon-dev) [feickert@login pandamonium]$ pandamon user.feickert
Traceback (most recent call last):
  File "/home/feickert/workarea/pandamonium/pandamon-dev/bin/pandamon", line 11, in <module>
    load_entry_point('pandamonium', 'console_scripts', 'pandamon')()
  File "/home/feickert/workarea/pandamonium/src/pandamonium/pandamon.py", line 303, in main
    reply = urlopen(req).read().decode('utf-8')
  File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 1258, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/lib64/python2.7/urllib2.py", line 1214, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)>
(pandamon-dev) [feickert@login pandamonium]$ lsetup emi
************************************************************************
Requested:  emi ... 
 Setting up emi 4.0.2-1_200423.fix1 ... 
>>>>>>>>>>>>>>>>>>>>>>>>> Information for user <<<<<<<<<<<<<<<<<<<<<<<<<
 emi:
   Your proxy has 11h:30m:52s remaining
************************************************************************
(pandamon-dev) [feickert@login pandamonium]$ pandamon user.feickert | tail
done        12331     100%   user.feickert.364701.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ1WithSW.e7142_s3126_r9364_p4128._2020-09-03_09-35/ 
done        12330     100%   user.feickert.364705.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ5WithSW.e7142_s3126_r9364_p4128._2020-09-03_09-35/ 
done        12329     100%   user.feickert.364712.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ12WithSW.e7142_s3126_r10201_p4128._2020-09-03_09-35/ 
done        12328     100%   user.feickert.364712.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ12WithSW.e7142_s3126_r9364_p4128._2020-09-03_09-35/ 
done        12322     100%   user.feickert.364711.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ11WithSW.e7142_s3126_r10201_p4128._2020-09-03_09-35/ 
done        12321     100%   user.feickert.364709.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ9WithSW.e7142_s3126_r9364_p4128._2020-09-03_09-35/ 
done        12320     100%   user.feickert.364706.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ6WithSW.e7142_s3126_r9364_p4128._2020-09-03_09-35/ 
done        12319     100%   user.feickert.364702.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ2WithSW.e7142_s3126_r9364_p4128._2020-09-03_09-35/ 
done        12318     100%   user.feickert.364700.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ0WithSW.e7142_s3126_r10201_p4128._2020-09-03_09-35/ 
done        12317     100%   user.feickert.364706.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ6WithSW.e7142_s3126_r10201_p4128._2020-09-03_09-35

though looking at the EMI setup.sh on CVMFS is kinda invormative

(pandamon-dev) [feickert@login pandamonium]$ printenv | grep EMI
EMI_MINBUILDVER_GCC=gcc48
EMI_MINBUILDVER_PYTHON=2.7.5
EMI_PYTHONBIN=python
EMI_TARBALL_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/emi/4.0.2-1_200423.fix1
ATLAS_LOCAL_EMI_VERSION=4.0.2-1_200423.fix1
EMI_UI_CONF=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/emi/4.0.2-1_200423.fix1
(pandamon-dev) [feickert@login pandamonium]$ cat /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/emi/4.0.2-1_200423.fix1/setup.sh 

export EMI_TARBALL_BASE=$ATLAS_LOCAL_ROOT_BASE/x86_64/emi/4.0.2-1_200423.fix1

if [ -e $EMI_TARBALL_BASE/usr/libexec/grid-env.sh ]; then
  source $EMI_TARBALL_BASE/usr/libexec/grid-env.sh
else
  export PATH=$EMI_TARBALL_BASE/core/usr/bin:/bin:$PATH
fi

# and overwrite ...

export PATH=$PATH:$EMI_TARBALL_BASE/usr/bin
export PATH=$PATH:$EMI_TARBALL_BASE/bin
export PATH=$PATH:$EMI_TARBALL_BASE/usr/sbin

# very important to have 64-bit libs first in path
export LD_LIBRARY_PATH=$EMI_TARBALL_BASE/alrbUsr/lib64/dcap:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$EMI_TARBALL_BASE/alrbUsr/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$EMI_TARBALL_BASE/alrbUsr/lib64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$EMI_TARBALL_BASE/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$EMI_TARBALL_BASE/lib64:$LD_LIBRARY_PATH

export GLITE_LOCATION=$EMI_TARBALL_BASE/usr
export GLITE_LOCATION_VAR=$EMI_TARBALL_BASE/var

export LCG_LOCATION=$EMI_TARBALL_BASE/usr
export VOMS_USERCONF=$EMI_TARBALL_BASE/etc/vomses
export SRM_PATH=$EMI_TARBALL_BASE/usr/share/srm
export GLOBUS_LOCATION=$EMI_TARBALL_BASE/usr
export BDII_LIST=lcg-bdii.cern.ch:2170
export X509_VOMSES=$EMI_TARBALL_BASE/etc/vomses
export X509_CERT_DIR=$ATLAS_LOCAL_ROOT_BASE/etc/grid-security-emi/certificates
export X509_VOMS_DIR=$EMI_TARBALL_BASE/etc/grid-security/vomsdir

# allow for switching between python and python3
if [ -z $EMI_PYTHONBIN ]; then
    export EMI_PYTHONBIN="python"
fi
which $EMI_PYTHONBIN >/dev/null 2>&1
if [ $? -ne 0 ]; then
    if [ "$EMI_PYTHONBIN" = "python3" ]; then
      # python3 not found but python points to python 3.X so use it !
      emi_tmpVal=`python -V 2>&1 | \awk '{print $2}' | \cut -d "." -f 1`
      if [ "$emi_tmpVal" = "3" ]; then
          export EMI_PYTHONBIN="python"
      else
          \echo "Error: python version 3 is unavailable"
          return 64    
      fi     
    else
      \echo "Error: $EMI_PYTHONBIN is not found in PATH"
      return 64
    fi
fi

emi_tmpVal=`command -v $EMI_PYTHONBIN`
if [[ $? -ne 0 ]] || [[ ! -e "$emi_tmpVal" ]]; then
    \echo "Warning: $EMI_PYTHONBIN does not seem to exist as a file'; unable to use as interpretor"
fi

emi_pythonFile=`which $EMI_PYTHONBIN`
emi_pythonArch=`file $emi_pythonFile | sed -e 's/.*ELF \(.*\)-bit.*/\1/'`
emi_pythonVersion=`$EMI_PYTHONBIN -V 2>&1 | awk '{print $2}'`
emi_pyVerMajor=`echo $emi_pythonVersion | cut -d "." -f 1`
emi_pyVerMinor=`echo $emi_pythonVersion | cut -d "." -f 2`
let emi_pyVerN=`expr $emi_pyVerMajor \* 10000 + $emi_pyVerMinor \* 100`

if [ "$emi_pythonArch" = "32" ]; then
    emi_pyLibPath=( `find $LCG_LOCATION/lib -name site-packages -type d | sort -ur ` )
else
    emi_pyLibPath=( `find $LCG_LOCATION/lib64 -name site-packages -type d | sort -ur ` )
fi

for item in ${emi_pyLibPath[@]}; do
    emi_thisLib=`echo $item | sed 's|.*python\(.*\)\/.*|\1|'`
    emi_libVerMajor=`echo $emi_thisLib | cut -d "." -f 1`
    emi_libVerMinor=`echo $emi_thisLib | cut -d "." -f 2`
    let emi_libVerN=`expr $emi_libVerMajor \* 10000 + $emi_libVerMinor \* 100`
    if [ $emi_pyVerN -ge $emi_libVerN ]; then      
      if [ -z $PYTHONPATH ]; then
        export PYTHONPATH=$item
      else
        export PYTHONPATH=$item:$PYTHONPATH   
      fi
      # for 64-bit, also add 32-bit lib if it exists
      item2=`echo $item | sed -e 's|lib64|lib|g'`
      if [ -d $item2 ]; then
        export PYTHONPATH=$PYTHONPATH:$item2
      fi
      break
    fi
done

if [ -z $PERL5LIB ]; then
  export PERL5LIB=$EMI_TARBALL_BASE/usr/lib64/perl5/vendor_perl
else
  export PERL5LIB=$PERL5LIB:$EMI_TARBALL_BASE/usr/lib64/perl5/vendor_perl
fi
export PERL5LIB="$PERL5LIB:$EMI_TARBALL_BASE/usr/lib/perl5/vendor_perl:$EMI_TARBALL_BASE/usr/share/perl5:$EMI_TARBALL_BASE/usr/share/perl5/vendor_perl"

export MANPATH=$EMI_TARBALL_BASE/usr/share/man:$MANPATH

export GFAL_PLUGIN_DIR=$EMI_TARBALL_BASE/usr/lib64/gfal2-plugins/
export GFAL_CONFIG_DIR=$EMI_TARBALL_BASE/etc/gfal2.d/

if [ "$emi_pyVerMajor" = "3" ]; then
  if [ ! -e "/lib64/libpython3.6m.so.1.0" ]; then
    export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$EMI_TARBALL_BASE/usr/local/lib64"
  fi 
  export PATH="$EMI_TARBALL_BASE/wrappers:$PATH"
fi

unset emi_pythonFile emi_pythonArch emi_pythonVersion emi_pyVerMajor emi_pyVerMinor emi_pyVerN emi_pyLibPath emi_thisLib emi_libVerMajor emi_libVerMinor emi_libVerN emi_tmpVal

export EMI_MINBUILDVER_GCC=gcc48
export EMI_MINBUILDVER_PYTHON=2.7.5

which kinda makes sense with what @kratsg has: https://github.com/kratsg/dockerimages/blob/b346d600671ba342fdce66fe4432930fd1b2ee11/pandamonium/Dockerfile#L8-L9

matthewfeickert commented 4 years ago

Comments from @kratsg:

For these files in the Docker image

ADD files/ca.repo /etc/yum.repos.d/ca.repo
ADD files/wlcg-centos7.repo /etc/yum.repos.d/wlcg-centos7.repo
ADD files/RPM-GPG-KEY-wlcg /etc/pki/rpm-gpg/RPM-GPG-KEY-wlcg

the ca.repo gets you ca-policy-egi-core and the wlcg gets you the atlas stuff: wlcg-voms-atlas. These are all public keys. You might be able to get the wlcg repo from https://linuxsoft.cern.ch/wlcg/centos7/x86_64/wlcg-repo-1.0.0-1.el7.noarch.rpm if you're going to be in a CentOS environment

yum install https://linuxsoft.cern.ch/wlcg/centos7/x86_64/wlcg-repo-1.0.0-1.el7.noarch.rpm

and the same for the ca.repo one, which is pulling from http://linuxsoft.cern.ch/mirror/repository.egi.eu/sw/production/cas/1/current/. Then you don't necessarily need the RPM-GPG-KEY anymore since that'll be handled by yum install.

Here's the ca.repo instructions: https://wiki.egi.eu/wiki/EGI_IGTF_Release#Installation and the voms instructions: https://twiki.cern.ch/twiki/bin/view/LCG/VOMSLSCfileConfiguration You can't get by without these two things — they're hard requirements.

matthewfeickert commented 4 years ago

From @tmaeno (a panda-client dev!):

The minimum requirement is that voms-proxy-info works correctly. "lsetup emi" sets environment variables for VOMS such as X509* and generates a grid proxy in /tmp.

(Aside: panda-client is on GitHub! We finally know where the source is.)

dguest commented 4 years ago

I think there's some confusion about "minimum requirement":

Just to give a basic example, this returns something from the command line on my mac:

curl -H '{"Accept": "application/json", "Content-Type": "application/json"}' 'https://bigpanda.cern.ch/tasks/?username=Dan+Guest&taskname=user.dguest%2A&limit=1&days=100&json=1'

pandamon is basically taking advantage of this interface, which obviously doesn't depend on panda client.

matthewfeickert commented 4 years ago

Just to give a basic example, this returns something from the command line on my mac:

curl -H '{"Accept": "application/json", "Content-Type": "application/json"}' 'https://bigpanda.cern.ch/tasks/?username=Dan+Guest&taskname=user.dguest%2A&limit=1&days=100&json=1'

You apparently have certificates installed on your Mac then as on my machine

$ curl -H '{"Accept": "application/json", "Content-Type": "application/json"}' 'https://bigpanda.cern.ch/tasks/?username=Dan+Guest&taskname=user.dguest%2A&limit=1&days=100&json=1'
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

same in a Docker image

$ docker run --rm -it debian:buster
# apt-get update && apt-get install curl
# curl -H '{"Accept": "application/json", "Content-Type": "application/json"}' 'https://bigpanda.cern.ch/tasks/?username=Dan+Guest&taskname=user.dguest%2A&limit=1&days=100&json=1'
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
dguest commented 4 years ago

and oddly, this script doesn't work for me: 

#!/usr/bin/env python3

from urllib.request import Request, urlopen
from urllib.parse import urlencode

header = {"Accept": "application/json", "Content-Type": "application/json"}
pars = {
    'json': 1,
    'days': 100,
    'username': 'Dan Guest',
    'taskname': 'user.dguest.*'}
url='https://bigpanda.cern.ch/tasks/?' + urlencode(pars)

if __name__ == '__main__':
    req = Request(url=url,headers=header)
    print(req.get_full_url())
    print(urlopen(req).read().decode('utf-8'))

it throws the same error

urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)>
dguest commented 4 years ago

@matthewfeickert, I see this when I ask curl for more info

> curl -vvv -H '{"Accept": "application/json", "Content-Type": "application/json"}' 'https://bigpanda.cern.ch/tasks/?username=Dan+Guest&taskname=user.dguest%2A&datasets=True&limit=10000&days=100&json=1'
*   Trying 188.185.76.222...
* TCP_NODELAY set
* Connected to bigpanda.cern.ch (188.185.76.222) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: DC=ch; DC=cern; OU=computers; CN=aipanda200.cern.ch
*  start date: Jul  3 22:19:02 2020 GMT
*  expire date: Aug  7 22:19:02 2021 GMT
*  subjectAltName: host "bigpanda.cern.ch" matched cert's "bigpanda.cern.ch"
*  issuer: DC=ch; DC=cern; CN=CERN Grid Certification Authority
*  SSL certificate verify ok.
> GET /tasks/?username=Dan+Guest&taskname=user.dguest%2A&datasets=True&limit=10000&days=100&json=1 HTTP/1.1
> Host: bigpanda.cern.ch
> User-Agent: curl/7.64.1
> Accept: */*
> {"Accept": "application/json", "Content-Type": "application/json"}
> 
< HTTP/1.1 200 OK
< Server: nginx
< Date: Fri, 11 Sep 2020 16:08:46 GMT
< Content-Type: application/json
< Content-Length: 296077
< Connection: keep-alive
< X-Frame-Options: SAMEORIGIN
<
matthewfeickert commented 4 years ago

@dguest Yeah, and inline with @kratsg Docker image I see that the first roadblock that is hit on my machine is the CAfile

$ curl -vvv -H '{"Accept": "application/json", "Content-Type": "application/json"}' 'https://bigpanda.cern.ch/tasks/?username=Dan+Guest&taskname=user.dguest%2A&limit=1&days=100&json=1'
*   Trying 188.185.76.222:443...
* TCP_NODELAY set
* Connected to bigpanda.cern.ch (188.185.76.222) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
dguest commented 4 years ago

OK, well I have a CERN root certificate installed in my keychain, I'm guessing (but it's a wild guess) that curl is somehow accessing it. And I guess python is (sometimes) also accessing it. There's probably some way to configure this on any machine, but I'd have to learn more about certificates...

kratsg commented 4 years ago

Can you check if pip install certifi fixes the standard issues now? In the past, the ROOT CERN CA was always manually signed and not from a standard CA -- but this has changed now, so it should be workable with just the mozilla certificate bundle.

matthewfeickert commented 4 years ago

Can you check if pip install certifi fixes the standard issues now?

@kratsg I've already tried this in the past and it fails. As an example again though:

$ docker run --rm -it python:3.8 /bin/bash
root@3e376ce251e5:/# pip install -q --upgrade pip setuptools wheel
root@3e376ce251e5:/# pip install -q certifi
root@3e376ce251e5:/# pip list | grep cert
certifi    2020.6.20
root@3e376ce251e5:/# curl -vvv -H '{"Accept": "application/json", "Content-Type": "application/json"}' 'https://bigpanda.cern.ch/tasks/?username=Dan+Guest&taskname=user.dguest%2A&limit=1&days=100&json=1'
*   Trying 188.185.76.222...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x562d6e629f50)
* Connected to bigpanda.cern.ch (188.185.76.222) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
kratsg commented 4 years ago

Just checked bigpanda's cert chain: https://www.digicert.com/help/

Screen Shot 2020-09-11 at 2 57 24 PM

The main reason is because the CERN CA is not a ROOT trusted authority (which is the fundamental issue anyway).

In fact, bigpanda didn't switch their certificate yet to Sectigo RSA Organization which CERN switched to about a year or so ago... I'll ping IT.

kratsg commented 4 years ago

https://its.cern.ch/jira/browse/ATLASPANDA-530

dguest commented 4 years ago

As someone commented on the above ticket, the certificate issue seems solved. And when I run pandamon on my laptop, this now works no matter how I install it. @matthewfeickert, can you also run pandamon on a minimal setup?

matthewfeickert commented 4 years ago

As someone commented on the above ticket, the certificate issue seems solved. And when I run pandamon on my laptop, this now works no matter how I install it. @matthewfeickert, can you also run pandamon on a minimal setup?

Yup!

$ docker run --rm -it -e USER=$USER python:3.8 /bin/bash
root@12405c956c4f:/# git clone --branch feat/package-as-library --single-branch https://github.com/matthewfeickert/pandamonium.git
Cloning into 'pandamonium'...
remote: Enumerating objects: 88, done.
remote: Counting objects: 100% (88/88), done.
remote: Compressing objects: 100% (49/49), done.
remote: Total 415 (delta 38), reused 73 (delta 32), pack-reused 327
Receiving objects: 100% (415/415), 98.77 KiB | 2.24 MiB/s, done.
Resolving deltas: 100% (207/207), done.
root@12405c956c4f:/# cd pandamonium/
root@12405c956c4f:/pandamonium# python -m pip install -e .
Obtaining file:///pandamonium
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
    Preparing wheel metadata ... done
Collecting panda-client>=1.0
  Downloading panda-client-1.4.36.tar.gz (177 kB)
     |████████████████████████████████| 177 kB 858 kB/s 
Building wheels for collected packages: panda-client
  Building wheel for panda-client (setup.py) ... done
  Created wheel for panda-client: filename=panda_client-1.4.36-py3-none-any.whl size=131052 sha256=94ee583493037ac8b755c1deab46c75ee89c6095de0fe65271ea8a63472caa86
  Stored in directory: /root/.cache/pip/wheels/f2/42/49/bbc160aba6d490c40a6fe58fd0c0ea5f20e9df254ec1627a92
Successfully built panda-client
Installing collected packages: panda-client, pandamonium
  Running setup.py develop for pandamonium
Successfully installed panda-client-1.4.36 pandamonium
root@12405c956c4f:/pandamonium# which pandamon
/usr/local/bin/pandamon
root@12405c956c4f:/pandamonium# pandamon -d 50 | tail
done        12331     100%   user.feickert.364701.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ1WithSW.e7142_s3126_r9364_p4128._2020-09-03_09-35/ 
done        12330     100%   user.feickert.364705.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ5WithSW.e7142_s3126_r9364_p4128._2020-09-03_09-35/ 
done        12329     100%   user.feickert.364712.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ12WithSW.e7142_s3126_r10201_p4128._2020-09-03_09-35/ 
done        12328     100%   user.feickert.364712.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ12WithSW.e7142_s3126_r9364_p4128._2020-09-03_09-35/ 
done        12322     100%   user.feickert.364711.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ11WithSW.e7142_s3126_r10201_p4128._2020-09-03_09-35/ 
done        12321     100%   user.feickert.364709.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ9WithSW.e7142_s3126_r9364_p4128._2020-09-03_09-35/ 
done        12320     100%   user.feickert.364706.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ6WithSW.e7142_s3126_r9364_p4128._2020-09-03_09-35/ 
done        12319     100%   user.feickert.364702.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ2WithSW.e7142_s3126_r9364_p4128._2020-09-03_09-35/ 
done        12318     100%   user.feickert.364700.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ0WithSW.e7142_s3126_r10201_p4128._2020-09-03_09-35/ 
done        12317     100%   user.feickert.364706.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ6WithSW.e7142_s3126_r10201_p4128._2020-09-03_09-35/

@dguest This means this Issue can get closed and we can move on PR #29!