search

cvmfs-contrib / cvmfs-csi

CSI driver for CernVM-FS
Apache License 2.0
19 stars 29 forks source link

Won't mount repositories #77

Closed DiamantisPa closed 1 year ago

DiamantisPa commented 1 year ago

I've been trying to to deploy cvmfs-csi on Kubernetes cluster but listing the cvmfs folder shows nothing.

I followed the instructions for deploying here: https://github.com/cvmfs-contrib/cvmfs-csi/blob/master/docs/deploying.md and for testing here: https://github.com/cvmfs-contrib/cvmfs-csi/blob/master/docs/how-to-use.md.

If i execute into the cvmfs-demo pod ls -l /my-cvmfs/atlas.cern.ch there is a delay but I just get the error file or directory not found.

My system is: OS: Centos8 Kubernetes: 1.26.0

These are the logs from a csi nodeplugin pod:

Defaulted container "registrar" out of: registrar, nodeplugin I0427 16:22:32.995552 589994 main.go:166] Version: v2.5.1 I0427 16:22:32.995594 589994 main.go:167] Running node-driver-registrar in mode=registration I0427 16:22:32.996079 589994 main.go:191] Attempting to open a gRPC connection with: "/csi/csi.sock" I0427 16:22:32.996117 589994 connection.go:154] Connecting to unix:///csi/csi.sock W0427 16:22:42.996364 589994 connection.go:173] Still connecting to unix:///csi/csi.sock W0427 16:22:52.997075 589994 connection.go:173] Still connecting to unix:///csi/csi.sock I0427 16:22:54.955114 589994 main.go:198] Calling CSI driver to discover driver name I0427 16:22:54.955147 589994 connection.go:183] GRPC call: /csi.v1.Identity/GetPluginInfo I0427 16:22:54.955152 589994 connection.go:184] GRPC request: {} I0427 16:22:54.960901 589994 connection.go:186] GRPC response: {"name":"cvmfs.csi.cern.ch","vendor_version":"v2.1.2"} I0427 16:22:54.960968 589994 connection.go:187] GRPC error: I0427 16:22:54.960977 589994 main.go:208] CSI driver name: "cvmfs.csi.cern.ch" I0427 16:22:54.961008 589994 node_register.go:53] Starting Registration Server at: /registration/cvmfs.csi.cern.ch-reg.sock I0427 16:22:54.961162 589994 node_register.go:62] Registration Server started at: /registration/cvmfs.csi.cern.ch-reg.sock I0427 16:22:54.961417 589994 node_register.go:92] Skipping HTTP server because endpoint is set to: "" I0427 16:22:55.685156 589994 main.go:102] Received GetInfo call: &InfoRequest{} I0427 16:22:55.685580 589994 main.go:109] "Kubelet registration probe created" path="/var/lib/kubelet/plugins/cvmfs.csi.cern.ch/registration" I0427 16:22:55.705064 589994 main.go:120] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}

I also had the same behaviour by changing the CVMFS_HTTP_PROXY='http://ca-proxy.cern.ch:3128' to DIRECT.

When trying to mount a repository using the repository storage class paratemeter, i get the error of FailedtoMount in the demo pod.

Would appreciate any ideas how to debug this.

gman0 commented 1 year ago

Hi @DiamantisPa, maybe you need to supply configs for CERN's repos? You can try to add this into your cvmfs-csi-config-d ConfigMap:

  cern.ch.conf: |
    CVMFS_SERVER_URL="http://cvmfs-stratum-one.cern.ch/cvmfs/@fqrn@;http://cernvmfs.gridpp.rl.ac.uk/cvmfs/@fqrn@;http://cvmfs.racf.bnl.gov/cvmfs/@fqrn@;http://cvmfs.fnal.gov/cvmfs/@fqrn@"
    CVMFS_PUBLIC_KEY='/etc/cvmfs/config.d/cern.ch.pub'

  cern.ch.pub: |
    -----BEGIN PUBLIC KEY-----
    MIGeMA0GCSqGSIb3DQEBAQUAA4GMADCBiAKBgHvy9PFYteIdwD5OxEosZoqpSUxP
    LddA+zRwD0lWfavbFoCJOJS23zPpIimQ0gzH3FAebR3x9JzVVa3JFLxIUj4Fv3d6
    /3fWhyK4Nmxc7B1Qh7wrRYlYtcDszGGk2DB0ETI83Jbzvfx1slvhMkqjWQ0c30w8
    BWglnUvav0hRRwJRAgMBAAE=
    -----END PUBLIC KEY-----
DiamantisPa commented 1 year ago

Hey @gman0, this is the cvmfs-csi-config-d i am using:

apiVersion: v1
data:
  cern.ch.conf: |
    CVMFS_SERVER_URL="http://cvmfs-stratum-one.cern.ch/cvmfs/@fqrn@;http://cernvmfs.gridpp.rl.ac.uk/cvmfs/@fqrn@;http://cvmfs.racf.bnl.gov/cvmfs/@fqrn@;http://cvmfs.fnal.gov/cvmfs/@fqrn@"
    CVMFS_PUBLIC_KEY='/etc/cvmfs/config.d/cern.ch.pub'
  cern.ch.pub: |
    -----BEGIN PUBLIC KEY-----
    MIGeMA0GCSqGSIb3DQEBAQUAA4GMADCBiAKBgHvy9PFYteIdwD5OxEosZoqpSUxP
    LddA+zRwD0lWfavbFoCJOJS23zPpIimQ0gzH3FAebR3x9JzVVa3JFLxIUj4Fv3d6
    /3fWhyK4Nmxc7B1Qh7wrRYlYtcDszGGk2DB0ETI83Jbzvfx1slvhMkqjWQ0c30w8
    BWglnUvav0hRRwJRAgMBAAE=
    -----END PUBLIC KEY-----
  ilc.desy.de.conf: |
    CVMFS_SERVER_URL='http://grid-cvmfs-one.desy.de:8000/cvmfs/@fqrn@;http://cvmfs-stratum-one.cern.ch:8000/cvmfs/@fqrn@;http://cvmfs-egi.gridpp.rl.ac.uk:8000/cvmfs/@fqrn@'
    CVMFS_PUBLIC_KEY='/etc/cvmfs/config.d/ilc.desy.de.pub'
  ilc.desy.de.pub: |
    -----BEGIN PUBLIC KEY-----
    MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA3pgrEIimdCPWG9cuhQ0d
    ZWfYxvHRz5hL4HvQlmvikLIlHxs2EApnGyAWdaHAeQ4IiY+JXQnGsS8e5Gr2cZRb
    Y6Ya19GrjMY1wv8fc+uU9kXp7TbHpl3mSQxERG4+wqosN3+IUaPHdnWGP11idOQB
    I0HsJ7PTRk9quFjx1ppkVITZN07+OdGBIzLK6SyDjy49IfL6DVZUH/Oc99IiXg6E
    NDN2UecnnjDEmcvQh2UjGSQ+0NHe36ttQKEnK58GvcSj2reUEaVKLRvPcrzT9o7c
    ugxcbkBGB3VfqSgfun8urekGEHx+vTNwu8rufBkAbQYdYCPBD3AGqSg+Kgi7i/gX
    cwIDAQAB
    -----END PUBLIC KEY-----
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: cvmfs
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2023-04-27T16:22:32Z"
  labels:
    app: cvmfs-csi
    app.kubernetes.io/managed-by: Helm
    chart: cvmfs-csi-2.1.2
    heritage: Helm
  name: cvmfs-csi-config-d
  namespace: default
  resourceVersion: "7657192"
  uid: fce78703-c95c-4b67-8e49-09558bc75f53

I also tried loading ilc.desy.de running kubectl exec -it cvmfs-demo -- ls -l /my-cvmfs/ilc.desy.de but with no result.

Checked to mount the repository directly inside the the csi nodeplugin pod: kubectl exec -it cvmfs-cvmfs-csi-nodeplugin-bfr6v -- mount -t cvmfs atlas.cern.ch /mnt -v

but i got

Defaulted container "registrar" out of: registrar, nodeplugin ERRO[0000] exec failed: unable to start container process: exec: "mount": executable file not found in $PATH command terminated with exit code 255

gman0 commented 1 year ago

@DiamantisPa can you run kubectl exec -it cvmfs-cvmfs-csi-nodeplugin-bfr6v -c nodeplugin -- mount -t cvmfs atlas.cern.ch /mnt -v instead please?

gman0 commented 1 year ago

And also can you check that in automount container the configmap is actually mounted in /etc/cvmfs/config.d? There was a bug in the chart prior to v2.1.2 (fixed in https://github.com/cvmfs-contrib/cvmfs-csi/releases/tag/v2.1.2).

DiamantisPa commented 1 year ago

@gman0 Right thanks for the notice. So i get

CernVM-FS: running with credentials 999:997 CernVM-FS: loading Fuse module... Failed to initialize root file catalog (16 - file catalog failure) command terminated with exit code 32

when running with CVMFS_HTTP_PROXY='http://ca-proxy.cern.ch:3128' or CVMFS_HTTP_PROXY='DIRECT'.

So, i guess its a client configuration error?

This is my cvmfs-csi-default-local ConfigMap

apiVersion: v1
data:
  default.local: |
    CVMFS_USE_GEOAPI=yes
    CVMFS_HTTP_PROXY='http://ca-proxy.cern.ch:3128'

    # It is advised to change these configs in the cache section of the helm values
    # and leave them unchanged here, so they auto-generate.
    CVMFS_QUOTA_LIMIT=1000
    CVMFS_CACHE_BASE=/cvmfs-localcache
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: cvmfs
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2023-04-27T16:22:32Z"
  labels:
    app: cvmfs-csi
    app.kubernetes.io/managed-by: Helm
    chart: cvmfs-csi-2.1.2
    heritage: Helm
  name: cvmfs-csi-default-local
  namespace: default
  resourceVersion: "7703284"
  uid: 756ca971-e91a-490e-8305-28d249ba905c

Running kubectl exec -it cvmfs-cvmfs-csi-nodeplugin-bfr6v -c nodeplugin -- ls /etc/cvmfs/config.d i can see the config files

[root@bril-master-1 ~]# kubectl exec -it cvmfs-cvmfs-csi-nodeplugin-bfr6v -c nodeplugin -- ls /etc/cvmfs/config.d
cern.ch.conf  cern.ch.pub  ilc.desy.de.conf  ilc.desy.de.pub

Also checked the Helm chart version its the v2.1.2

Are there any other logs that i can look into ?

DiamantisPa commented 1 year ago

Sorry, you mentioned to check that the automount container has the config files, not the nodeplugin container.

[root@bril-master-1 ~]# kubectl exec -it cvmfs-demo -- ls /etc/cvmfs/config.d
ls: /etc/cvmfs/config.d: No such file or directory
command terminated with exit code 1

The folder cvmfs doesn't seem to exist in the automount container.

gman0 commented 1 year ago

Yes, please make sure you deploy the driver with correctly set extraVolumes and extraVolumeMounts, or use the 2.1.2 chart.

DiamantisPa commented 1 year ago

After mentioning the automount container, I had a further look into the repo and saw that each nodeplugin pod should have 3 containers, one registrar, one nodeplugin and the automount.

When i deploy using the Helm chart from the repo

helm install cvmfs cern/cvmfs-csi

The automount container doesn't seem to be created in the pod. There is only the nodeplugin container and the registrar container.

So i tried to deploy from source

helm install cvmfs ./cvmfs-csi/deployments/helm/cvmfs-csi

Now the automount container is initialized but fails with exit code 127, some file missing.

By the way when deploying from source the default value cvmfs-csi:latest fails since there is no such tag in the repository. It should be v2.1.2 instead.

gman0 commented 1 year ago

@DiamantisPa Sorry for the confusion. Can you please revert the changes in your deployment (or deploy 2.1.2), check again that in the nodeplugin container /etc/cvmfs/config.d has the files you expect?

Then, please paste the output of mount -t cvmfs atlas.cern.ch /mnt -v -o debug from that container.

DiamantisPa commented 1 year ago

@gman0 Thanks for the guidelines. I re-deployed cvmfs the v2.1.2. 

[root@bril-master-1 example-cvmfs]# helm list
NAME    NAMESPACE   REVISION    UPDATED                                     STATUS      CHART           APP VERSION
cvmfs   default     1           2023-05-09 13:54:53.005889708 +0200 CEST    deployed    cvmfs-csi-2.1.2 2.1.2

I edited the cvmfs-csi-config-d to add the https proxy for the atlas repository based on the Atlas twiki

apiVersion: v1
data:
  atlas.cern.ch.local: |
    CVMFS_HTTP_PROXY='http://ca-proxy-atlas.cern.ch:3128;http://ca-proxy.cern.ch:3128'
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: cvmfs
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2023-05-09T11:54:53Z"
  labels:
    app: cvmfs-csi
    app.kubernetes.io/managed-by: Helm
    chart: cvmfs-csi-2.1.2
    heritage: Helm
  name: cvmfs-csi-config-d
  namespace: default
  resourceVersion: "9838570"
  uid: c744c574-3201-407a-b85b-9e0040f0c3a9

I can see inside the nodeplugin container at /etc/cvmfs/config.d the .local file.

sh-4.2# ls /etc/cvmfs/config.d/
atlas.cern.ch.local

When i run mount -t cvmfs atlas.cern.ch /mnt -v -o debug inside the nodeplugin container, it fails.

sh-4.2# mount -t cvmfs atlas.cern.ch /mnt -v -o debug
Debug: using library /usr/lib64/libcvmfs_fuse_stub.so
CernVM-FS: running with credentials 999:997
CernVM-FS: running in debug mode
CernVM-FS: loading Fuse module... (cvmfs) Parsing config file /etc/cvmfs/default.conf    [05-09-2023 12:09:57 UTC]
(cvmfs) execve'd /bin/sh (PID: 3481015)    [05-09-2023 12:09:57 UTC]
(cvmfs) Parsing config file /etc/cvmfs/default.d/50-cern.conf    [05-09-2023 12:09:57 UTC]
(cvmfs) execve'd /bin/sh (PID: 3481017)    [05-09-2023 12:09:58 UTC]
(cvmfs) Parsing config file /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/default.conf    [05-09-2023 12:09:58 UTC]
(cvmfs) configuration repository directory does not exist: /cvmfs/cvmfs-config.cern.ch/etc/cvmfs    [05-09-2023 12:09:58 UTC]
(cvmfs) Parsing config file /etc/cvmfs/default.local    [05-09-2023 12:09:58 UTC]
(cvmfs) execve'd /bin/sh (PID: 3481020)    [05-09-2023 12:09:58 UTC]
(cvmfs) Parsing config file /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d/cern.ch.conf    [05-09-2023 12:09:58 UTC]
(cvmfs) configuration repository directory does not exist: /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d    [05-09-2023 12:09:58 UTC]
(cvmfs) Parsing config file /etc/cvmfs/domain.d/cern.ch.conf    [05-09-2023 12:09:58 UTC]
(cvmfs) execve'd /bin/sh (PID: 3481023)    [05-09-2023 12:09:58 UTC]
(cvmfs) Parsing config file /etc/cvmfs/domain.d/cern.ch.local    [05-09-2023 12:09:58 UTC]
(cvmfs) Parsing config file /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/config.d/atlas.cern.ch.conf    [05-09-2023 12:09:58 UTC]
(cvmfs) configuration repository directory does not exist: /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/config.d    [05-09-2023 12:09:58 UTC]
(cvmfs) Parsing config file /etc/cvmfs/config.d/atlas.cern.ch.conf    [05-09-2023 12:09:58 UTC]
(cvmfs) Parsing config file /etc/cvmfs/config.d/atlas.cern.ch.local    [05-09-2023 12:09:58 UTC]
(cvmfs) execve'd /bin/sh (PID: 3481026)    [05-09-2023 12:09:58 UTC]
(cvmfs) Options:
CVMFS_BACKOFF_INIT=2    # from /etc/cvmfs/default.conf
CVMFS_BACKOFF_MAX=10    # from /etc/cvmfs/default.conf
CVMFS_BASE_ENV=1    # from /etc/cvmfs/default.conf
CVMFS_CACHE_BASE=/cvmfs-localcache    # from /etc/cvmfs/default.local
CVMFS_CHECK_PERMISSIONS=yes    # from /etc/cvmfs/default.conf
CVMFS_CLAIM_OWNERSHIP=yes    # from /etc/cvmfs/default.conf
CVMFS_CLIENT_PROFILE=    # from /etc/cvmfs/default.conf
CVMFS_CONFIG_REPOSITORY=cvmfs-config.cern.ch    # from /etc/cvmfs/default.d/50-cern.conf
CVMFS_DEFAULT_DOMAIN=cern.ch    # from /etc/cvmfs/default.d/50-cern.conf
CVMFS_HOST_RESET_AFTER=1800    # from /etc/cvmfs/default.conf
CVMFS_HTTP_PROXY='http://ca-proxy-atlas.cern.ch:3128;http://ca-proxy.cern.ch:3128'    # from /etc/cvmfs/config.d/atlas.cern.ch.local
CVMFS_KEYS_DIR=/etc/cvmfs/keys/cern.ch    # from /etc/cvmfs/domain.d/cern.ch.conf
CVMFS_LOW_SPEED_LIMIT=1024    # from /etc/cvmfs/default.conf
CVMFS_MAX_RETRIES=1    # from /etc/cvmfs/default.conf
CVMFS_MOUNT_DIR=/cvmfs    # from /etc/cvmfs/default.conf
CVMFS_NFILES=131072    # from /etc/cvmfs/default.conf
CVMFS_PROXY_RESET_AFTER=300    # from /etc/cvmfs/default.conf
CVMFS_QUOTA_LIMIT=1000    # from /etc/cvmfs/default.local
CVMFS_RELOAD_SOCKETS=/var/run/cvmfs    # from /etc/cvmfs/default.conf
CVMFS_SEND_INFO_HEADER=no    # from /etc/cvmfs/default.conf
CVMFS_SERVER_URL='http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch;http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch;http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch;http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch'    # from /etc/cvmfs/domain.d/cern.ch.conf
CVMFS_SHARED_CACHE=yes    # from /etc/cvmfs/default.conf
CVMFS_STRICT_MOUNT=no    # from /etc/cvmfs/default.conf
CVMFS_TIMEOUT=5    # from /etc/cvmfs/default.conf
CVMFS_TIMEOUT_DIRECT=10    # from /etc/cvmfs/default.conf
CVMFS_USER=cvmfs    # from /etc/cvmfs/default.conf
CVMFS_USE_GEOAPI=yes    # from /etc/cvmfs/domain.d/cern.ch.conf
    [05-09-2023 12:09:58 UTC]
(cvmfs) setting up cache manager instance default    [05-09-2023 12:09:58 UTC]
(quota) trying to connect to existing pipe    [05-09-2023 12:09:58 UTC]
(cvmfs) execve'd /usr/bin/cvmfs2 (PID: 3481028)    [05-09-2023 12:09:58 UTC]
(quota) starting quota manager    [05-09-2023 12:09:58 UTC]
(quota) connected to a new cache manager    [05-09-2023 12:09:58 UTC]
(quota) received limit 1048576000, threshold 524288000    [05-09-2023 12:09:58 UTC]
(cvmfs) CernVM-FS: quota initialized, current size 0MB    [05-09-2023 12:09:58 UTC]
(cvmfs) CernVM-FS: using public key(s) /etc/cvmfs/keys/cern.ch/cern-it1.cern.ch.pub:/etc/cvmfs/keys/cern.ch/cern-it4.cern.ch.pub:/etc/cvmfs/keys/cern.ch/cern-it5.cern.ch.pub    [05-09-2023 12:09:58 UTC]
(download) first fallback proxy group 2    [05-09-2023 12:09:58 UTC]
(download) full proxy list http://ca-proxy-atlas.cern.ch:3128;http://ca-proxy.cern.ch:3128    [05-09-2023 12:09:58 UTC]
(download) resolving 2 proxy addresses    [05-09-2023 12:09:58 UTC]
(dns) failed to resolve ca-proxy-atlas.cern.ch - 2 (DNS query timeout), ttl 60    [05-09-2023 12:10:07 UTC]
(dns) failed to resolve ca-proxy.cern.ch - 2 (DNS query timeout), ttl 60    [05-09-2023 12:10:07 UTC]
(download) failed to resolve IP addresses for ca-proxy-atlas.cern.ch (2 - DNS query timeout)    [05-09-2023 12:10:07 UTC]
(download) failed to resolve IP addresses for ca-proxy.cern.ch (2 - DNS query timeout)    [05-09-2023 12:10:07 UTC]
(download) installed 2 proxies in 2 load-balance groups    [05-09-2023 12:10:07 UTC]
(download) switching proxy from (none) to http://ca-proxy-atlas.cern.ch:3128 (set proxies)    [05-09-2023 12:10:07 UTC]
(download) requesting ordered server list from http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch    [05-09-2023 12:10:07 UTC]
(download) replacing @proxy@ by ca-proxy-atlas.cern.ch    [05-09-2023 12:10:07 UTC]
(download) escaped http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/api/v1.0/geo/ca-proxy-atlas.cern.ch/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch to http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/api/v1.0/geo/ca-proxy-atlas.cern.ch/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch    [05-09-2023 12:10:07 UTC]
(download) Verify downloaded url http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch, proxy http://ca-proxy-atlas.cern.ch:3128 (curl error 7)    [05-09-2023 12:10:07 UTC]
(download) Trying again on same curl handle, same url: 1, error code 6    [05-09-2023 12:10:07 UTC]
(download) backing off for 1688 ms    [05-09-2023 12:10:07 UTC]
(download) Verify downloaded url http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch, proxy http://ca-proxy-atlas.cern.ch:3128 (curl error 7)    [05-09-2023 12:10:09 UTC]
(download) Trying again on same curl handle, same url: 0, error code 6    [05-09-2023 12:10:09 UTC]
(download) switching proxy from http://ca-proxy-atlas.cern.ch:3128 to http://ca-proxy.cern.ch:3128 (failed proxy)    [05-09-2023 12:10:09 UTC]
(download) 1 proxies remain in group    [05-09-2023 12:10:09 UTC]
(download) replacing @proxy@ by ca-proxy.cern.ch    [05-09-2023 12:10:09 UTC]
(download) escaped http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/api/v1.0/geo/ca-proxy.cern.ch/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch to http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/api/v1.0/geo/ca-proxy.cern.ch/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch    [05-09-2023 12:10:09 UTC]
(download) Verify downloaded url http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch, proxy http://ca-proxy.cern.ch:3128 (curl error 7)    [05-09-2023 12:10:09 UTC]
(download) download failed (error 6 - proxy connection problem)    [05-09-2023 12:10:09 UTC]
(download) GeoAPI request http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch failed with error 6 [proxy connection problem]    [05-09-2023 12:10:09 UTC]
(download) requesting ordered server list from http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch    [05-09-2023 12:10:09 UTC]
(download) replacing @proxy@ by ca-proxy.cern.ch    [05-09-2023 12:10:09 UTC]
(download) escaped http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/api/v1.0/geo/ca-proxy.cern.ch/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch to http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/api/v1.0/geo/ca-proxy.cern.ch/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch    [05-09-2023 12:10:09 UTC]
(download) Verify downloaded url http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch, proxy http://ca-proxy.cern.ch:3128 (curl error 7)    [05-09-2023 12:10:09 UTC]
(download) Trying again on same curl handle, same url: 1, error code 6    [05-09-2023 12:10:09 UTC]
(download) backing off for 302 ms    [05-09-2023 12:10:09 UTC]
(download) Verify downloaded url http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch, proxy http://ca-proxy.cern.ch:3128 (curl error 7)    [05-09-2023 12:10:09 UTC]
(download) Trying again on same curl handle, same url: 0, error code 6    [05-09-2023 12:10:09 UTC]
(download) switching proxy from http://ca-proxy.cern.ch:3128 to http://ca-proxy-atlas.cern.ch:3128 (failed proxy)    [05-09-2023 12:10:09 UTC]
(download) 1 proxies remain in group    [05-09-2023 12:10:09 UTC]
(download) replacing @proxy@ by ca-proxy-atlas.cern.ch    [05-09-2023 12:10:09 UTC]
(download) escaped http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/api/v1.0/geo/ca-proxy-atlas.cern.ch/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch to http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/api/v1.0/geo/ca-proxy-atlas.cern.ch/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch    [05-09-2023 12:10:09 UTC]
(download) Verify downloaded url http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch, proxy http://ca-proxy-atlas.cern.ch:3128 (curl error 7)    [05-09-2023 12:10:09 UTC]
(download) download failed (error 6 - proxy connection problem)    [05-09-2023 12:10:09 UTC]
(download) GeoAPI request http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch failed with error 6 [proxy connection problem]    [05-09-2023 12:10:09 UTC]
(download) requesting ordered server list from http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch    [05-09-2023 12:10:09 UTC]
(download) replacing @proxy@ by ca-proxy-atlas.cern.ch    [05-09-2023 12:10:09 UTC]
(download) escaped http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/api/v1.0/geo/ca-proxy-atlas.cern.ch/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch to http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/api/v1.0/geo/ca-proxy-atlas.cern.ch/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch    [05-09-2023 12:10:09 UTC]
(download) Verify downloaded url http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch, proxy http://ca-proxy-atlas.cern.ch:3128 (curl error 7)    [05-09-2023 12:10:09 UTC]
(download) Trying again on same curl handle, same url: 1, error code 6    [05-09-2023 12:10:09 UTC]
(download) backing off for 1300 ms    [05-09-2023 12:10:09 UTC]
(download) Verify downloaded url http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch, proxy http://ca-proxy-atlas.cern.ch:3128 (curl error 7)    [05-09-2023 12:10:10 UTC]
(download) Trying again on same curl handle, same url: 0, error code 6    [05-09-2023 12:10:10 UTC]
(download) switching proxy from http://ca-proxy-atlas.cern.ch:3128 to http://ca-proxy.cern.ch:3128 (failed proxy)    [05-09-2023 12:10:10 UTC]
(download) 1 proxies remain in group    [05-09-2023 12:10:10 UTC]
(download) replacing @proxy@ by ca-proxy.cern.ch    [05-09-2023 12:10:10 UTC]
(download) escaped http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/api/v1.0/geo/ca-proxy.cern.ch/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch to http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/api/v1.0/geo/ca-proxy.cern.ch/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch    [05-09-2023 12:10:10 UTC]
(download) Verify downloaded url http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch, proxy http://ca-proxy.cern.ch:3128 (curl error 7)    [05-09-2023 12:10:10 UTC]
(download) download failed (error 6 - proxy connection problem)    [05-09-2023 12:10:10 UTC]
(download) GeoAPI request http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch failed with error 6 [proxy connection problem]    [05-09-2023 12:10:10 UTC]
(download) failed to retrieve geographic order from stratum 1 servers    [05-09-2023 12:10:10 UTC]
(download) switching proxy from (none) to http://ca-proxy.cern.ch:3128 (cloned)    [05-09-2023 12:10:10 UTC]
(download) first fallback proxy group 1    [05-09-2023 12:10:10 UTC]
(download) full proxy list DIRECT    [05-09-2023 12:10:10 UTC]
(download) resolving 1 proxy addresses    [05-09-2023 12:10:10 UTC]
(dns) empty hostname    [05-09-2023 12:10:10 UTC]
(download) installed 1 proxies in 1 load-balance groups    [05-09-2023 12:10:10 UTC]
(download) switching proxy from http://ca-proxy.cern.ch:3128 to DIRECT (set proxies)    [05-09-2023 12:10:10 UTC]
(cvmfs) DNS roaming is disabled for this repository.    [05-09-2023 12:10:10 UTC]
(catalog) constructing client catalog manager    [05-09-2023 12:10:10 UTC]
(catalog) Initialize catalog    [05-09-2023 12:10:10 UTC]
(cache) unable to read local checksum    [05-09-2023 12:10:10 UTC]
(download) escaped http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/.cvmfspublished to http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/.cvmfspublished    [05-09-2023 12:10:10 UTC]
(download) Verify downloaded url /.cvmfspublished, proxy http://ca-proxy.cern.ch:3128 (curl error 7)    [05-09-2023 12:10:10 UTC]
(download) Trying again on same curl handle, same url: 1, error code 6    [05-09-2023 12:10:10 UTC]
(download) backing off for 1428 ms    [05-09-2023 12:10:10 UTC]
(download) Verify downloaded url /.cvmfspublished, proxy http://ca-proxy.cern.ch:3128 (curl error 7)    [05-09-2023 12:10:12 UTC]
(download) Trying again on same curl handle, same url: 0, error code 6    [05-09-2023 12:10:12 UTC]
(download) switching proxy from http://ca-proxy.cern.ch:3128 to http://ca-proxy-atlas.cern.ch:3128 (failed proxy)    [05-09-2023 12:10:12 UTC]
(download) 1 proxies remain in group    [05-09-2023 12:10:12 UTC]
(download) escaped http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/.cvmfspublished to http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/.cvmfspublished    [05-09-2023 12:10:12 UTC]
(download) Verify downloaded url /.cvmfspublished, proxy http://ca-proxy-atlas.cern.ch:3128 (curl error 7)    [05-09-2023 12:10:12 UTC]
(download) make it a host failure    [05-09-2023 12:10:12 UTC]
(download) Trying again on same curl handle, same url: 0, error code 5    [05-09-2023 12:10:12 UTC]
(download) switching host from http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch to http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch (all proxies failed, trying host fail-over)    [05-09-2023 12:10:12 UTC]
(download) escaped http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/.cvmfspublished to http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/.cvmfspublished    [05-09-2023 12:10:12 UTC]
(download) Verify downloaded url /.cvmfspublished, proxy http://ca-proxy-atlas.cern.ch:3128 (curl error 7)    [05-09-2023 12:10:12 UTC]
(download) Trying again on same curl handle, same url: 0, error code 6    [05-09-2023 12:10:12 UTC]
(download) switching proxy from http://ca-proxy-atlas.cern.ch:3128 to http://ca-proxy.cern.ch:3128 (failed proxy)    [05-09-2023 12:10:12 UTC]
(download) 1 proxies remain in group    [05-09-2023 12:10:12 UTC]
(download) escaped http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/.cvmfspublished to http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/.cvmfspublished    [05-09-2023 12:10:12 UTC]
(download) Verify downloaded url /.cvmfspublished, proxy http://ca-proxy.cern.ch:3128 (curl error 7)    [05-09-2023 12:10:12 UTC]
(download) switching proxy from http://ca-proxy.cern.ch:3128 to http://ca-proxy-atlas.cern.ch:3128 (reset proxies for host failover)    [05-09-2023 12:10:12 UTC]
(download) make it a host failure    [05-09-2023 12:10:12 UTC]
(download) Trying again on same curl handle, same url: 0, error code 5    [05-09-2023 12:10:12 UTC]
(download) switching host from http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch to http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch (all proxies failed, trying host fail-over)    [05-09-2023 12:10:12 UTC]
(download) escaped http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/.cvmfspublished to http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/.cvmfspublished    [05-09-2023 12:10:12 UTC]
(download) Verify downloaded url /.cvmfspublished, proxy http://ca-proxy-atlas.cern.ch:3128 (curl error 7)    [05-09-2023 12:10:12 UTC]
(download) Trying again on same curl handle, same url: 0, error code 6    [05-09-2023 12:10:12 UTC]
(download) switching proxy from http://ca-proxy-atlas.cern.ch:3128 to http://ca-proxy.cern.ch:3128 (failed proxy)    [05-09-2023 12:10:12 UTC]
(download) 1 proxies remain in group    [05-09-2023 12:10:12 UTC]
(download) escaped http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/.cvmfspublished to http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/.cvmfspublished    [05-09-2023 12:10:12 UTC]
(download) Verify downloaded url /.cvmfspublished, proxy http://ca-proxy.cern.ch:3128 (curl error 7)    [05-09-2023 12:10:12 UTC]
(download) switching proxy from http://ca-proxy.cern.ch:3128 to http://ca-proxy-atlas.cern.ch:3128 (reset proxies for host failover)    [05-09-2023 12:10:12 UTC]
(download) make it a host failure    [05-09-2023 12:10:12 UTC]
(download) Trying again on same curl handle, same url: 0, error code 5    [05-09-2023 12:10:12 UTC]
(download) switching host from http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch to http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch (all proxies failed, trying host fail-over)    [05-09-2023 12:10:12 UTC]
(download) escaped http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/.cvmfspublished to http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/.cvmfspublished    [05-09-2023 12:10:12 UTC]
(download) Verify downloaded url /.cvmfspublished, proxy http://ca-proxy-atlas.cern.ch:3128 (curl error 7)    [05-09-2023 12:10:12 UTC]
(download) Trying again on same curl handle, same url: 0, error code 6    [05-09-2023 12:10:12 UTC]
(download) switching proxy from http://ca-proxy-atlas.cern.ch:3128 to http://ca-proxy.cern.ch:3128 (failed proxy)    [05-09-2023 12:10:12 UTC]
(download) 1 proxies remain in group    [05-09-2023 12:10:12 UTC]
(download) escaped http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/.cvmfspublished to http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/.cvmfspublished    [05-09-2023 12:10:12 UTC]
(download) Verify downloaded url /.cvmfspublished, proxy http://ca-proxy.cern.ch:3128 (curl error 7)    [05-09-2023 12:10:12 UTC]
(download) download failed (error 6 - proxy connection problem)    [05-09-2023 12:10:12 UTC]
(cvmfs) failed to download repository manifest (6 - proxy connection problem)    [05-09-2023 12:10:12 UTC]
(cache) failed to fetch manifest (1 - failed to download)    [05-09-2023 12:10:12 UTC]
(cache) miss ./00/00000000000000000000000000000000000000 (-2)    [05-09-2023 12:10:12 UTC]
(cache) miss ./00/00000000000000000000000000000000000000 (-2)    [05-09-2023 12:10:12 UTC]
(cache) downloading file catalog at atlas.cern.ch:/    [05-09-2023 12:10:12 UTC]
(cache) start transaction on ./txn/fetchppQ39O has result 16    [05-09-2023 12:10:12 UTC]
(cache) miss: file catalog at atlas.cern.ch:/ /data/00/00000000000000000000000000000000000000C    [05-09-2023 12:10:12 UTC]
(download) escaped http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C to http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C    [05-09-2023 12:10:12 UTC]
(download) Verify downloaded url /data/00/00000000000000000000000000000000000000C, proxy http://ca-proxy.cern.ch:3128 (curl error 7)    [05-09-2023 12:10:12 UTC]
(download) Trying again on same curl handle, same url: 1, error code 6    [05-09-2023 12:10:12 UTC]
(download) backing off for 1328 ms    [05-09-2023 12:10:12 UTC]
(download) Verify downloaded url /data/00/00000000000000000000000000000000000000C, proxy http://ca-proxy.cern.ch:3128 (curl error 7)    [05-09-2023 12:10:13 UTC]
(download) Trying again on same curl handle, same url: 0, error code 6    [05-09-2023 12:10:13 UTC]
(download) switching proxy from http://ca-proxy.cern.ch:3128 to http://ca-proxy-atlas.cern.ch:3128 (failed proxy)    [05-09-2023 12:10:13 UTC]
(download) 1 proxies remain in group    [05-09-2023 12:10:13 UTC]
(download) escaped http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C to http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C    [05-09-2023 12:10:13 UTC]
(download) Verify downloaded url /data/00/00000000000000000000000000000000000000C, proxy http://ca-proxy-atlas.cern.ch:3128 (curl error 7)    [05-09-2023 12:10:13 UTC]
(download) make it a host failure    [05-09-2023 12:10:13 UTC]
(download) Trying again on same curl handle, same url: 0, error code 5    [05-09-2023 12:10:13 UTC]
(download) switching host from http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch to http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch (all proxies failed, trying host fail-over)    [05-09-2023 12:10:13 UTC]
(download) escaped http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C to http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C    [05-09-2023 12:10:13 UTC]
(download) Verify downloaded url /data/00/00000000000000000000000000000000000000C, proxy http://ca-proxy-atlas.cern.ch:3128 (curl error 7)    [05-09-2023 12:10:13 UTC]
(download) Trying again on same curl handle, same url: 0, error code 6    [05-09-2023 12:10:13 UTC]
(download) switching proxy from http://ca-proxy-atlas.cern.ch:3128 to http://ca-proxy.cern.ch:3128 (failed proxy)    [05-09-2023 12:10:13 UTC]
(download) 1 proxies remain in group    [05-09-2023 12:10:13 UTC]
(download) escaped http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C to http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C    [05-09-2023 12:10:13 UTC]
(download) Verify downloaded url /data/00/00000000000000000000000000000000000000C, proxy http://ca-proxy.cern.ch:3128 (curl error 7)    [05-09-2023 12:10:13 UTC]
(download) switching proxy from http://ca-proxy.cern.ch:3128 to http://ca-proxy-atlas.cern.ch:3128 (reset proxies for host failover)    [05-09-2023 12:10:13 UTC]
(download) make it a host failure    [05-09-2023 12:10:13 UTC]
(download) Trying again on same curl handle, same url: 0, error code 5    [05-09-2023 12:10:13 UTC]
(download) switching host from http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch to http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch (all proxies failed, trying host fail-over)    [05-09-2023 12:10:13 UTC]
(download) escaped http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C to http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C    [05-09-2023 12:10:13 UTC]
(download) Verify downloaded url /data/00/00000000000000000000000000000000000000C, proxy http://ca-proxy-atlas.cern.ch:3128 (curl error 7)    [05-09-2023 12:10:13 UTC]
(download) Trying again on same curl handle, same url: 0, error code 6    [05-09-2023 12:10:13 UTC]
(download) switching proxy from http://ca-proxy-atlas.cern.ch:3128 to http://ca-proxy.cern.ch:3128 (failed proxy)    [05-09-2023 12:10:13 UTC]
(download) 1 proxies remain in group    [05-09-2023 12:10:13 UTC]
(download) escaped http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C to http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C    [05-09-2023 12:10:13 UTC]
(download) Verify downloaded url /data/00/00000000000000000000000000000000000000C, proxy http://ca-proxy.cern.ch:3128 (curl error 7)    [05-09-2023 12:10:13 UTC]
(download) switching proxy from http://ca-proxy.cern.ch:3128 to http://ca-proxy-atlas.cern.ch:3128 (reset proxies for host failover)    [05-09-2023 12:10:13 UTC]
(download) make it a host failure    [05-09-2023 12:10:13 UTC]
(download) Trying again on same curl handle, same url: 0, error code 5    [05-09-2023 12:10:13 UTC]
(download) switching host from http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch to http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch (all proxies failed, trying host fail-over)    [05-09-2023 12:10:13 UTC]
(download) escaped http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C to http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C    [05-09-2023 12:10:13 UTC]
(download) Verify downloaded url /data/00/00000000000000000000000000000000000000C, proxy http://ca-proxy-atlas.cern.ch:3128 (curl error 7)    [05-09-2023 12:10:13 UTC]
(download) Trying again on same curl handle, same url: 0, error code 6    [05-09-2023 12:10:13 UTC]
(download) switching proxy from http://ca-proxy-atlas.cern.ch:3128 to http://ca-proxy.cern.ch:3128 (failed proxy)    [05-09-2023 12:10:13 UTC]
(download) 1 proxies remain in group    [05-09-2023 12:10:13 UTC]
(download) escaped http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C to http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C    [05-09-2023 12:10:13 UTC]
(download) Verify downloaded url /data/00/00000000000000000000000000000000000000C, proxy http://ca-proxy.cern.ch:3128 (curl error 7)    [05-09-2023 12:10:13 UTC]
(download) download failed (error 6 - proxy connection problem)    [05-09-2023 12:10:13 UTC]
(cache) failed to fetch file catalog at atlas.cern.ch:/ (hash: 0000000000000000000000000000000000000000, error 6 [proxy connection problem])    [05-09-2023 12:10:13 UTC]
(cache) abort ./txn/fetchppQ39O    [05-09-2023 12:10:13 UTC]
(catalog) failed to load catalog '' (3 - failed to load catalog)    [05-09-2023 12:10:13 UTC]
(catalog) failed to initialize root catalog    [05-09-2023 12:10:13 UTC]
Failed to initialize root file catalog (16 - file catalog failure)
(cache) unpinning / unloading all catalogs    [05-09-2023 12:10:13 UTC]

It seems that the proxy fails.

gman0 commented 1 year ago

Yup, can you try setting CVMFS_HTTP_PROXY=DIRECT and sending the logs of that?

DiamantisPa commented 1 year ago

I updated cvmfs-csi-default-local.yaml with the CVMFS_HTTP_PROXY=DIRECT.

[root@bril-master-1 example-cvmfs]# kubectl exec -it cvmfs-cvmfs-csi-nodeplugin-lqzdm -c nodeplugin -- /bin/sh
sh-4.2# cat /etc/cvmfs/default.local 
CVMFS_USE_GEOAPI=yes
CVMFS_HTTP_PROXY=DIRECT

# It is advised to change these configs in the cache section of the helm values
# and leave them unchanged here, so they auto-generate.
CVMFS_QUOTA_LIMIT=1000
CVMFS_CACHE_BASE=/cvmfs-localcache

Mounting again fails, i think for the same reason.

sh-4.2# mount -t cvmfs atlas.cern.ch /mnt -v -o debug
Debug: using library /usr/lib64/libcvmfs_fuse_stub.so
CernVM-FS: running with credentials 999:997
CernVM-FS: running in debug mode
CernVM-FS: loading Fuse module... (cvmfs) Parsing config file /etc/cvmfs/default.conf    [05-09-2023 12:28:28 UTC]
(cvmfs) execve'd /bin/sh (PID: 274704)    [05-09-2023 12:28:29 UTC]
(cvmfs) Parsing config file /etc/cvmfs/default.d/50-cern.conf    [05-09-2023 12:28:29 UTC]
(cvmfs) execve'd /bin/sh (PID: 274706)    [05-09-2023 12:28:29 UTC]
(cvmfs) Parsing config file /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/default.conf    [05-09-2023 12:28:29 UTC]
(cvmfs) configuration repository directory does not exist: /cvmfs/cvmfs-config.cern.ch/etc/cvmfs    [05-09-2023 12:28:29 UTC]
(cvmfs) Parsing config file /etc/cvmfs/default.local    [05-09-2023 12:28:29 UTC]
(cvmfs) execve'd /bin/sh (PID: 274709)    [05-09-2023 12:28:29 UTC]
(cvmfs) Parsing config file /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d/cern.ch.conf    [05-09-2023 12:28:29 UTC]
(cvmfs) configuration repository directory does not exist: /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d    [05-09-2023 12:28:29 UTC]
(cvmfs) Parsing config file /etc/cvmfs/domain.d/cern.ch.conf    [05-09-2023 12:28:29 UTC]
(cvmfs) execve'd /bin/sh (PID: 274712)    [05-09-2023 12:28:29 UTC]
(cvmfs) Parsing config file /etc/cvmfs/domain.d/cern.ch.local    [05-09-2023 12:28:29 UTC]
(cvmfs) Parsing config file /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/config.d/atlas.cern.ch.conf    [05-09-2023 12:28:29 UTC]
(cvmfs) configuration repository directory does not exist: /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/config.d    [05-09-2023 12:28:29 UTC]
(cvmfs) Parsing config file /etc/cvmfs/config.d/atlas.cern.ch.conf    [05-09-2023 12:28:29 UTC]
(cvmfs) Parsing config file /etc/cvmfs/config.d/atlas.cern.ch.local    [05-09-2023 12:28:29 UTC]
(cvmfs) execve'd /bin/sh (PID: 274715)    [05-09-2023 12:28:29 UTC]
(cvmfs) Options:
CVMFS_BACKOFF_INIT=2    # from /etc/cvmfs/default.conf
CVMFS_BACKOFF_MAX=10    # from /etc/cvmfs/default.conf
CVMFS_BASE_ENV=1    # from /etc/cvmfs/default.conf
CVMFS_CACHE_BASE=/cvmfs-localcache    # from /etc/cvmfs/default.local
CVMFS_CHECK_PERMISSIONS=yes    # from /etc/cvmfs/default.conf
CVMFS_CLAIM_OWNERSHIP=yes    # from /etc/cvmfs/default.conf
CVMFS_CLIENT_PROFILE=    # from /etc/cvmfs/default.conf
CVMFS_CONFIG_REPOSITORY=cvmfs-config.cern.ch    # from /etc/cvmfs/default.d/50-cern.conf
CVMFS_DEFAULT_DOMAIN=cern.ch    # from /etc/cvmfs/default.d/50-cern.conf
CVMFS_HOST_RESET_AFTER=1800    # from /etc/cvmfs/default.conf
CVMFS_HTTP_PROXY='http://ca-proxy-atlas.cern.ch:3128;http://ca-proxy.cern.ch:3128'    # from /etc/cvmfs/config.d/atlas.cern.ch.local
CVMFS_KEYS_DIR=/etc/cvmfs/keys/cern.ch    # from /etc/cvmfs/domain.d/cern.ch.conf
CVMFS_LOW_SPEED_LIMIT=1024    # from /etc/cvmfs/default.conf
CVMFS_MAX_RETRIES=1    # from /etc/cvmfs/default.conf
CVMFS_MOUNT_DIR=/cvmfs    # from /etc/cvmfs/default.conf
CVMFS_NFILES=131072    # from /etc/cvmfs/default.conf
CVMFS_PROXY_RESET_AFTER=300    # from /etc/cvmfs/default.conf
CVMFS_QUOTA_LIMIT=1000    # from /etc/cvmfs/default.local
CVMFS_RELOAD_SOCKETS=/var/run/cvmfs    # from /etc/cvmfs/default.conf
CVMFS_SEND_INFO_HEADER=no    # from /etc/cvmfs/default.conf
CVMFS_SERVER_URL='http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch;http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch;http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch;http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch'    # from /etc/cvmfs/domain.d/cern.ch.conf
CVMFS_SHARED_CACHE=yes    # from /etc/cvmfs/default.conf
CVMFS_STRICT_MOUNT=no    # from /etc/cvmfs/default.conf
CVMFS_TIMEOUT=5    # from /etc/cvmfs/default.conf
CVMFS_TIMEOUT_DIRECT=10    # from /etc/cvmfs/default.conf
CVMFS_USER=cvmfs    # from /etc/cvmfs/default.conf
CVMFS_USE_GEOAPI=yes    # from /etc/cvmfs/domain.d/cern.ch.conf
    [05-09-2023 12:28:29 UTC]
(cvmfs) setting up cache manager instance default    [05-09-2023 12:28:29 UTC]
(quota) trying to connect to existing pipe    [05-09-2023 12:28:29 UTC]
(cvmfs) execve'd /usr/bin/cvmfs2 (PID: 274717)    [05-09-2023 12:28:29 UTC]
(quota) starting quota manager    [05-09-2023 12:28:29 UTC]
(quota) connected to a new cache manager    [05-09-2023 12:28:29 UTC]
(quota) received limit 1048576000, threshold 524288000    [05-09-2023 12:28:29 UTC]
(cvmfs) CernVM-FS: quota initialized, current size 0MB    [05-09-2023 12:28:29 UTC]
(cvmfs) CernVM-FS: using public key(s) /etc/cvmfs/keys/cern.ch/cern-it1.cern.ch.pub:/etc/cvmfs/keys/cern.ch/cern-it4.cern.ch.pub:/etc/cvmfs/keys/cern.ch/cern-it5.cern.ch.pub    [05-09-2023 12:28:29 UTC]
(download) first fallback proxy group 2    [05-09-2023 12:28:29 UTC]
(download) full proxy list http://ca-proxy-atlas.cern.ch:3128;http://ca-proxy.cern.ch:3128    [05-09-2023 12:28:29 UTC]
(download) resolving 2 proxy addresses    [05-09-2023 12:28:29 UTC]
(dns) failed to resolve ca-proxy-atlas.cern.ch - 2 (DNS query timeout), ttl 60    [05-09-2023 12:28:38 UTC]
(dns) failed to resolve ca-proxy.cern.ch - 2 (DNS query timeout), ttl 60    [05-09-2023 12:28:38 UTC]
(download) failed to resolve IP addresses for ca-proxy-atlas.cern.ch (2 - DNS query timeout)    [05-09-2023 12:28:38 UTC]
(download) failed to resolve IP addresses for ca-proxy.cern.ch (2 - DNS query timeout)    [05-09-2023 12:28:38 UTC]
(download) installed 2 proxies in 2 load-balance groups    [05-09-2023 12:28:38 UTC]
(download) switching proxy from (none) to http://ca-proxy-atlas.cern.ch:3128 (set proxies)    [05-09-2023 12:28:38 UTC]
(download) requesting ordered server list from http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch    [05-09-2023 12:28:38 UTC]
(download) replacing @proxy@ by ca-proxy-atlas.cern.ch    [05-09-2023 12:28:38 UTC]
(download) escaped http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/api/v1.0/geo/ca-proxy-atlas.cern.ch/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch to http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/api/v1.0/geo/ca-proxy-atlas.cern.ch/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch    [05-09-2023 12:28:38 UTC]
(download) Verify downloaded url http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch, proxy http://ca-proxy-atlas.cern.ch:3128 (curl error 7)    [05-09-2023 12:28:38 UTC]
(download) Trying again on same curl handle, same url: 1, error code 6    [05-09-2023 12:28:38 UTC]
(download) backing off for 444 ms    [05-09-2023 12:28:38 UTC]
(download) Verify downloaded url http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch, proxy http://ca-proxy-atlas.cern.ch:3128 (curl error 7)    [05-09-2023 12:28:39 UTC]
(download) Trying again on same curl handle, same url: 0, error code 6    [05-09-2023 12:28:39 UTC]
(download) switching proxy from http://ca-proxy-atlas.cern.ch:3128 to http://ca-proxy.cern.ch:3128 (failed proxy)    [05-09-2023 12:28:39 UTC]
(download) 1 proxies remain in group    [05-09-2023 12:28:39 UTC]
(download) replacing @proxy@ by ca-proxy.cern.ch    [05-09-2023 12:28:39 UTC]
(download) escaped http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/api/v1.0/geo/ca-proxy.cern.ch/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch to http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/api/v1.0/geo/ca-proxy.cern.ch/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch    [05-09-2023 12:28:39 UTC]
(download) Verify downloaded url http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch, proxy http://ca-proxy.cern.ch:3128 (curl error 7)    [05-09-2023 12:28:39 UTC]
(download) download failed (error 6 - proxy connection problem)    [05-09-2023 12:28:39 UTC]
(download) GeoAPI request http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch failed with error 6 [proxy connection problem]    [05-09-2023 12:28:39 UTC]
(download) requesting ordered server list from http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch    [05-09-2023 12:28:39 UTC]
(download) replacing @proxy@ by ca-proxy.cern.ch    [05-09-2023 12:28:39 UTC]
(download) escaped http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/api/v1.0/geo/ca-proxy.cern.ch/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch to http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/api/v1.0/geo/ca-proxy.cern.ch/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch    [05-09-2023 12:28:39 UTC]
(download) Verify downloaded url http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch, proxy http://ca-proxy.cern.ch:3128 (curl error 7)    [05-09-2023 12:28:39 UTC]
(download) Trying again on same curl handle, same url: 1, error code 6    [05-09-2023 12:28:39 UTC]
(download) backing off for 1259 ms    [05-09-2023 12:28:39 UTC]
(download) Verify downloaded url http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch, proxy http://ca-proxy.cern.ch:3128 (curl error 7)    [05-09-2023 12:28:40 UTC]
(download) Trying again on same curl handle, same url: 0, error code 6    [05-09-2023 12:28:40 UTC]
(download) switching proxy from http://ca-proxy.cern.ch:3128 to http://ca-proxy-atlas.cern.ch:3128 (failed proxy)    [05-09-2023 12:28:40 UTC]
(download) 1 proxies remain in group    [05-09-2023 12:28:40 UTC]
(download) replacing @proxy@ by ca-proxy-atlas.cern.ch    [05-09-2023 12:28:40 UTC]
(download) escaped http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/api/v1.0/geo/ca-proxy-atlas.cern.ch/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch to http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/api/v1.0/geo/ca-proxy-atlas.cern.ch/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch    [05-09-2023 12:28:40 UTC]
(download) Verify downloaded url http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch, proxy http://ca-proxy-atlas.cern.ch:3128 (curl error 7)    [05-09-2023 12:28:40 UTC]
(download) download failed (error 6 - proxy connection problem)    [05-09-2023 12:28:40 UTC]
(download) GeoAPI request http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch failed with error 6 [proxy connection problem]    [05-09-2023 12:28:40 UTC]
(download) requesting ordered server list from http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch    [05-09-2023 12:28:40 UTC]
(download) replacing @proxy@ by ca-proxy-atlas.cern.ch    [05-09-2023 12:28:40 UTC]
(download) escaped http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/api/v1.0/geo/ca-proxy-atlas.cern.ch/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch to http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/api/v1.0/geo/ca-proxy-atlas.cern.ch/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch    [05-09-2023 12:28:40 UTC]
(download) Verify downloaded url http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch, proxy http://ca-proxy-atlas.cern.ch:3128 (curl error 7)    [05-09-2023 12:28:40 UTC]
(download) Trying again on same curl handle, same url: 1, error code 6    [05-09-2023 12:28:40 UTC]
(download) backing off for 1547 ms    [05-09-2023 12:28:40 UTC]
(download) Verify downloaded url http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch, proxy http://ca-proxy-atlas.cern.ch:3128 (curl error 7)    [05-09-2023 12:28:41 UTC]
(download) Trying again on same curl handle, same url: 0, error code 6    [05-09-2023 12:28:41 UTC]
(download) switching proxy from http://ca-proxy-atlas.cern.ch:3128 to http://ca-proxy.cern.ch:3128 (failed proxy)    [05-09-2023 12:28:41 UTC]
(download) 1 proxies remain in group    [05-09-2023 12:28:41 UTC]
(download) replacing @proxy@ by ca-proxy.cern.ch    [05-09-2023 12:28:41 UTC]
(download) escaped http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/api/v1.0/geo/ca-proxy.cern.ch/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch to http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/api/v1.0/geo/ca-proxy.cern.ch/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch    [05-09-2023 12:28:41 UTC]
(download) Verify downloaded url http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch, proxy http://ca-proxy.cern.ch:3128 (curl error 7)    [05-09-2023 12:28:41 UTC]
(download) download failed (error 6 - proxy connection problem)    [05-09-2023 12:28:41 UTC]
(download) GeoAPI request http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch failed with error 6 [proxy connection problem]    [05-09-2023 12:28:41 UTC]
(download) failed to retrieve geographic order from stratum 1 servers    [05-09-2023 12:28:41 UTC]
(download) switching proxy from (none) to http://ca-proxy.cern.ch:3128 (cloned)    [05-09-2023 12:28:41 UTC]
(download) first fallback proxy group 1    [05-09-2023 12:28:41 UTC]
(download) full proxy list DIRECT    [05-09-2023 12:28:41 UTC]
(download) resolving 1 proxy addresses    [05-09-2023 12:28:41 UTC]
(dns) empty hostname    [05-09-2023 12:28:41 UTC]
(download) installed 1 proxies in 1 load-balance groups    [05-09-2023 12:28:41 UTC]
(download) switching proxy from http://ca-proxy.cern.ch:3128 to DIRECT (set proxies)    [05-09-2023 12:28:41 UTC]
(cvmfs) DNS roaming is disabled for this repository.    [05-09-2023 12:28:41 UTC]
(catalog) constructing client catalog manager    [05-09-2023 12:28:41 UTC]
(catalog) Initialize catalog    [05-09-2023 12:28:41 UTC]
(cache) unable to read local checksum    [05-09-2023 12:28:41 UTC]
(download) escaped http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/.cvmfspublished to http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/.cvmfspublished    [05-09-2023 12:28:41 UTC]
(download) Verify downloaded url /.cvmfspublished, proxy http://ca-proxy.cern.ch:3128 (curl error 7)    [05-09-2023 12:28:41 UTC]
(download) Trying again on same curl handle, same url: 1, error code 6    [05-09-2023 12:28:41 UTC]
(download) backing off for 503 ms    [05-09-2023 12:28:41 UTC]
(download) Verify downloaded url /.cvmfspublished, proxy http://ca-proxy.cern.ch:3128 (curl error 7)    [05-09-2023 12:28:42 UTC]
(download) Trying again on same curl handle, same url: 0, error code 6    [05-09-2023 12:28:42 UTC]
(download) switching proxy from http://ca-proxy.cern.ch:3128 to http://ca-proxy-atlas.cern.ch:3128 (failed proxy)    [05-09-2023 12:28:42 UTC]
(download) 1 proxies remain in group    [05-09-2023 12:28:42 UTC]
(download) escaped http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/.cvmfspublished to http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/.cvmfspublished    [05-09-2023 12:28:42 UTC]
(download) Verify downloaded url /.cvmfspublished, proxy http://ca-proxy-atlas.cern.ch:3128 (curl error 7)    [05-09-2023 12:28:42 UTC]
(download) make it a host failure    [05-09-2023 12:28:42 UTC]
(download) Trying again on same curl handle, same url: 0, error code 5    [05-09-2023 12:28:42 UTC]
(download) switching host from http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch to http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch (all proxies failed, trying host fail-over)    [05-09-2023 12:28:42 UTC]
(download) escaped http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/.cvmfspublished to http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/.cvmfspublished    [05-09-2023 12:28:42 UTC]
(download) Verify downloaded url /.cvmfspublished, proxy http://ca-proxy-atlas.cern.ch:3128 (curl error 7)    [05-09-2023 12:28:42 UTC]
(download) Trying again on same curl handle, same url: 0, error code 6    [05-09-2023 12:28:42 UTC]
(download) switching proxy from http://ca-proxy-atlas.cern.ch:3128 to http://ca-proxy.cern.ch:3128 (failed proxy)    [05-09-2023 12:28:42 UTC]
(download) 1 proxies remain in group    [05-09-2023 12:28:42 UTC]
(download) escaped http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/.cvmfspublished to http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/.cvmfspublished    [05-09-2023 12:28:42 UTC]
(download) Verify downloaded url /.cvmfspublished, proxy http://ca-proxy.cern.ch:3128 (curl error 7)    [05-09-2023 12:28:42 UTC]
(download) switching proxy from http://ca-proxy.cern.ch:3128 to http://ca-proxy-atlas.cern.ch:3128 (reset proxies for host failover)    [05-09-2023 12:28:42 UTC]
(download) make it a host failure    [05-09-2023 12:28:42 UTC]
(download) Trying again on same curl handle, same url: 0, error code 5    [05-09-2023 12:28:42 UTC]
(download) switching host from http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch to http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch (all proxies failed, trying host fail-over)    [05-09-2023 12:28:42 UTC]
(download) escaped http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/.cvmfspublished to http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/.cvmfspublished    [05-09-2023 12:28:42 UTC]
(download) Verify downloaded url /.cvmfspublished, proxy http://ca-proxy-atlas.cern.ch:3128 (curl error 7)    [05-09-2023 12:28:42 UTC]
(download) Trying again on same curl handle, same url: 0, error code 6    [05-09-2023 12:28:42 UTC]
(download) switching proxy from http://ca-proxy-atlas.cern.ch:3128 to http://ca-proxy.cern.ch:3128 (failed proxy)    [05-09-2023 12:28:42 UTC]
(download) 1 proxies remain in group    [05-09-2023 12:28:42 UTC]
(download) escaped http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/.cvmfspublished to http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/.cvmfspublished    [05-09-2023 12:28:42 UTC]
(download) Verify downloaded url /.cvmfspublished, proxy http://ca-proxy.cern.ch:3128 (curl error 7)    [05-09-2023 12:28:42 UTC]
(download) switching proxy from http://ca-proxy.cern.ch:3128 to http://ca-proxy-atlas.cern.ch:3128 (reset proxies for host failover)    [05-09-2023 12:28:42 UTC]
(download) make it a host failure    [05-09-2023 12:28:42 UTC]
(download) Trying again on same curl handle, same url: 0, error code 5    [05-09-2023 12:28:42 UTC]
(download) switching host from http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch to http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch (all proxies failed, trying host fail-over)    [05-09-2023 12:28:42 UTC]
(download) escaped http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/.cvmfspublished to http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/.cvmfspublished    [05-09-2023 12:28:42 UTC]
(download) Verify downloaded url /.cvmfspublished, proxy http://ca-proxy-atlas.cern.ch:3128 (curl error 7)    [05-09-2023 12:28:42 UTC]
(download) Trying again on same curl handle, same url: 0, error code 6    [05-09-2023 12:28:42 UTC]
(download) switching proxy from http://ca-proxy-atlas.cern.ch:3128 to http://ca-proxy.cern.ch:3128 (failed proxy)    [05-09-2023 12:28:42 UTC]
(download) 1 proxies remain in group    [05-09-2023 12:28:42 UTC]
(download) escaped http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/.cvmfspublished to http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/.cvmfspublished    [05-09-2023 12:28:42 UTC]
(download) Verify downloaded url /.cvmfspublished, proxy http://ca-proxy.cern.ch:3128 (curl error 7)    [05-09-2023 12:28:42 UTC]
(download) download failed (error 6 - proxy connection problem)    [05-09-2023 12:28:42 UTC]
(cvmfs) failed to download repository manifest (6 - proxy connection problem)    [05-09-2023 12:28:42 UTC]
(cache) failed to fetch manifest (1 - failed to download)    [05-09-2023 12:28:42 UTC]
(cache) miss ./00/00000000000000000000000000000000000000 (-2)    [05-09-2023 12:28:42 UTC]
(cache) miss ./00/00000000000000000000000000000000000000 (-2)    [05-09-2023 12:28:42 UTC]
(cache) downloading file catalog at atlas.cern.ch:/    [05-09-2023 12:28:42 UTC]
(cache) start transaction on ./txn/fetch47LSDT has result 16    [05-09-2023 12:28:42 UTC]
(cache) miss: file catalog at atlas.cern.ch:/ /data/00/00000000000000000000000000000000000000C    [05-09-2023 12:28:42 UTC]
(download) escaped http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C to http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C    [05-09-2023 12:28:42 UTC]
(download) Verify downloaded url /data/00/00000000000000000000000000000000000000C, proxy http://ca-proxy.cern.ch:3128 (curl error 7)    [05-09-2023 12:28:42 UTC]
(download) Trying again on same curl handle, same url: 1, error code 6    [05-09-2023 12:28:42 UTC]
(download) backing off for 317 ms    [05-09-2023 12:28:42 UTC]
(download) Verify downloaded url /data/00/00000000000000000000000000000000000000C, proxy http://ca-proxy.cern.ch:3128 (curl error 7)    [05-09-2023 12:28:42 UTC]
(download) Trying again on same curl handle, same url: 0, error code 6    [05-09-2023 12:28:42 UTC]
(download) switching proxy from http://ca-proxy.cern.ch:3128 to http://ca-proxy-atlas.cern.ch:3128 (failed proxy)    [05-09-2023 12:28:42 UTC]
(download) 1 proxies remain in group    [05-09-2023 12:28:42 UTC]
(download) escaped http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C to http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C    [05-09-2023 12:28:42 UTC]
(download) Verify downloaded url /data/00/00000000000000000000000000000000000000C, proxy http://ca-proxy-atlas.cern.ch:3128 (curl error 7)    [05-09-2023 12:28:42 UTC]
(download) make it a host failure    [05-09-2023 12:28:42 UTC]
(download) Trying again on same curl handle, same url: 0, error code 5    [05-09-2023 12:28:42 UTC]
(download) switching host from http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch to http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch (all proxies failed, trying host fail-over)    [05-09-2023 12:28:42 UTC]
(download) escaped http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C to http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C    [05-09-2023 12:28:42 UTC]
(download) Verify downloaded url /data/00/00000000000000000000000000000000000000C, proxy http://ca-proxy-atlas.cern.ch:3128 (curl error 7)    [05-09-2023 12:28:42 UTC]
(download) Trying again on same curl handle, same url: 0, error code 6    [05-09-2023 12:28:42 UTC]
(download) switching proxy from http://ca-proxy-atlas.cern.ch:3128 to http://ca-proxy.cern.ch:3128 (failed proxy)    [05-09-2023 12:28:42 UTC]
(download) 1 proxies remain in group    [05-09-2023 12:28:42 UTC]
(download) escaped http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C to http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C    [05-09-2023 12:28:42 UTC]
(download) Verify downloaded url /data/00/00000000000000000000000000000000000000C, proxy http://ca-proxy.cern.ch:3128 (curl error 7)    [05-09-2023 12:28:42 UTC]
(download) switching proxy from http://ca-proxy.cern.ch:3128 to http://ca-proxy-atlas.cern.ch:3128 (reset proxies for host failover)    [05-09-2023 12:28:42 UTC]
(download) make it a host failure    [05-09-2023 12:28:42 UTC]
(download) Trying again on same curl handle, same url: 0, error code 5    [05-09-2023 12:28:42 UTC]
(download) switching host from http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch to http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch (all proxies failed, trying host fail-over)    [05-09-2023 12:28:42 UTC]
(download) escaped http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C to http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C    [05-09-2023 12:28:42 UTC]
(download) Verify downloaded url /data/00/00000000000000000000000000000000000000C, proxy http://ca-proxy-atlas.cern.ch:3128 (curl error 7)    [05-09-2023 12:28:42 UTC]
(download) Trying again on same curl handle, same url: 0, error code 6    [05-09-2023 12:28:42 UTC]
(download) switching proxy from http://ca-proxy-atlas.cern.ch:3128 to http://ca-proxy.cern.ch:3128 (failed proxy)    [05-09-2023 12:28:42 UTC]
(download) 1 proxies remain in group    [05-09-2023 12:28:42 UTC]
(download) escaped http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C to http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C    [05-09-2023 12:28:42 UTC]
(download) Verify downloaded url /data/00/00000000000000000000000000000000000000C, proxy http://ca-proxy.cern.ch:3128 (curl error 7)    [05-09-2023 12:28:42 UTC]
(download) switching proxy from http://ca-proxy.cern.ch:3128 to http://ca-proxy-atlas.cern.ch:3128 (reset proxies for host failover)    [05-09-2023 12:28:42 UTC]
(download) make it a host failure    [05-09-2023 12:28:42 UTC]
(download) Trying again on same curl handle, same url: 0, error code 5    [05-09-2023 12:28:42 UTC]
(download) switching host from http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch to http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch (all proxies failed, trying host fail-over)    [05-09-2023 12:28:42 UTC]
(download) escaped http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C to http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C    [05-09-2023 12:28:42 UTC]
(download) Verify downloaded url /data/00/00000000000000000000000000000000000000C, proxy http://ca-proxy-atlas.cern.ch:3128 (curl error 7)    [05-09-2023 12:28:42 UTC]
(download) Trying again on same curl handle, same url: 0, error code 6    [05-09-2023 12:28:42 UTC]
(download) switching proxy from http://ca-proxy-atlas.cern.ch:3128 to http://ca-proxy.cern.ch:3128 (failed proxy)    [05-09-2023 12:28:42 UTC]
(download) 1 proxies remain in group    [05-09-2023 12:28:42 UTC]
(download) escaped http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C to http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C    [05-09-2023 12:28:42 UTC]
(download) Verify downloaded url /data/00/00000000000000000000000000000000000000C, proxy http://ca-proxy.cern.ch:3128 (curl error 7)    [05-09-2023 12:28:42 UTC]
(download) download failed (error 6 - proxy connection problem)    [05-09-2023 12:28:42 UTC]
(cache) failed to fetch file catalog at atlas.cern.ch:/ (hash: 0000000000000000000000000000000000000000, error 6 [proxy connection problem])    [05-09-2023 12:28:42 UTC]
(cache) abort ./txn/fetch47LSDT    [05-09-2023 12:28:42 UTC]
(catalog) failed to load catalog '' (3 - failed to load catalog)    [05-09-2023 12:28:42 UTC]
(catalog) failed to initialize root catalog    [05-09-2023 12:28:42 UTC]
Failed to initialize root file catalog (16 - file catalog failure)
(cache) unpinning / unloading all catalogs    [05-09-2023 12:28:42 UTC]
gman0 commented 1 year ago 
CVMFS_HTTP_PROXY='http://ca-proxy-atlas.cern.ch:3128;http://ca-proxy.cern.ch:3128'    # from /etc/cvmfs/config.d/atlas.cern.ch.local

It still uses the previously set proxy. It takes some time until the updated ConfigMap contents are propagated to the respective Pods.

You can either wait a bit and try again until this line indeed says CVMFS_HTTP_PROXY=DIRECT, or restart the nodeplugin Pods.

Also make sure you're actually setting the value. Even though you set CVMFS_HTTP_PROXY in /etc/cvmfs/default.local, it gets overridden by a setting in /etc/cvmfs/config.d/atlas.cern.ch.local (see the debug output).

DiamantisPa commented 1 year ago

Right, i fixed that. The proxy problem seems to be fixed now, but now seems to fail because the download takes too much time. Should I increase the CVMFS_LOW_SPEED_LIMIT=1024 maybe?

sh-4.2# mount -t cvmfs atlas.cern.ch /mnt -v -o debug
Debug: using library /usr/lib64/libcvmfs_fuse_stub.so
CernVM-FS: running with credentials 999:997
CernVM-FS: running in debug mode
CernVM-FS: loading Fuse module... (cvmfs) Parsing config file /etc/cvmfs/default.conf    [05-09-2023 13:49:18 UTC]
(cvmfs) execve'd /bin/sh (PID: 287428)    [05-09-2023 13:49:19 UTC]
(cvmfs) Parsing config file /etc/cvmfs/default.d/50-cern.conf    [05-09-2023 13:49:19 UTC]
(cvmfs) execve'd /bin/sh (PID: 287430)    [05-09-2023 13:49:19 UTC]
(cvmfs) Parsing config file /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/default.conf    [05-09-2023 13:49:19 UTC]
(cvmfs) configuration repository directory does not exist: /cvmfs/cvmfs-config.cern.ch/etc/cvmfs    [05-09-2023 13:49:19 UTC]
(cvmfs) Parsing config file /etc/cvmfs/default.local    [05-09-2023 13:49:19 UTC]
(cvmfs) execve'd /bin/sh (PID: 287433)    [05-09-2023 13:49:19 UTC]
(cvmfs) Parsing config file /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d/cern.ch.conf    [05-09-2023 13:49:19 UTC]
(cvmfs) configuration repository directory does not exist: /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d    [05-09-2023 13:49:19 UTC]
(cvmfs) Parsing config file /etc/cvmfs/domain.d/cern.ch.conf    [05-09-2023 13:49:19 UTC]
(cvmfs) execve'd /bin/sh (PID: 287436)    [05-09-2023 13:49:19 UTC]
(cvmfs) Parsing config file /etc/cvmfs/domain.d/cern.ch.local    [05-09-2023 13:49:19 UTC]
(cvmfs) Parsing config file /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/config.d/atlas.cern.ch.conf    [05-09-2023 13:49:19 UTC]
(cvmfs) configuration repository directory does not exist: /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/config.d    [05-09-2023 13:49:19 UTC]
(cvmfs) Parsing config file /etc/cvmfs/config.d/atlas.cern.ch.conf    [05-09-2023 13:49:19 UTC]
(cvmfs) Parsing config file /etc/cvmfs/config.d/atlas.cern.ch.local    [05-09-2023 13:49:19 UTC]
(cvmfs) Options:
CVMFS_BACKOFF_INIT=2    # from /etc/cvmfs/default.conf
CVMFS_BACKOFF_MAX=10    # from /etc/cvmfs/default.conf
CVMFS_BASE_ENV=1    # from /etc/cvmfs/default.conf
CVMFS_CACHE_BASE=/cvmfs-localcache    # from /etc/cvmfs/default.local
CVMFS_CHECK_PERMISSIONS=yes    # from /etc/cvmfs/default.conf
CVMFS_CLAIM_OWNERSHIP=yes    # from /etc/cvmfs/default.conf
CVMFS_CLIENT_PROFILE=    # from /etc/cvmfs/default.conf
CVMFS_CONFIG_REPOSITORY=cvmfs-config.cern.ch    # from /etc/cvmfs/default.d/50-cern.conf
CVMFS_DEFAULT_DOMAIN=cern.ch    # from /etc/cvmfs/default.d/50-cern.conf
CVMFS_HOST_RESET_AFTER=1800    # from /etc/cvmfs/default.conf
CVMFS_HTTP_PROXY=DIRECT    # from /etc/cvmfs/default.local
CVMFS_KEYS_DIR=/etc/cvmfs/keys/cern.ch    # from /etc/cvmfs/domain.d/cern.ch.conf
CVMFS_LOW_SPEED_LIMIT=1024    # from /etc/cvmfs/default.conf
CVMFS_MAX_RETRIES=1    # from /etc/cvmfs/default.conf
CVMFS_MOUNT_DIR=/cvmfs    # from /etc/cvmfs/default.conf
CVMFS_NFILES=131072    # from /etc/cvmfs/default.conf
CVMFS_PROXY_RESET_AFTER=300    # from /etc/cvmfs/default.conf
CVMFS_QUOTA_LIMIT=1000    # from /etc/cvmfs/default.local
CVMFS_RELOAD_SOCKETS=/var/run/cvmfs    # from /etc/cvmfs/default.conf
CVMFS_SEND_INFO_HEADER=no    # from /etc/cvmfs/default.conf
CVMFS_SERVER_URL='http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch;http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch;http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch;http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch'    # from /etc/cvmfs/domain.d/cern.ch.conf
CVMFS_SHARED_CACHE=yes    # from /etc/cvmfs/default.conf
CVMFS_STRICT_MOUNT=no    # from /etc/cvmfs/default.conf
CVMFS_TIMEOUT=5    # from /etc/cvmfs/default.conf
CVMFS_TIMEOUT_DIRECT=10    # from /etc/cvmfs/default.conf
CVMFS_USER=cvmfs    # from /etc/cvmfs/default.conf
CVMFS_USE_GEOAPI=yes    # from /etc/cvmfs/domain.d/cern.ch.conf
    [05-09-2023 13:49:19 UTC]
(cvmfs) setting up cache manager instance default    [05-09-2023 13:49:19 UTC]
(quota) trying to connect to existing pipe    [05-09-2023 13:49:19 UTC]
(cvmfs) execve'd /usr/bin/cvmfs2 (PID: 287439)    [05-09-2023 13:49:19 UTC]
(quota) starting quota manager    [05-09-2023 13:49:19 UTC]
(quota) connected to a new cache manager    [05-09-2023 13:49:19 UTC]
(quota) received limit 1048576000, threshold 524288000    [05-09-2023 13:49:19 UTC]
(cvmfs) CernVM-FS: quota initialized, current size 0MB    [05-09-2023 13:49:19 UTC]
(cvmfs) CernVM-FS: using public key(s) /etc/cvmfs/keys/cern.ch/cern-it1.cern.ch.pub:/etc/cvmfs/keys/cern.ch/cern-it4.cern.ch.pub:/etc/cvmfs/keys/cern.ch/cern-it5.cern.ch.pub    [05-09-2023 13:49:19 UTC]
(download) first fallback proxy group 1    [05-09-2023 13:49:19 UTC]
(download) full proxy list DIRECT    [05-09-2023 13:49:19 UTC]
(download) resolving 1 proxy addresses    [05-09-2023 13:49:19 UTC]
(dns) empty hostname    [05-09-2023 13:49:19 UTC]
(download) installed 1 proxies in 1 load-balance groups    [05-09-2023 13:49:19 UTC]
(download) switching proxy from (none) to DIRECT (set proxies)    [05-09-2023 13:49:19 UTC]
(download) requesting ordered server list from http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch    [05-09-2023 13:49:19 UTC]
(download) replacing @proxy@ by 19af6d99-c77e-7344-2abd-317255c39b72    [05-09-2023 13:49:19 UTC]
(download) escaped http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/api/v1.0/geo/19af6d99-c77e-7344-2abd-317255c39b72/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch to http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/api/v1.0/geo/19af6d99-c77e-7344-2abd-317255c39b72/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch    [05-09-2023 13:49:19 UTC]
(download) Verify downloaded url http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch, proxy DIRECT (curl error 28)    [05-09-2023 13:49:29 UTC]
(download) Trying again on same curl handle, same url: 1, error code 15    [05-09-2023 13:49:29 UTC]
(download) backing off for 1745 ms    [05-09-2023 13:49:29 UTC]
(download) Verify downloaded url http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch, proxy DIRECT (curl error 28)    [05-09-2023 13:49:41 UTC]
(download) download failed (error 15 - host serving data too slowly)    [05-09-2023 13:49:41 UTC]
(download) GeoAPI request http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch failed with error 15 [host serving data too slowly]    [05-09-2023 13:49:41 UTC]
(download) requesting ordered server list from http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch    [05-09-2023 13:49:41 UTC]
(download) replacing @proxy@ by 19af6d99-c77e-7344-2abd-317255c39b72    [05-09-2023 13:49:41 UTC]
(download) escaped http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/api/v1.0/geo/19af6d99-c77e-7344-2abd-317255c39b72/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch to http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/api/v1.0/geo/19af6d99-c77e-7344-2abd-317255c39b72/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch    [05-09-2023 13:49:41 UTC]
(download) Verify downloaded url http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch, proxy DIRECT (curl error 28)    [05-09-2023 13:49:51 UTC]
(download) Trying again on same curl handle, same url: 1, error code 15    [05-09-2023 13:49:51 UTC]
(download) backing off for 430 ms    [05-09-2023 13:49:51 UTC]
(download) Verify downloaded url http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch, proxy DIRECT (curl error 28)    [05-09-2023 13:50:01 UTC]
(download) download failed (error 15 - host serving data too slowly)    [05-09-2023 13:50:01 UTC]
(download) GeoAPI request http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch failed with error 15 [host serving data too slowly]    [05-09-2023 13:50:01 UTC]
(download) requesting ordered server list from http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch    [05-09-2023 13:50:01 UTC]
(download) replacing @proxy@ by 19af6d99-c77e-7344-2abd-317255c39b72    [05-09-2023 13:50:01 UTC]
(download) escaped http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/api/v1.0/geo/19af6d99-c77e-7344-2abd-317255c39b72/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch to http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/api/v1.0/geo/19af6d99-c77e-7344-2abd-317255c39b72/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch    [05-09-2023 13:50:01 UTC]
(download) Verify downloaded url http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch, proxy DIRECT (curl error 28)    [05-09-2023 13:50:11 UTC]
(download) Trying again on same curl handle, same url: 1, error code 15    [05-09-2023 13:50:11 UTC]
(download) backing off for 1709 ms    [05-09-2023 13:50:11 UTC]
(download) Verify downloaded url http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch, proxy DIRECT (curl error 28)    [05-09-2023 13:50:23 UTC]
(download) download failed (error 15 - host serving data too slowly)    [05-09-2023 13:50:23 UTC]
(download) GeoAPI request http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch failed with error 15 [host serving data too slowly]    [05-09-2023 13:50:23 UTC]
(download) failed to retrieve geographic order from stratum 1 servers    [05-09-2023 13:50:23 UTC]
(download) switching proxy from (none) to DIRECT (cloned)    [05-09-2023 13:50:23 UTC]
(download) first fallback proxy group 1    [05-09-2023 13:50:23 UTC]
(download) full proxy list DIRECT    [05-09-2023 13:50:23 UTC]
(download) resolving 1 proxy addresses    [05-09-2023 13:50:23 UTC]
(dns) empty hostname    [05-09-2023 13:50:23 UTC]
(download) installed 1 proxies in 1 load-balance groups    [05-09-2023 13:50:23 UTC]
(cvmfs) DNS roaming is disabled for this repository.    [05-09-2023 13:50:23 UTC]
(catalog) constructing client catalog manager    [05-09-2023 13:50:23 UTC]
(catalog) Initialize catalog    [05-09-2023 13:50:23 UTC]
(cache) unable to read local checksum    [05-09-2023 13:50:23 UTC]
(download) escaped http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/.cvmfspublished to http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/.cvmfspublished    [05-09-2023 13:50:23 UTC]
(download) Verify downloaded url /.cvmfspublished, proxy DIRECT (curl error 28)    [05-09-2023 13:50:33 UTC]
(download) Trying again on same curl handle, same url: 1, error code 15    [05-09-2023 13:50:33 UTC]
(download) backing off for 217 ms    [05-09-2023 13:50:33 UTC]
(download) Verify downloaded url /.cvmfspublished, proxy DIRECT (curl error 28)    [05-09-2023 13:50:43 UTC]
(download) Trying again on same curl handle, same url: 0, error code 15    [05-09-2023 13:50:43 UTC]
(download) switching host from http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch to http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch (host serving data too slowly)    [05-09-2023 13:50:43 UTC]
(download) escaped http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/.cvmfspublished to http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/.cvmfspublished    [05-09-2023 13:50:43 UTC]
(download) Verify downloaded url /.cvmfspublished, proxy DIRECT (curl error 28)    [05-09-2023 13:50:53 UTC]
(download) Trying again on same curl handle, same url: 0, error code 15    [05-09-2023 13:50:53 UTC]
(download) switching host from http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch to http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch (host serving data too slowly)    [05-09-2023 13:50:53 UTC]
(download) escaped http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/.cvmfspublished to http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/.cvmfspublished    [05-09-2023 13:50:53 UTC]
(download) Verify downloaded url /.cvmfspublished, proxy DIRECT (curl error 28)    [05-09-2023 13:51:03 UTC]
(download) Trying again on same curl handle, same url: 0, error code 15    [05-09-2023 13:51:03 UTC]
(download) switching host from http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch to http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch (host serving data too slowly)    [05-09-2023 13:51:03 UTC]
(download) escaped http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/.cvmfspublished to http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/.cvmfspublished    [05-09-2023 13:51:03 UTC]
(download) Verify downloaded url /.cvmfspublished, proxy DIRECT (curl error 28)    [05-09-2023 13:51:13 UTC]
(download) download failed (error 15 - host serving data too slowly)    [05-09-2023 13:51:13 UTC]
(cvmfs) failed to download repository manifest (15 - host serving data too slowly)    [05-09-2023 13:51:13 UTC]
(cache) failed to fetch manifest (1 - failed to download)    [05-09-2023 13:51:13 UTC]
(cache) miss ./00/00000000000000000000000000000000000000 (-2)    [05-09-2023 13:51:13 UTC]
(cache) miss ./00/00000000000000000000000000000000000000 (-2)    [05-09-2023 13:51:13 UTC]
(cache) downloading file catalog at atlas.cern.ch:/    [05-09-2023 13:51:13 UTC]
(cache) start transaction on ./txn/fetchiI6Mha has result 16    [05-09-2023 13:51:13 UTC]
(cache) miss: file catalog at atlas.cern.ch:/ /data/00/00000000000000000000000000000000000000C    [05-09-2023 13:51:13 UTC]
(download) escaped http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C to http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C    [05-09-2023 13:51:13 UTC]
(download) Verify downloaded url /data/00/00000000000000000000000000000000000000C, proxy DIRECT (curl error 28)    [05-09-2023 13:51:23 UTC]
(download) Trying again on same curl handle, same url: 1, error code 15    [05-09-2023 13:51:23 UTC]
(download) backing off for 561 ms    [05-09-2023 13:51:23 UTC]
(download) Verify downloaded url /data/00/00000000000000000000000000000000000000C, proxy DIRECT (curl error 28)    [05-09-2023 13:51:34 UTC]
(download) Trying again on same curl handle, same url: 0, error code 15    [05-09-2023 13:51:34 UTC]
(download) switching host from http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch to http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch (host serving data too slowly)    [05-09-2023 13:51:34 UTC]
(download) escaped http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C to http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C    [05-09-2023 13:51:34 UTC]
(download) Verify downloaded url /data/00/00000000000000000000000000000000000000C, proxy DIRECT (curl error 28)    [05-09-2023 13:51:44 UTC]
(download) Trying again on same curl handle, same url: 0, error code 15    [05-09-2023 13:51:44 UTC]
(download) switching host from http://cvmfs-stratum-one.cern.ch/cvmfs/atlas.cern.ch to http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch (host serving data too slowly)    [05-09-2023 13:51:44 UTC]
(download) escaped http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C to http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C    [05-09-2023 13:51:44 UTC]
(download) Verify downloaded url /data/00/00000000000000000000000000000000000000C, proxy DIRECT (curl error 28)    [05-09-2023 13:51:54 UTC]
(download) Trying again on same curl handle, same url: 0, error code 15    [05-09-2023 13:51:54 UTC]
(download) switching host from http://cernvmfs.gridpp.rl.ac.uk/cvmfs/atlas.cern.ch to http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch (host serving data too slowly)    [05-09-2023 13:51:54 UTC]
(download) escaped http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C to http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/atlas.cern.ch/data/00/00000000000000000000000000000000000000C    [05-09-2023 13:51:54 UTC]
(download) Verify downloaded url /data/00/00000000000000000000000000000000000000C, proxy DIRECT (curl error 28)    [05-09-2023 13:52:04 UTC]
(download) download failed (error 15 - host serving data too slowly)    [05-09-2023 13:52:04 UTC]
(cache) failed to fetch file catalog at atlas.cern.ch:/ (hash: 0000000000000000000000000000000000000000, error 15 [host serving data too slowly])    [05-09-2023 13:52:04 UTC]
(cache) abort ./txn/fetchiI6Mha    [05-09-2023 13:52:04 UTC]
(catalog) failed to load catalog '' (3 - failed to load catalog)    [05-09-2023 13:52:04 UTC]
(catalog) failed to initialize root catalog    [05-09-2023 13:52:04 UTC]
Failed to initialize root file catalog (16 - file catalog failure)
(cache) unpinning / unloading all catalogs    [05-09-2023 13:52:04 UTC]
DiamantisPa commented 1 year ago

Seems that i can't do ping google.com from inside the nodeplugin container.

gman0 commented 1 year ago

Make sure Pods in your cluster have internet connectivity. Pinging from nodeplugin works for me.

DiamantisPa commented 1 year ago

Okay i managed to make it work.

I could ping IPs but not hostnames so it seemed as a DNS problem.

I restarted the core-dns kubectl -n kube-system rollout restart deployment coredns and mounting now works. Also the automount pods work.

I checked the mounting in another node of the cluster and was getting the error host connection problem In that node i disabled the firewall and now it works as well

Although disabling the firewall doesn't seem proper way of doing things.

Thanks a lot for the help @gman0 .

gman0 commented 1 year ago

I'm closing the issue issue was resolved (and wasn't related to CVMFS in particular too). Please re-open or open a new issue if you have more questions.