EMCECS / ECS-CommunityEdition

ECS Community Edition "Free & Frictionless"
https://community.emc.com/community/products/ecs
Other
131 stars 94 forks source link

Storage Pool Management - Stuck on Initializing - Multi and Single node installs #220

Closed cblackuk closed 7 years ago

cblackuk commented 7 years ago

image image

I cannot get Storage Pools to initialise... It is stuck on Not Ready -> initializing on Single and Multi nodes setup...

I have been trying to install ECS Community Edition on vSphere, vCloud Director and now on OpenStack.

Single and Multi node installations alike.

I could see ez_setup.py error described here https://github.com/EMCECS/ECS-CommunityEdition/issues/196

I "fixed" it but still my problem persists. I have now spent about 2 weeks trying to deploy it without success.

I can create buckets and import files without an issue, the "only" problem so far is that it is just not initialising no matter what I do.

Can somebody help please?

My install "log":

sudo python step1_ecs_singlenode_install.py --disks vdb --hostname ecstest --ethadapter eth0 --- Parsed Configuration --- Hostname: ecstest Ethadapter: eth0 Disk[s]: ['vdb'] Docker Image Name: emccorp/ecs-software-3.0.0 Docker Image Tag: latest Use Internet to download image and packages: True [10/May/2017 19:34:16] INFO [root:784] Starting Step 1: Configuration of Host Machine to run the ECS Docker Container: emccorp/ecs-software-3.0.0:latest [10/May/2017 19:34:16] INFO [root:59] Performing installation of the following package: wget . Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile

[10/May/2017 19:34:19] INFO [root:164] Updating the /etc/hosts file with the Parameter Hostname (Ignoring) Hostname Exists: 192.168.1.90 ecstest

[10/May/2017 19:34:19] INFO [root:204] Creating the Network.json file with Ethernet Adapter: eth0 Hostname: ecstest and IP: 192.168.1.90: [10/May/2017 19:34:19] INFO [root:207] {"private_interface_name":"eth0","public_interface_name":"eth0","hostname":"ecstest","data_ip":"192.168.1.90","mgmt_ip":"192.168.1.90","replication_ip":"192.168.1.90"} [10/May/2017 19:34:19] INFO [root:237] Creating the seeds file with IP address: 192.168.1.90 [10/May/2017 19:34:19] INFO [root:268] Partitioning the disk '/dev/vdb' Device does not contain a recognized partition table Building a new DOS disklabel with disk identifier 0xede50803. [10/May/2017 19:34:19] INFO [root:276] Make File filesystem in '/dev/vdb1' meta-data=/dev/vdb1 isize=512 agcount=4, agsize=13107136 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=0, sparse=0 data = bsize=4096 blocks=52428544, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=25599, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 [10/May/2017 19:34:20] INFO [root:281] Make /ecs/uuid-6eee34f0-1b83-4881-8690-4837af07c566 Directory in attached Volume [10/May/2017 19:34:20] INFO [root:285] Mount attached /dev/vdb1 to /ecs/uuid-6eee34f0-1b83-4881-8690-4837af07c566 volume. [10/May/2017 19:34:20] INFO [root:322] Executing the additional preparation script in '/dev/vdb1' create file 0000 create file 0001 create file 0002 create file 0003 create file 0004 create file 0005 create file 0006 create file 0007 create file 0008 create file 0009 create file 0010 create file 0011 create file 0012 create file 0013 create file 0014 create file 0015 create file 0016 create file 0017 create file 0018 field u not found field u not found field u not found field u not found field u not found field u not found field u not found field u not found field u not found field u not found field u not found field u not found field u not found field u not found field u not found field u not found field u not found field u not found field u not found Phase 1 - find and verify superblock... Phase 2 - using internal log

May 10 19:34:22 ecstest dockerd-current[17568]: time="2017-05-10T19:34:22.815783705Z" level=info msg="Graph migration to content-addressability took 0.00 seconds" May 10 19:34:22 ecstest dockerd-current[17568]: time="2017-05-10T19:34:22.816378540Z" level=warning msg="mountpoint for pids not found" May 10 19:34:22 ecstest dockerd-current[17568]: time="2017-05-10T19:34:22.816742808Z" level=info msg="Loading containers: start." May 10 19:34:22 ecstest dockerd-current[17568]: time="2017-05-10T19:34:22.826810858Z" level=info msg="Firewalld running: false" May 10 19:34:22 ecstest dockerd-current[17568]: time="2017-05-10T19:34:22.923963321Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address" May 10 19:34:22 ecstest dockerd-current[17568]: time="2017-05-10T19:34:22.957997997Z" level=info msg="Loading containers: done." May 10 19:34:22 ecstest dockerd-current[17568]: time="2017-05-10T19:34:22.958043651Z" level=info msg="Daemon has completed initialization" May 10 19:34:22 ecstest dockerd-current[17568]: time="2017-05-10T19:34:22.958058820Z" level=info msg="Docker daemon" commit="3a094bd/1.12.6" graphdriver=devicemapper version=1.12.6 May 10 19:34:22 ecstest dockerd-current[17568]: time="2017-05-10T19:34:22.963597382Z" level=info msg="API listen on /var/run/docker.sock" May 10 19:34:22 ecstest systemd[1]: Started Docker Application Container Engine. [10/May/2017 19:34:22] INFO [root:407] Set container to start on boot. [10/May/2017 19:34:23] INFO [root:431] Execute the Docker Container. [10/May/2017 19:34:23] INFO [root:433] docker run -d -e SS_GENCONFIG=1 -v /dev/urandom:/dev/random:z -v /ecs:/dae:z -v /host:/host:z -v /var/log/vipr/emcvipr-object:/var/log:z -v /data:/data:z --net=host --name=ecsstandalone emccorp/ecs-software-3.0.0:latest 3cca831356592343729afba5e1fb970b6913fa123c9ce97eb7bdd1bde8fc1952 [10/May/2017 19:34:23] INFO [root:437] Check the Docker processes. CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 3cca83135659 emccorp/ecs-software-3.0.0:latest "/opt/vipr/boot/boot." Less than a second ago Up Less than a second ecsstandalone [10/May/2017 19:34:23] INFO [root:463] Backup object properties files [10/May/2017 19:34:23] INFO [root:467] Backup application config file [10/May/2017 19:34:23] INFO [root:471] Backup common-object properties file [10/May/2017 19:34:23] INFO [root:475] Backup ssm properties file [10/May/2017 19:34:23] INFO [root:479] Copy object properties files to host [10/May/2017 19:34:23] INFO [root:483] Copy common-object properties files to host [10/May/2017 19:34:24] INFO [root:487] Copy ssm properties files to host [10/May/2017 19:34:24] INFO [root:491] Modify BlobSvc config for single node [10/May/2017 19:34:24] INFO [root:495] Modify Directory Table config for single node [10/May/2017 19:34:24] INFO [root:499] Modify SSM config for small footprint [10/May/2017 19:34:24] INFO [root:503] Copy modified files to container [10/May/2017 19:34:24] INFO [root:515] Adding python setuptools to container % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 12537 100 12537 0 0 123k 0 --:--:-- --:--:-- --:--:-- 124k ez_setup.py is deprecated and when using it setuptools will be pinned to 33.1.1 since it's the last version that supports setuptools self upgrade/installation, check https://github.com/pypa/setuptools/issues/581 for more info; use pip to install setuptools Downloading https://pypi.io/packages/source/s/setuptools/setuptools-33.1.1.zip Extracting in /tmp/tmp1XVK7b Now working in /tmp/tmp1XVK7b/setuptools-33.1.1 Installing Setuptools running install running bdist_egg running egg_info writing requirements to setuptools.egg-info/requires.txt writing setuptools.egg-info/PKG-INFO writing top-level names to setuptools.egg-info/top_level.txt writing dependency_links to setuptools.egg-info/dependency_links.txt writing entry points to setuptools.egg-info/entry_points.txt reading manifest file 'setuptools.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no files found matching '*' under directory 'setuptools/_vendor' writing manifest file 'setuptools.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_py creating build creating build/lib copying easy_install.py -> build/lib creating build/lib/pkg_resources copying pkg_resources/init.py -> build/lib/pkg_resources creating build/lib/setuptools copying setuptools/init.py -> build/lib/setuptools copying setuptools/archive_util.py -> build/lib/setuptools copying setuptools/config.py -> build/lib/setuptools copying setuptools/depends.py -> build/lib/setuptools copying setuptools/dist.py -> build/lib/setuptools copying setuptools/extension.py -> build/lib/setuptools copying setuptools/glob.py -> build/lib/setuptools copying setuptools/launch.py -> build/lib/setuptools copying setuptools/lib2to3_ex.py -> build/lib/setuptools copying setuptools/monkey.py -> build/lib/setuptools copying setuptools/msvc.py -> build/lib/setuptools copying setuptools/namespaces.py -> build/lib/setuptools copying setuptools/package_index.py -> build/lib/setuptools copying setuptools/py26compat.py -> build/lib/setuptools copying setuptools/py27compat.py -> build/lib/setuptools copying setuptools/py31compat.py -> build/lib/setuptools copying setuptools/py33compat.py -> build/lib/setuptools copying setuptools/py36compat.py -> build/lib/setuptools copying setuptools/sandbox.py -> build/lib/setuptools copying setuptools/site-patch.py -> build/lib/setuptools copying setuptools/ssl_support.py -> build/lib/setuptools copying setuptools/unicode_utils.py -> build/lib/setuptools copying setuptools/version.py -> build/lib/setuptools copying setuptools/windows_support.py -> build/lib/setuptools creating build/lib/pkg_resources/_vendor copying pkg_resources/_vendor/init.py -> build/lib/pkg_resources/_vendor copying pkg_resources/_vendor/appdirs.py -> build/lib/pkg_resources/_vendor copying pkg_resources/_vendor/pyparsing.py -> build/lib/pkg_resources/_vendor copying pkg_resources/_vendor/six.py -> build/lib/pkg_resources/_vendor creating build/lib/pkg_resources/extern copying pkg_resources/extern/init.py -> build/lib/pkg_resources/extern creating build/lib/pkg_resources/_vendor/packaging copying pkg_resources/_vendor/packaging/about.py -> build/lib/pkg_resources/_vendor/packaging copying pkg_resources/_vendor/packaging/init.py -> build/lib/pkg_resources/_vendor/packaging copying pkg_resources/_vendor/packaging/_compat.py -> build/lib/pkg_resources/_vendor/packaging copying pkg_resources/_vendor/packaging/_structures.py -> build/lib/pkg_resources/_vendor/packaging copying pkg_resources/_vendor/packaging/markers.py -> build/lib/pkg_resources/_vendor/packaging copying pkg_resources/_vendor/packaging/requirements.py -> build/lib/pkg_resources/_vendor/packaging copying pkg_resources/_vendor/packaging/specifiers.py -> build/lib/pkg_resources/_vendor/packaging copying pkg_resources/_vendor/packaging/utils.py -> build/lib/pkg_resources/_vendor/packaging copying pkg_resources/_vendor/packaging/version.py -> build/lib/pkg_resources/_vendor/packaging creating build/lib/setuptools/command copying setuptools/command/init.py -> build/lib/setuptools/command copying setuptools/command/alias.py -> build/lib/setuptools/command copying setuptools/command/bdist_egg.py -> build/lib/setuptools/command copying setuptools/command/bdist_rpm.py -> build/lib/setuptools/command copying setuptools/command/bdist_wininst.py -> build/lib/setuptools/command copying setuptools/command/build_ext.py -> build/lib/setuptools/command copying setuptools/command/build_py.py -> build/lib/setuptools/command copying setuptools/command/develop.py -> build/lib/setuptools/command copying setuptools/command/easy_install.py -> build/lib/setuptools/command copying setuptools/command/egg_info.py -> build/lib/setuptools/command copying setuptools/command/install.py -> build/lib/setuptools/command copying setuptools/command/install_egg_info.py -> build/lib/setuptools/command copying setuptools/command/install_lib.py -> build/lib/setuptools/command copying setuptools/command/install_scripts.py -> build/lib/setuptools/command copying setuptools/command/py36compat.py -> build/lib/setuptools/command copying setuptools/command/register.py -> build/lib/setuptools/command copying setuptools/command/rotate.py -> build/lib/setuptools/command copying setuptools/command/saveopts.py -> build/lib/setuptools/command copying setuptools/command/sdist.py -> build/lib/setuptools/command copying setuptools/command/setopt.py -> build/lib/setuptools/command copying setuptools/command/test.py -> build/lib/setuptools/command copying setuptools/command/upload.py -> build/lib/setuptools/command copying setuptools/command/upload_docs.py -> build/lib/setuptools/command creating build/lib/setuptools/extern copying setuptools/extern/init.py -> build/lib/setuptools/extern copying setuptools/script (dev).tmpl -> build/lib/setuptools copying setuptools/script.tmpl -> build/lib/setuptools copying setuptools/cli-32.exe -> build/lib/setuptools copying setuptools/cli-64.exe -> build/lib/setuptools copying setuptools/cli.exe -> build/lib/setuptools copying setuptools/gui-32.exe -> build/lib/setuptools copying setuptools/gui-64.exe -> build/lib/setuptools copying setuptools/gui.exe -> build/lib/setuptools copying setuptools/command/launcher manifest.xml -> build/lib/setuptools/command creating build/bdist.linux-x86_64 creating build/bdist.linux-x86_64/egg copying build/lib/easy_install.py -> build/bdist.linux-x86_64/egg creating build/bdist.linux-x86_64/egg/pkg_resources copying build/lib/pkg_resources/init.py -> build/bdist.linux-x86_64/egg/pkg_resources creating build/bdist.linux-x86_64/egg/pkg_resources/_vendor copying build/lib/pkg_resources/_vendor/init.py -> build/bdist.linux-x86_64/egg/pkg_resources/_vendor copying build/lib/pkg_resources/_vendor/appdirs.py -> build/bdist.linux-x86_64/egg/pkg_resources/_vendor copying build/lib/pkg_resources/_vendor/pyparsing.py -> build/bdist.linux-x86_64/egg/pkg_resources/_vendor copying build/lib/pkg_resources/_vendor/six.py -> build/bdist.linux-x86_64/egg/pkg_resources/_vendor creating build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging copying build/lib/pkg_resources/_vendor/packaging/about.py -> build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging copying build/lib/pkg_resources/_vendor/packaging/init.py -> build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging copying build/lib/pkg_resources/_vendor/packaging/_compat.py -> build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging copying build/lib/pkg_resources/_vendor/packaging/_structures.py -> build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging copying build/lib/pkg_resources/_vendor/packaging/markers.py -> build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging copying build/lib/pkg_resources/_vendor/packaging/requirements.py -> build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging copying build/lib/pkg_resources/_vendor/packaging/specifiers.py -> build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging copying build/lib/pkg_resources/_vendor/packaging/utils.py -> build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging copying build/lib/pkg_resources/_vendor/packaging/version.py -> build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging creating build/bdist.linux-x86_64/egg/pkg_resources/extern copying build/lib/pkg_resources/extern/init.py -> build/bdist.linux-x86_64/egg/pkg_resources/extern creating build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/init.py -> build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/archive_util.py -> build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/config.py -> build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/depends.py -> build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/dist.py -> build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/extension.py -> build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/glob.py -> build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/launch.py -> build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/lib2to3_ex.py -> build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/monkey.py -> build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/msvc.py -> build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/namespaces.py -> build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/package_index.py -> build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/py26compat.py -> build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/py27compat.py -> build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/py31compat.py -> build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/py33compat.py -> build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/py36compat.py -> build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/sandbox.py -> build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/site-patch.py -> build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/ssl_support.py -> build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/unicode_utils.py -> build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/version.py -> build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/windows_support.py -> build/bdist.linux-x86_64/egg/setuptools creating build/bdist.linux-x86_64/egg/setuptools/command copying build/lib/setuptools/command/init.py -> build/bdist.linux-x86_64/egg/setuptools/command copying build/lib/setuptools/command/alias.py -> build/bdist.linux-x86_64/egg/setuptools/command copying build/lib/setuptools/command/bdist_egg.py -> build/bdist.linux-x86_64/egg/setuptools/command copying build/lib/setuptools/command/bdist_rpm.py -> build/bdist.linux-x86_64/egg/setuptools/command copying build/lib/setuptools/command/bdist_wininst.py -> build/bdist.linux-x86_64/egg/setuptools/command copying build/lib/setuptools/command/build_ext.py -> build/bdist.linux-x86_64/egg/setuptools/command copying build/lib/setuptools/command/build_py.py -> build/bdist.linux-x86_64/egg/setuptools/command copying build/lib/setuptools/command/develop.py -> build/bdist.linux-x86_64/egg/setuptools/command copying build/lib/setuptools/command/easy_install.py -> build/bdist.linux-x86_64/egg/setuptools/command copying build/lib/setuptools/command/egg_info.py -> build/bdist.linux-x86_64/egg/setuptools/command copying build/lib/setuptools/command/install.py -> build/bdist.linux-x86_64/egg/setuptools/command copying build/lib/setuptools/command/install_egg_info.py -> build/bdist.linux-x86_64/egg/setuptools/command copying build/lib/setuptools/command/install_lib.py -> build/bdist.linux-x86_64/egg/setuptools/command copying build/lib/setuptools/command/install_scripts.py -> build/bdist.linux-x86_64/egg/setuptools/command copying build/lib/setuptools/command/py36compat.py -> build/bdist.linux-x86_64/egg/setuptools/command copying build/lib/setuptools/command/register.py -> build/bdist.linux-x86_64/egg/setuptools/command copying build/lib/setuptools/command/rotate.py -> build/bdist.linux-x86_64/egg/setuptools/command copying build/lib/setuptools/command/saveopts.py -> build/bdist.linux-x86_64/egg/setuptools/command copying build/lib/setuptools/command/sdist.py -> build/bdist.linux-x86_64/egg/setuptools/command copying build/lib/setuptools/command/setopt.py -> build/bdist.linux-x86_64/egg/setuptools/command copying build/lib/setuptools/command/test.py -> build/bdist.linux-x86_64/egg/setuptools/command copying build/lib/setuptools/command/upload.py -> build/bdist.linux-x86_64/egg/setuptools/command copying build/lib/setuptools/command/upload_docs.py -> build/bdist.linux-x86_64/egg/setuptools/command copying build/lib/setuptools/command/launcher manifest.xml -> build/bdist.linux-x86_64/egg/setuptools/command creating build/bdist.linux-x86_64/egg/setuptools/extern copying build/lib/setuptools/extern/init.py -> build/bdist.linux-x86_64/egg/setuptools/extern copying build/lib/setuptools/script (dev).tmpl -> build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/script.tmpl -> build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/cli-32.exe -> build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/cli-64.exe -> build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/cli.exe -> build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/gui-32.exe -> build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/gui-64.exe -> build/bdist.linux-x86_64/egg/setuptools copying build/lib/setuptools/gui.exe -> build/bdist.linux-x86_64/egg/setuptools byte-compiling build/bdist.linux-x86_64/egg/easy_install.py to easy_install.pyc byte-compiling build/bdist.linux-x86_64/egg/pkg_resources/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/pkg_resources/_vendor/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/pkg_resources/_vendor/appdirs.py to appdirs.pyc byte-compiling build/bdist.linux-x86_64/egg/pkg_resources/_vendor/pyparsing.py to pyparsing.pyc byte-compiling build/bdist.linux-x86_64/egg/pkg_resources/_vendor/six.py to six.pyc byte-compiling build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging/about.py to about.pyc byte-compiling build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging/_compat.py to _compat.pyc byte-compiling build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging/_structures.py to _structures.pyc byte-compiling build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging/markers.py to markers.pyc byte-compiling build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging/requirements.py to requirements.pyc byte-compiling build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging/specifiers.py to specifiers.pyc byte-compiling build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging/utils.py to utils.pyc byte-compiling build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging/version.py to version.pyc byte-compiling build/bdist.linux-x86_64/egg/pkg_resources/extern/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/archive_util.py to archive_util.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/config.py to config.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/depends.py to depends.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/dist.py to dist.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/extension.py to extension.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/glob.py to glob.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/launch.py to launch.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/lib2to3_ex.py to lib2to3_ex.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/monkey.py to monkey.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/msvc.py to msvc.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/namespaces.py to namespaces.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/package_index.py to package_index.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/py26compat.py to py26compat.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/py27compat.py to py27compat.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/py31compat.py to py31compat.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/py33compat.py to py33compat.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/py36compat.py to py36compat.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/sandbox.py to sandbox.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/site-patch.py to site-patch.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/ssl_support.py to ssl_support.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/unicode_utils.py to unicode_utils.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/version.py to version.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/windows_support.py to windows_support.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/alias.py to alias.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/bdist_egg.py to bdist_egg.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/bdist_rpm.py to bdist_rpm.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/bdist_wininst.py to bdist_wininst.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/build_ext.py to build_ext.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/build_py.py to build_py.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/develop.py to develop.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py to easy_install.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/egg_info.py to egg_info.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/install.py to install.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/install_egg_info.py to install_egg_info.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/install_lib.py to install_lib.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/install_scripts.py to install_scripts.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/py36compat.py to py36compat.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/register.py to register.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/rotate.py to rotate.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/saveopts.py to saveopts.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/sdist.py to sdist.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/setopt.py to setopt.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/test.py to test.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/upload.py to upload.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/upload_docs.py to upload_docs.pyc byte-compiling build/bdist.linux-x86_64/egg/setuptools/extern/init.py to init.pyc creating build/bdist.linux-x86_64/egg/EGG-INFO copying setuptools.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO copying setuptools.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying setuptools.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying setuptools.egg-info/entry_points.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying setuptools.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying setuptools.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying setuptools.egg-info/zip-safe -> build/bdist.linux-x86_64/egg/EGG-INFO creating dist creating 'dist/setuptools-33.1.1-py2.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it removing 'build/bdist.linux-x86_64/egg' (and everything under it) Processing setuptools-33.1.1-py2.7.egg Copying setuptools-33.1.1-py2.7.egg to /usr/lib/python2.7/site-packages Adding setuptools 33.1.1 to easy-install.pth file Installing easy_install script to /usr/bin Installing easy_install-2.7 script to /usr/bin

Installed /usr/lib/python2.7/site-packages/setuptools-33.1.1-py2.7.egg Processing dependencies for setuptools==33.1.1 Finished processing dependencies for setuptools==33.1.1 [10/May/2017 19:34:26] INFO [root:519] Adding python requests library to container % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 136 0 136 0 0 359 0 --:--:-- --:--:-- --:--:-- 358 0 0 0 3389k 0 0 2157k 0 --:--:-- 0:00:01 --:--:-- 7991k kennethreitz-requests-0280a18/ kennethreitz-requests-0280a18/.coveragerc kennethreitz-requests-0280a18/.gitignore kennethreitz-requests-0280a18/.travis.yml kennethreitz-requests-0280a18/AUTHORS.rst kennethreitz-requests-0280a18/CONTRIBUTING.md kennethreitz-requests-0280a18/HISTORY.rst kennethreitz-requests-0280a18/LICENSE kennethreitz-requests-0280a18/MANIFEST.in kennethreitz-requests-0280a18/Makefile kennethreitz-requests-0280a18/NOTICE kennethreitz-requests-0280a18/Pipfile kennethreitz-requests-0280a18/Pipfile.lock kennethreitz-requests-0280a18/README.rst kennethreitz-requests-0280a18/_appveyor/ kennethreitz-requests-0280a18/_appveyor/install.ps1 kennethreitz-requests-0280a18/appveyor.yml kennethreitz-requests-0280a18/docs/ kennethreitz-requests-0280a18/docs/Makefile kennethreitz-requests-0280a18/docs/_static/ kennethreitz-requests-0280a18/docs/_static/konami.js kennethreitz-requests-0280a18/docs/_static/requests-logo-small.png kennethreitz-requests-0280a18/docs/_static/requests-sidebar.png kennethreitz-requests-0280a18/docs/_templates/ kennethreitz-requests-0280a18/docs/_templates/hacks.html kennethreitz-requests-0280a18/docs/_templates/sidebarintro.html kennethreitz-requests-0280a18/docs/_templates/sidebarlogo.html kennethreitz-requests-0280a18/docs/_themes/ kennethreitz-requests-0280a18/docs/_themes/.gitignore kennethreitz-requests-0280a18/docs/_themes/LICENSE kennethreitz-requests-0280a18/docs/_themes/flask_theme_support.py kennethreitz-requests-0280a18/docs/api.rst kennethreitz-requests-0280a18/docs/community/ kennethreitz-requests-0280a18/docs/community/faq.rst kennethreitz-requests-0280a18/docs/community/out-there.rst kennethreitz-requests-0280a18/docs/community/recommended.rst kennethreitz-requests-0280a18/docs/community/release-process.rst kennethreitz-requests-0280a18/docs/community/support.rst kennethreitz-requests-0280a18/docs/community/updates.rst kennethreitz-requests-0280a18/docs/community/vulnerabilities.rst kennethreitz-requests-0280a18/docs/conf.py kennethreitz-requests-0280a18/docs/dev/ kennethreitz-requests-0280a18/docs/dev/authors.rst kennethreitz-requests-0280a18/docs/dev/contributing.rst kennethreitz-requests-0280a18/docs/dev/philosophy.rst kennethreitz-requests-0280a18/docs/dev/todo.rst kennethreitz-requests-0280a18/docs/index.rst kennethreitz-requests-0280a18/docs/make.bat kennethreitz-requests-0280a18/docs/user/ kennethreitz-requests-0280a18/docs/user/advanced.rst kennethreitz-requests-0280a18/docs/user/authentication.rst kennethreitz-requests-0280a18/docs/user/install.rst kennethreitz-requests-0280a18/docs/user/intro.rst kennethreitz-requests-0280a18/docs/user/quickstart.rst kennethreitz-requests-0280a18/ext/ kennethreitz-requests-0280a18/ext/requests-logo.ai kennethreitz-requests-0280a18/ext/requests-logo.svg kennethreitz-requests-0280a18/requests/ kennethreitz-requests-0280a18/requests/init.py kennethreitz-requests-0280a18/requests/_internal_utils.py kennethreitz-requests-0280a18/requests/adapters.py kennethreitz-requests-0280a18/requests/api.py kennethreitz-requests-0280a18/requests/auth.py kennethreitz-requests-0280a18/requests/cacert.pem kennethreitz-requests-0280a18/requests/certs.py kennethreitz-requests-0280a18/requests/compat.py kennethreitz-requests-0280a18/requests/cookies.py kennethreitz-requests-0280a18/requests/exceptions.py kennethreitz-requests-0280a18/requests/hooks.py kennethreitz-requests-0280a18/requests/models.py kennethreitz-requests-0280a18/requests/packages/ kennethreitz-requests-0280a18/requests/packages/README.rst kennethreitz-requests-0280a18/requests/packages/init.py kennethreitz-requests-0280a18/requests/packages/chardet/ kennethreitz-requests-0280a18/requests/packages/chardet/init.py kennethreitz-requests-0280a18/requests/packages/chardet/big5freq.py kennethreitz-requests-0280a18/requests/packages/chardet/big5prober.py kennethreitz-requests-0280a18/requests/packages/chardet/chardistribution.py kennethreitz-requests-0280a18/requests/packages/chardet/charsetgroupprober.py kennethreitz-requests-0280a18/requests/packages/chardet/charsetprober.py kennethreitz-requests-0280a18/requests/packages/chardet/cli/ kennethreitz-requests-0280a18/requests/packages/chardet/cli/init.py kennethreitz-requests-0280a18/requests/packages/chardet/cli/chardetect.py kennethreitz-requests-0280a18/requests/packages/chardet/codingstatemachine.py kennethreitz-requests-0280a18/requests/packages/chardet/compat.py kennethreitz-requests-0280a18/requests/packages/chardet/cp949prober.py kennethreitz-requests-0280a18/requests/packages/chardet/enums.py kennethreitz-requests-0280a18/requests/packages/chardet/escprober.py kennethreitz-requests-0280a18/requests/packages/chardet/escsm.py kennethreitz-requests-0280a18/requests/packages/chardet/eucjpprober.py kennethreitz-requests-0280a18/requests/packages/chardet/euckrfreq.py kennethreitz-requests-0280a18/requests/packages/chardet/euckrprober.py kennethreitz-requests-0280a18/requests/packages/chardet/euctwfreq.py kennethreitz-requests-0280a18/requests/packages/chardet/euctwprober.py kennethreitz-requests-0280a18/requests/packages/chardet/gb2312freq.py kennethreitz-requests-0280a18/requests/packages/chardet/gb2312prober.py kennethreitz-requests-0280a18/requests/packages/chardet/hebrewprober.py kennethreitz-requests-0280a18/requests/packages/chardet/jisfreq.py kennethreitz-requests-0280a18/requests/packages/chardet/jpcntx.py kennethreitz-requests-0280a18/requests/packages/chardet/langbulgarianmodel.py kennethreitz-requests-0280a18/requests/packages/chardet/langcyrillicmodel.py kennethreitz-requests-0280a18/requests/packages/chardet/langgreekmodel.py kennethreitz-requests-0280a18/requests/packages/chardet/langhebrewmodel.py kennethreitz-requests-0280a18/requests/packages/chardet/langhungarianmodel.py kennethreitz-requests-0280a18/requests/packages/chardet/langthaimodel.py kennethreitz-requests-0280a18/requests/packages/chardet/langturkishmodel.py kennethreitz-requests-0280a18/requests/packages/chardet/latin1prober.py kennethreitz-requests-0280a18/requests/packages/chardet/mbcharsetprober.py kennethreitz-requests-0280a18/requests/packages/chardet/mbcsgroupprober.py kennethreitz-requests-0280a18/requests/packages/chardet/mbcssm.py kennethreitz-requests-0280a18/requests/packages/chardet/sbcharsetprober.py kennethreitz-requests-0280a18/requests/packages/chardet/sbcsgroupprober.py kennethreitz-requests-0280a18/requests/packages/chardet/sjisprober.py kennethreitz-requests-0280a18/requests/packages/chardet/universaldetector.py kennethreitz-requests-0280a18/requests/packages/chardet/utf8prober.py kennethreitz-requests-0280a18/requests/packages/chardet/version.py kennethreitz-requests-0280a18/requests/packages/idna/ kennethreitz-requests-0280a18/requests/packages/idna/init.py kennethreitz-requests-0280a18/requests/packages/idna/codec.py kennethreitz-requests-0280a18/requests/packages/idna/compat.py kennethreitz-requests-0280a18/requests/packages/idna/core.py kennethreitz-requests-0280a18/requests/packages/idna/idnadata.py kennethreitz-requests-0280a18/requests/packages/idna/intranges.py kennethreitz-requests-0280a18/requests/packages/idna/uts46data.py kennethreitz-requests-0280a18/requests/packages/urllib3/ kennethreitz-requests-0280a18/requests/packages/urllib3/init.py kennethreitz-requests-0280a18/requests/packages/urllib3/_collections.py kennethreitz-requests-0280a18/requests/packages/urllib3/connection.py kennethreitz-requests-0280a18/requests/packages/urllib3/connectionpool.py kennethreitz-requests-0280a18/requests/packages/urllib3/contrib/ kennethreitz-requests-0280a18/requests/packages/urllib3/contrib/init.py kennethreitz-requests-0280a18/requests/packages/urllib3/contrib/_securetransport/ kennethreitz-requests-0280a18/requests/packages/urllib3/contrib/_securetransport/init.py kennethreitz-requests-0280a18/requests/packages/urllib3/contrib/_securetransport/bindings.py kennethreitz-requests-0280a18/requests/packages/urllib3/contrib/_securetransport/low_level.py kennethreitz-requests-0280a18/requests/packages/urllib3/contrib/appengine.py kennethreitz-requests-0280a18/requests/packages/urllib3/contrib/ntlmpool.py kennethreitz-requests-0280a18/requests/packages/urllib3/contrib/pyopenssl.py kennethreitz-requests-0280a18/requests/packages/urllib3/contrib/securetransport.py kennethreitz-requests-0280a18/requests/packages/urllib3/contrib/socks.py kennethreitz-requests-0280a18/requests/packages/urllib3/exceptions.py kennethreitz-requests-0280a18/requests/packages/urllib3/fields.py kennethreitz-requests-0280a18/requests/packages/urllib3/filepost.py kennethreitz-requests-0280a18/requests/packages/urllib3/packages/ kennethreitz-requests-0280a18/requests/packages/urllib3/packages/init.py kennethreitz-requests-0280a18/requests/packages/urllib3/packages/backports/ kennethreitz-requests-0280a18/requests/packages/urllib3/packages/backports/init.py kennethreitz-requests-0280a18/requests/packages/urllib3/packages/backports/makefile.py kennethreitz-requests-0280a18/requests/packages/urllib3/packages/ordered_dict.py kennethreitz-requests-0280a18/requests/packages/urllib3/packages/six.py kennethreitz-requests-0280a18/requests/packages/urllib3/packages/ssl_match_hostname/ kennethreitz-requests-0280a18/requests/packages/urllib3/packages/ssl_match_hostname/init.py kennethreitz-requests-0280a18/requests/packages/urllib3/packages/ssl_match_hostname/implementation.py kennethreitz-requests-0280a18/requests/packages/urllib3/poolmanager.py kennethreitz-requests-0280a18/requests/packages/urllib3/request.py kennethreitz-requests-0280a18/requests/packages/urllib3/response.py kennethreitz-requests-0280a18/requests/packages/urllib3/util/ kennethreitz-requests-0280a18/requests/packages/urllib3/util/init.py kennethreitz-requests-0280a18/requests/packages/urllib3/util/connection.py kennethreitz-requests-0280a18/requests/packages/urllib3/util/request.py kennethreitz-requests-0280a18/requests/packages/urllib3/util/response.py kennethreitz-requests-0280a18/requests/packages/urllib3/util/retry.py kennethreitz-requests-0280a18/requests/packages/urllib3/util/selectors.py kennethreitz-requests-0280a18/requests/packages/urllib3/util/ssl.py kennethreitz-requests-0280a18/requests/packages/urllib3/util/timeout.py kennethreitz-requests-0280a18/requests/packages/urllib3/util/url.py kennethreitz-requests-0280a18/requests/packages/urllib3/util/wait.py kennethreitz-requests-0280a18/requests/sessions.py kennethreitz-requests-0280a18/requests/status_codes.py kennethreitz-requests-0280a18/requests/structures.py kennethreitz-requests-0280a18/requests/utils.py kennethreitz-requests-0280a18/setup.py kennethreitz-requests-0280a18/tests/ kennethreitz-requests-0280a18/tests/init.py kennethreitz-requests-0280a18/tests/compat.py kennethreitz-requests-0280a18/tests/conftest.py kennethreitz-requests-0280a18/tests/test_hooks.py kennethreitz-requests-0280a18/tests/test_lowlevel.py kennethreitz-requests-0280a18/tests/test_requests.py kennethreitz-requests-0280a18/tests/test_structures.py kennethreitz-requests-0280a18/tests/test_testserver.py kennethreitz-requests-0280a18/tests/test_utils.py kennethreitz-requests-0280a18/tests/testserver/ kennethreitz-requests-0280a18/tests/testserver/init.py kennethreitz-requests-0280a18/tests/testserver/server.py kennethreitz-requests-0280a18/tests/utils.py running install running bdist_egg running egg_info creating requests.egg-info writing requirements to requests.egg-info/requires.txt writing requests.egg-info/PKG-INFO writing top-level names to requests.egg-info/top_level.txt writing dependency_links to requests.egg-info/dependency_links.txt writing manifest file 'requests.egg-info/SOURCES.txt' reading manifest file 'requests.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'requests.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_py creating build creating build/lib creating build/lib/requests copying requests/init.py -> build/lib/requests copying requests/_internal_utils.py -> build/lib/requests copying requests/adapters.py -> build/lib/requests copying requests/api.py -> build/lib/requests copying requests/auth.py -> build/lib/requests copying requests/certs.py -> build/lib/requests copying requests/compat.py -> build/lib/requests copying requests/cookies.py -> build/lib/requests copying requests/exceptions.py -> build/lib/requests copying requests/hooks.py -> build/lib/requests copying requests/models.py -> build/lib/requests copying requests/sessions.py -> build/lib/requests copying requests/status_codes.py -> build/lib/requests copying requests/structures.py -> build/lib/requests copying requests/utils.py -> build/lib/requests creating build/lib/requests/packages copying requests/packages/init.py -> build/lib/requests/packages creating build/lib/requests/packages/chardet copying requests/packages/chardet/init.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/big5freq.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/big5prober.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/chardistribution.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/charsetgroupprober.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/charsetprober.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/codingstatemachine.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/compat.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/cp949prober.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/enums.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/escprober.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/escsm.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/eucjpprober.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/euckrfreq.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/euckrprober.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/euctwfreq.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/euctwprober.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/gb2312freq.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/gb2312prober.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/hebrewprober.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/jisfreq.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/jpcntx.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/langbulgarianmodel.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/langcyrillicmodel.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/langgreekmodel.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/langhebrewmodel.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/langhungarianmodel.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/langthaimodel.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/langturkishmodel.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/latin1prober.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/mbcharsetprober.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/mbcsgroupprober.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/mbcssm.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/sbcharsetprober.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/sbcsgroupprober.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/sjisprober.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/universaldetector.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/utf8prober.py -> build/lib/requests/packages/chardet copying requests/packages/chardet/version.py -> build/lib/requests/packages/chardet creating build/lib/requests/packages/chardet/cli copying requests/packages/chardet/cli/init.py -> build/lib/requests/packages/chardet/cli copying requests/packages/chardet/cli/chardetect.py -> build/lib/requests/packages/chardet/cli creating build/lib/requests/packages/idna copying requests/packages/idna/init.py -> build/lib/requests/packages/idna copying requests/packages/idna/codec.py -> build/lib/requests/packages/idna copying requests/packages/idna/compat.py -> build/lib/requests/packages/idna copying requests/packages/idna/core.py -> build/lib/requests/packages/idna copying requests/packages/idna/idnadata.py -> build/lib/requests/packages/idna copying requests/packages/idna/intranges.py -> build/lib/requests/packages/idna copying requests/packages/idna/uts46data.py -> build/lib/requests/packages/idna creating build/lib/requests/packages/urllib3 copying requests/packages/urllib3/init.py -> build/lib/requests/packages/urllib3 copying requests/packages/urllib3/_collections.py -> build/lib/requests/packages/urllib3 copying requests/packages/urllib3/connection.py -> build/lib/requests/packages/urllib3 copying requests/packages/urllib3/connectionpool.py -> build/lib/requests/packages/urllib3 copying requests/packages/urllib3/exceptions.py -> build/lib/requests/packages/urllib3 copying requests/packages/urllib3/fields.py -> build/lib/requests/packages/urllib3 copying requests/packages/urllib3/filepost.py -> build/lib/requests/packages/urllib3 copying requests/packages/urllib3/poolmanager.py -> build/lib/requests/packages/urllib3 copying requests/packages/urllib3/request.py -> build/lib/requests/packages/urllib3 copying requests/packages/urllib3/response.py -> build/lib/requests/packages/urllib3 creating build/lib/requests/packages/urllib3/packages copying requests/packages/urllib3/packages/init.py -> build/lib/requests/packages/urllib3/packages copying requests/packages/urllib3/packages/ordereddict.py -> build/lib/requests/packages/urllib3/packages copying requests/packages/urllib3/packages/six.py -> build/lib/requests/packages/urllib3/packages creating build/lib/requests/packages/urllib3/contrib copying requests/packages/urllib3/contrib/init.py -> build/lib/requests/packages/urllib3/contrib copying requests/packages/urllib3/contrib/appengine.py -> build/lib/requests/packages/urllib3/contrib copying requests/packages/urllib3/contrib/ntlmpool.py -> build/lib/requests/packages/urllib3/contrib copying requests/packages/urllib3/contrib/pyopenssl.py -> build/lib/requests/packages/urllib3/contrib copying requests/packages/urllib3/contrib/securetransport.py -> build/lib/requests/packages/urllib3/contrib copying requests/packages/urllib3/contrib/socks.py -> build/lib/requests/packages/urllib3/contrib creating build/lib/requests/packages/urllib3/util copying requests/packages/urllib3/util/init.py -> build/lib/requests/packages/urllib3/util copying requests/packages/urllib3/util/connection.py -> build/lib/requests/packages/urllib3/util copying requests/packages/urllib3/util/request.py -> build/lib/requests/packages/urllib3/util copying requests/packages/urllib3/util/response.py -> build/lib/requests/packages/urllib3/util copying requests/packages/urllib3/util/retry.py -> build/lib/requests/packages/urllib3/util copying requests/packages/urllib3/util/selectors.py -> build/lib/requests/packages/urllib3/util copying requests/packages/urllib3/util/ssl.py -> build/lib/requests/packages/urllib3/util copying requests/packages/urllib3/util/timeout.py -> build/lib/requests/packages/urllib3/util copying requests/packages/urllib3/util/url.py -> build/lib/requests/packages/urllib3/util copying requests/packages/urllib3/util/wait.py -> build/lib/requests/packages/urllib3/util creating build/lib/requests/packages/urllib3/packages/ssl_match_hostname copying requests/packages/urllib3/packages/ssl_match_hostname/init.py -> build/lib/requests/packages/urllib3/packages/ssl_match_hostname copying requests/packages/urllib3/packages/ssl_match_hostname/_implementation.py -> build/lib/requests/packages/urllib3/packages/ssl_match_hostname creating build/lib/requests/packages/urllib3/packages/backports copying requests/packages/urllib3/packages/backports/init.py -> build/lib/requests/packages/urllib3/packages/backports copying requests/packages/urllib3/packages/backports/makefile.py -> build/lib/requests/packages/urllib3/packages/backports creating build/lib/requests/packages/urllib3/contrib/_securetransport copying requests/packages/urllib3/contrib/_securetransport/init.py -> build/lib/requests/packages/urllib3/contrib/_securetransport copying requests/packages/urllib3/contrib/_securetransport/bindings.py -> build/lib/requests/packages/urllib3/contrib/_securetransport copying requests/packages/urllib3/contrib/_securetransport/low_level.py -> build/lib/requests/packages/urllib3/contrib/_securetransport copying requests/cacert.pem -> build/lib/requests creating build/bdist.linux-x86_64 creating build/bdist.linux-x86_64/egg creating build/bdist.linux-x86_64/egg/requests copying build/lib/requests/init.py -> build/bdist.linux-x86_64/egg/requests copying build/lib/requests/_internal_utils.py -> build/bdist.linux-x86_64/egg/requests copying build/lib/requests/adapters.py -> build/bdist.linux-x86_64/egg/requests copying build/lib/requests/api.py -> build/bdist.linux-x86_64/egg/requests copying build/lib/requests/auth.py -> build/bdist.linux-x86_64/egg/requests copying build/lib/requests/certs.py -> build/bdist.linux-x86_64/egg/requests copying build/lib/requests/compat.py -> build/bdist.linux-x86_64/egg/requests copying build/lib/requests/cookies.py -> build/bdist.linux-x86_64/egg/requests copying build/lib/requests/exceptions.py -> build/bdist.linux-x86_64/egg/requests copying build/lib/requests/hooks.py -> build/bdist.linux-x86_64/egg/requests copying build/lib/requests/models.py -> build/bdist.linux-x86_64/egg/requests copying build/lib/requests/sessions.py -> build/bdist.linux-x86_64/egg/requests copying build/lib/requests/status_codes.py -> build/bdist.linux-x86_64/egg/requests copying build/lib/requests/structures.py -> build/bdist.linux-x86_64/egg/requests copying build/lib/requests/utils.py -> build/bdist.linux-x86_64/egg/requests creating build/bdist.linux-x86_64/egg/requests/packages copying build/lib/requests/packages/init.py -> build/bdist.linux-x86_64/egg/requests/packages creating build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/init.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/big5freq.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/big5prober.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/chardistribution.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/charsetgroupprober.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/charsetprober.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/codingstatemachine.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/compat.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/cp949prober.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/enums.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/escprober.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/escsm.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/eucjpprober.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/euckrfreq.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/euckrprober.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/euctwfreq.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/euctwprober.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/gb2312freq.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/gb2312prober.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/hebrewprober.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/jisfreq.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/jpcntx.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/langbulgarianmodel.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/langcyrillicmodel.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/langgreekmodel.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/langhebrewmodel.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/langhungarianmodel.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/langthaimodel.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/langturkishmodel.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/latin1prober.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/mbcharsetprober.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/mbcsgroupprober.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/mbcssm.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/sbcharsetprober.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/sbcsgroupprober.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/sjisprober.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/universaldetector.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/utf8prober.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet copying build/lib/requests/packages/chardet/version.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet creating build/bdist.linux-x86_64/egg/requests/packages/chardet/cli copying build/lib/requests/packages/chardet/cli/init.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet/cli copying build/lib/requests/packages/chardet/cli/chardetect.py -> build/bdist.linux-x86_64/egg/requests/packages/chardet/cli creating build/bdist.linux-x86_64/egg/requests/packages/idna copying build/lib/requests/packages/idna/init.py -> build/bdist.linux-x86_64/egg/requests/packages/idna copying build/lib/requests/packages/idna/codec.py -> build/bdist.linux-x86_64/egg/requests/packages/idna copying build/lib/requests/packages/idna/compat.py -> build/bdist.linux-x86_64/egg/requests/packages/idna copying build/lib/requests/packages/idna/core.py -> build/bdist.linux-x86_64/egg/requests/packages/idna copying build/lib/requests/packages/idna/idnadata.py -> build/bdist.linux-x86_64/egg/requests/packages/idna copying build/lib/requests/packages/idna/intranges.py -> build/bdist.linux-x86_64/egg/requests/packages/idna copying build/lib/requests/packages/idna/uts46data.py -> build/bdist.linux-x86_64/egg/requests/packages/idna creating build/bdist.linux-x86_64/egg/requests/packages/urllib3 copying build/lib/requests/packages/urllib3/init.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3 copying build/lib/requests/packages/urllib3/_collections.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3 copying build/lib/requests/packages/urllib3/connection.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3 copying build/lib/requests/packages/urllib3/connectionpool.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3 copying build/lib/requests/packages/urllib3/exceptions.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3 copying build/lib/requests/packages/urllib3/fields.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3 copying build/lib/requests/packages/urllib3/filepost.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3 copying build/lib/requests/packages/urllib3/poolmanager.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3 copying build/lib/requests/packages/urllib3/request.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3 copying build/lib/requests/packages/urllib3/response.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3 creating build/bdist.linux-x86_64/egg/requests/packages/urllib3/packages copying build/lib/requests/packages/urllib3/packages/init.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3/packages copying build/lib/requests/packages/urllib3/packages/ordered_dict.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3/packages copying build/lib/requests/packages/urllib3/packages/six.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3/packages creating build/bdist.linux-x86_64/egg/requests/packages/urllib3/packages/ssl_match_hostname copying build/lib/requests/packages/urllib3/packages/ssl_match_hostname/init.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3/packages/ssl_match_hostname copying build/lib/requests/packages/urllib3/packages/ssl_match_hostname/_implementation.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3/packages/ssl_match_hostname creating build/bdist.linux-x86_64/egg/requests/packages/urllib3/packages/backports copying build/lib/requests/packages/urllib3/packages/backports/init.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3/packages/backports copying build/lib/requests/packages/urllib3/packages/backports/makefile.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3/packages/backports creating build/bdist.linux-x86_64/egg/requests/packages/urllib3/contrib copying build/lib/requests/packages/urllib3/contrib/init.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3/contrib copying build/lib/requests/packages/urllib3/contrib/appengine.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3/contrib copying build/lib/requests/packages/urllib3/contrib/ntlmpool.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3/contrib copying build/lib/requests/packages/urllib3/contrib/pyopenssl.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3/contrib copying build/lib/requests/packages/urllib3/contrib/securetransport.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3/contrib copying build/lib/requests/packages/urllib3/contrib/socks.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3/contrib creating build/bdist.linux-x86_64/egg/requests/packages/urllib3/contrib/_securetransport copying build/lib/requests/packages/urllib3/contrib/_securetransport/init.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3/contrib/_securetransport copying build/lib/requests/packages/urllib3/contrib/_securetransport/bindings.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3/contrib/_securetransport copying build/lib/requests/packages/urllib3/contrib/_securetransport/low_level.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3/contrib/_securetransport creating build/bdist.linux-x86_64/egg/requests/packages/urllib3/util copying build/lib/requests/packages/urllib3/util/init.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3/util copying build/lib/requests/packages/urllib3/util/connection.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3/util copying build/lib/requests/packages/urllib3/util/request.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3/util copying build/lib/requests/packages/urllib3/util/response.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3/util copying build/lib/requests/packages/urllib3/util/retry.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3/util copying build/lib/requests/packages/urllib3/util/selectors.py -> build/bdist.linux-x8664/egg/requests/packages/urllib3/util copying build/lib/requests/packages/urllib3/util/ssl.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3/util copying build/lib/requests/packages/urllib3/util/timeout.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3/util copying build/lib/requests/packages/urllib3/util/url.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3/util copying build/lib/requests/packages/urllib3/util/wait.py -> build/bdist.linux-x86_64/egg/requests/packages/urllib3/util copying build/lib/requests/cacert.pem -> build/bdist.linux-x86_64/egg/requests byte-compiling build/bdist.linux-x86_64/egg/requests/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/_internal_utils.py to _internal_utils.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/adapters.py to adapters.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/api.py to api.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/auth.py to auth.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/certs.py to certs.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/compat.py to compat.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/cookies.py to cookies.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/exceptions.py to exceptions.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/hooks.py to hooks.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/models.py to models.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/sessions.py to sessions.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/status_codes.py to status_codes.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/structures.py to structures.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/utils.py to utils.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/big5freq.py to big5freq.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/big5prober.py to big5prober.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/chardistribution.py to chardistribution.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/charsetgroupprober.py to charsetgroupprober.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/charsetprober.py to charsetprober.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/codingstatemachine.py to codingstatemachine.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/compat.py to compat.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/cp949prober.py to cp949prober.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/enums.py to enums.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/escprober.py to escprober.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/escsm.py to escsm.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/eucjpprober.py to eucjpprober.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/euckrfreq.py to euckrfreq.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/euckrprober.py to euckrprober.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/euctwfreq.py to euctwfreq.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/euctwprober.py to euctwprober.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/gb2312freq.py to gb2312freq.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/gb2312prober.py to gb2312prober.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/hebrewprober.py to hebrewprober.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/jisfreq.py to jisfreq.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/jpcntx.py to jpcntx.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/langbulgarianmodel.py to langbulgarianmodel.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/langcyrillicmodel.py to langcyrillicmodel.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/langgreekmodel.py to langgreekmodel.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/langhebrewmodel.py to langhebrewmodel.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/langhungarianmodel.py to langhungarianmodel.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/langthaimodel.py to langthaimodel.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/langturkishmodel.py to langturkishmodel.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/latin1prober.py to latin1prober.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/mbcharsetprober.py to mbcharsetprober.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/mbcsgroupprober.py to mbcsgroupprober.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/mbcssm.py to mbcssm.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/sbcharsetprober.py to sbcharsetprober.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/sbcsgroupprober.py to sbcsgroupprober.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/sjisprober.py to sjisprober.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/universaldetector.py to universaldetector.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/utf8prober.py to utf8prober.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/version.py to version.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/cli/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/chardet/cli/chardetect.py to chardetect.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/idna/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/idna/codec.py to codec.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/idna/compat.py to compat.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/idna/core.py to core.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/idna/idnadata.py to idnadata.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/idna/intranges.py to intranges.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/idna/uts46data.py to uts46data.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/_collections.py to _collections.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/connection.py to connection.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/connectionpool.py to connectionpool.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/exceptions.py to exceptions.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/fields.py to fields.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/filepost.py to filepost.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/poolmanager.py to poolmanager.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/request.py to request.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/response.py to response.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/packages/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/packages/ordered_dict.py to ordered_dict.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/packages/six.py to six.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/packages/ssl_match_hostname/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/packages/ssl_match_hostname/_implementation.py to _implementation.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/packages/backports/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/packages/backports/makefile.py to makefile.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/contrib/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/contrib/appengine.py to appengine.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/contrib/ntlmpool.py to ntlmpool.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/contrib/pyopenssl.py to pyopenssl.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/contrib/securetransport.py to securetransport.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/contrib/socks.py to socks.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/contrib/_securetransport/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/contrib/_securetransport/bindings.py to bindings.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/contrib/_securetransport/low_level.py to low_level.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/util/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/util/connection.py to connection.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/util/request.py to request.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/util/response.py to response.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/util/retry.py to retry.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/util/selectors.py to selectors.pyc byte-compiling build/bdist.linux-x8664/egg/requests/packages/urllib3/util/ssl.py to ssl_.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/util/timeout.py to timeout.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/util/url.py to url.pyc byte-compiling build/bdist.linux-x86_64/egg/requests/packages/urllib3/util/wait.py to wait.pyc creating build/bdist.linux-x86_64/egg/EGG-INFO copying requests.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO copying requests.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying requests.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying requests.egg-info/not-zip-safe -> build/bdist.linux-x86_64/egg/EGG-INFO copying requests.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying requests.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO creating dist creating 'dist/requests-2.14.2-py2.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it removing 'build/bdist.linux-x86_64/egg' (and everything under it) Processing requests-2.14.2-py2.7.egg creating /usr/lib/python2.7/site-packages/requests-2.14.2-py2.7.egg Extracting requests-2.14.2-py2.7.egg to /usr/lib/python2.7/site-packages Adding requests 2.14.2 to easy-install.pth file

Installed /usr/lib/python2.7/site-packages/requests-2.14.2-py2.7.egg Processing dependencies for requests==2.14.2 Finished processing dependencies for requests==2.14.2 [10/May/2017 19:34:29] INFO [root:524] Cleaning up python packages [10/May/2017 19:34:29] INFO [root:527] Flush VNeST data [10/May/2017 19:34:29] INFO [root:530] Stop container ecsstandalone [10/May/2017 19:34:39] INFO [root:533] Start container ecsstandalone [10/May/2017 19:34:39] INFO [root:536] Clean up local files [10/May/2017 19:34:39] INFO [root:551] Waiting on Authentication Service. This may take several minutes. Executing getAuthToken: curl -i -k https://192.168.1.90:4443/login -u root:ChangeMe % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to 192.168.1.90:4443; Connection refused [10/May/2017 19:35:09] INFO [root:564] Problem reaching authentication server. Retrying shortly. Executing getAuthToken: curl -i -k https://192.168.1.90:4443/login -u root:ChangeMe % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to 192.168.1.90:4443; Connection refused [10/May/2017 19:35:39] INFO [root:564] Problem reaching authentication server. Retrying shortly. Executing getAuthToken: curl -i -k https://192.168.1.90:4443/login -u root:ChangeMe % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to 192.168.1.90:4443; Connection refused [10/May/2017 19:36:09] INFO [root:564] Problem reaching authentication server. Retrying shortly.

Ports view

[centos@ecstest ecs-single-node]$ sudo netstat -anp | grep LISTEN | grep 90 tcp 0 0 192.168.1.90:9099 0.0.0.0: LISTEN 8437/storageserver tcp 0 0 192.168.1.90:9069 0.0.0.0: LISTEN 8437/storageserver tcp 0 0 192.168.1.90:80 0.0.0.0: LISTEN 10588/nginx: master tcp 0 0 192.168.1.90:4443 0.0.0.0: LISTEN 10588/nginx: master tcp 0 0 192.168.1.90:443 0.0.0.0: LISTEN 10588/nginx: master tcp6 0 0 192.168.1.90:9096 ::: LISTEN 8540/georeceiver tcp6 0 0 192.168.1.90:1096 ::: LISTEN 8405/rm tcp6 0 0 192.168.1.90:9097 ::: LISTEN 8540/georeceiver tcp6 0 0 192.168.1.90:9898 ::: LISTEN 8498/provisionsvc tcp6 0 0 192.168.1.90:1098 ::: LISTEN 8487/blobsvc tcp6 0 0 192.168.1.90:9098 ::: LISTEN 8540/georeceiver tcp6 0 0 192.168.1.90:9100 ::: LISTEN 8574/dtquery tcp6 0 0 192.168.1.90:9101 ::: LISTEN 8574/dtquery tcp6 0 0 192.168.1.90:1198 ::: LISTEN 8498/provisionsvc tcp6 0 0 192.168.1.90:10000 ::: LISTEN 8479/dataheadsvc tcp6 0 0 192.168.1.90:9040 ::: LISTEN 8479/dataheadsvc tcp6 0 0 192.168.1.90:9201 ::: LISTEN 8406/stat tcp6 0 0 192.168.1.90:3218 ::: LISTEN 8479/dataheadsvc tcp6 0 0 127.0.0.1:9010 ::: LISTEN 8440/objcontrolsvc tcp6 0 0 192.168.1.90:1298 ::: LISTEN 8479/dataheadsvc tcp6 0 0 127.0.0.1:9011 ::: LISTEN 8440/objcontrolsvc tcp6 0 0 192.168.1.90:9203 ::: LISTEN 8451/metering tcp6 0 0 192.168.1.90:9204 ::: LISTEN 8403/vnest tcp6 0 0 192.168.1.90:9205 ::: LISTEN 8403/vnest tcp6 0 0 192.168.1.90:9209 ::: LISTEN 8551/eventsvc tcp6 0 0 192.168.1.90:9211 ::: LISTEN 8878/authsvc tcp6 0 0 127.0.0.1:64443 ::: LISTEN 8590/ecsportalsvc tcp6 0 0 192.168.1.90:9020 ::: LISTEN 8479/dataheadsvc tcp6 0 0 192.168.1.90:9212 ::: LISTEN 8440/objcontrolsvc tcp6 0 0 192.168.1.90:9021 ::: LISTEN 8479/dataheadsvc tcp6 0 0 192.168.1.90:9022 ::: LISTEN 8479/dataheadsvc tcp6 0 0 192.168.1.90:9023 ::: LISTEN 8479/dataheadsvc tcp6 0 0 192.168.1.90:9024 ::: LISTEN 8479/dataheadsvc tcp6 0 0 192.168.1.90:9888 ::: LISTEN 8504/resourcesvc tcp6 0 0 192.168.1.90:2049 ::: LISTEN 8479/dataheadsvc tcp6 0 0 192.168.1.90:9025 ::: LISTEN 8479/dataheadsvc tcp6 0 0 192.168.1.90:9091 ::: LISTEN 8410/cm tcp6 0 0 192.168.1.90:9028 ::: LISTEN 8487/blobsvc tcp6 0 0 192.168.1.90:9029 ::: LISTEN 8487/blobsvc tcp6 0 0 192.168.1.90:2181 ::: LISTEN 8399/coordinatorsvc tcp6 0 0 192.168.1.90:9094 ::: LISTEN 8540/georeceiver tcp6 0 0 192.168.1.90:9095 ::: LISTEN 8540/georeceiver tcp6 0 0 192.168.1.90:1095 :::* LISTEN 8461/ssm

Disk View

find /ecs/ /ecs/ /ecs/uuid-3c40926e-aed0-439a-98d5-eef5c45c296d /ecs/uuid-6eee34f0-1b83-4881-8690-4837af07c566 /ecs/uuid-6eee34f0-1b83-4881-8690-4837af07c566/0000 /ecs/uuid-6eee34f0-1b83-4881-8690-4837af07c566/0001 /ecs/uuid-6eee34f0-1b83-4881-8690-4837af07c566/0002 /ecs/uuid-6eee34f0-1b83-4881-8690-4837af07c566/0003 /ecs/uuid-6eee34f0-1b83-4881-8690-4837af07c566/0004 /ecs/uuid-6eee34f0-1b83-4881-8690-4837af07c566/0005 /ecs/uuid-6eee34f0-1b83-4881-8690-4837af07c566/0006 /ecs/uuid-6eee34f0-1b83-4881-8690-4837af07c566/0007 /ecs/uuid-6eee34f0-1b83-4881-8690-4837af07c566/0008 /ecs/uuid-6eee34f0-1b83-4881-8690-4837af07c566/0009 /ecs/uuid-6eee34f0-1b83-4881-8690-4837af07c566/0010 /ecs/uuid-6eee34f0-1b83-4881-8690-4837af07c566/0011 /ecs/uuid-6eee34f0-1b83-4881-8690-4837af07c566/0012 /ecs/uuid-6eee34f0-1b83-4881-8690-4837af07c566/0013 /ecs/uuid-6eee34f0-1b83-4881-8690-4837af07c566/0014 /ecs/uuid-6eee34f0-1b83-4881-8690-4837af07c566/0015 /ecs/uuid-6eee34f0-1b83-4881-8690-4837af07c566/0016 /ecs/uuid-6eee34f0-1b83-4881-8690-4837af07c566/0017 /ecs/uuid-6eee34f0-1b83-4881-8690-4837af07c566/0018

Error Logs view

[centos@ecstest emcvipr-object]$ cd /var/log/vipr/emcvipr-object/ [centos@ecstest emcvipr-object]$ sudo tail -f * | grep -i error tail: error reading ‘audit’: Is a directory tail: audit: cannot follow end of this type of file; giving up on this name ==> blobsvc-error.log <== tail: error reading ‘cores’: Is a directory tail: cores: cannot follow end of this type of file; giving up on this name 2017-05-10T20:06:17,970 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd383d:5:2a2, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd383d:5:2a2' 2017-05-10T20:06:22,972 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd383d:5:2a4, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd383d:5:2a4' 2017-05-10T20:06:27,973 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd383d:5:2a6, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd383d:5:2a6' 2017-05-10T20:06:32,975 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd383d:5:2a8, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd383d:5:2a8' 2017-05-10T20:06:37,976 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd383d:5:2aa, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd383d:5:2aa' 2017-05-10T20:06:42,978 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd383d:5:2ac, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd383d:5:2ac' 2017-05-10T20:06:47,980 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd383d:5:2ae, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd383d:5:2ae' 2017-05-10T20:06:52,982 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd383d:5:2b0, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd383d:5:2b0' 2017-05-10T20:06:57,984 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd383d:5:2b2, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd383d:5:2b2' 2017-05-10T20:07:02,985 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd383d:5:2b4, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd383d:5:2b4' 2017-05-10T20:06:57,984 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd383d:5:2b2, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd383d:5:2b2' 2017-05-10T20:07:02,985 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd383d:5:2b4, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd383d:5:2b4' ==> btree-gc-error.log <== 2017-05-10T19:45:02,821 [main] ERROR BTreeGC.java (line 219) No root record found. Exiting 2017-05-10T19:45:02,821 [main] ERROR BTreeGC.java (line 219) No root record found. Exiting ==> cm-error.log <== 2017-05-10T20:06:16,061 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd324e:6:29e, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd324e:6:29e' 2017-05-10T20:06:21,063 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd324e:6:2a0, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd324e:6:2a0' 2017-05-10T20:06:26,064 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd324e:6:2a2, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd324e:6:2a2' 2017-05-10T20:06:31,066 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd324e:6:2a4, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd324e:6:2a4' 2017-05-10T20:06:36,067 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd324e:6:2a6, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd324e:6:2a6' 2017-05-10T20:06:41,068 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd324e:6:2a8, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd324e:6:2a8' 2017-05-10T20:06:46,070 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd324e:6:2aa, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd324e:6:2aa' 2017-05-10T20:06:51,071 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd324e:6:2ac, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd324e:6:2ac' 2017-05-10T20:06:56,073 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd324e:6:2ae, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd324e:6:2ae' 2017-05-10T20:07:01,074 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd324e:6:2b0, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd324e:6:2b0' ==> dataheadsvc-error.log <== 2017-05-10T20:06:18,776 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3e60:4:2a4, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3e60:4:2a4' 2017-05-10T20:06:23,778 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3e60:4:2a6, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3e60:4:2a6' 2017-05-10T20:06:28,779 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3e60:4:2a8, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3e60:4:2a8' 2017-05-10T20:06:33,780 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3e60:4:2aa, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3e60:4:2aa' 2017-05-10T20:06:38,782 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3e60:4:2ac, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3e60:4:2ac' 2017-05-10T20:06:43,783 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3e60:4:2ae, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3e60:4:2ae' 2017-05-10T20:06:48,785 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3e60:4:2b0, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3e60:4:2b0' 2017-05-10T20:06:53,787 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3e60:4:2b2, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3e60:4:2b2' 2017-05-10T20:06:58,788 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3e60:4:2b4, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3e60:4:2b4' 2017-05-10T20:07:03,789 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3e60:4:2b6, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3e60:4:2b6' 2017-05-10T20:07:03,789 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3e60:4:2b6, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3e60:4:2b6' ==> dtquery-error.log <== 2017-05-10T20:06:31,417 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3983:9:2a2, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3983:9:2a2' 2017-05-10T20:06:36,419 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3983:9:2a4, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3983:9:2a4' 2017-05-10T20:06:41,420 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3983:9:2a6, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3983:9:2a6' 2017-05-10T20:06:46,422 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3983:9:2a8, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3983:9:2a8' 2017-05-10T20:06:51,424 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3983:9:2aa, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3983:9:2aa' 2017-05-10T20:06:56,425 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3983:9:2ac, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3983:9:2ac' 2017-05-10T20:07:01,427 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3983:9:2ae, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3983:9:2ae' 2017-05-10T20:06:58,658 [RequestStats1] INFO RequestStatTracker.java (line 147) # of 500 and 503 errors in the last hour: 0 2017-05-10T20:07:01,427 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3983:9:2ae, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3983:9:2ae' May 10, 2017 7:37:29 PM com.sun.jersey.spi.inject.Errors processErrorMessages 2017-05-10T19:42:16,815 [ZoneScanner-000] ERROR DirectoryOwnershipRecordMap.java (line 201) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:957babf1-e4cc-4d33-8488-40f196a4ac14, type: RT, keyHash: 3724, level: 0] 2017-05-10T19:42:22,821 [ZoneScanner-000] ERROR DirectoryOwnershipRecordMap.java (line 201) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:957babf1-e4cc-4d33-8488-40f196a4ac14, type: RT, keyHash: 3724, level: 0] 2017-05-10T19:42:31,826 [ZoneScanner-000] ERROR DirectoryOwnershipRecordMap.java (line 201) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:957babf1-e4cc-4d33-8488-40f196a4ac14, type: RT, keyHash: 3724, level: 0] 2017-05-10T19:42:43,829 [ZoneScanner-000] ERROR DirectoryOwnershipRecordMap.java (line 201) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:957babf1-e4cc-4d33-8488-40f196a4ac14, type: RT, keyHash: 3724, level: 0] 2017-05-10T19:42:58,832 [ZoneScanner-000] ERROR DirectoryOwnershipRecordMap.java (line 201) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:957babf1-e4cc-4d33-8488-40f196a4ac14, type: RT, keyHash: 3724, level: 0] 2017-05-10T19:43:16,835 [ZoneScanner-000] ERROR DirectoryOwnershipRecordMap.java (line 201) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:957babf1-e4cc-4d33-8488-40f196a4ac14, type: RT, keyHash: 3724, level: 0] 2017-05-10T19:43:37,838 [ZoneScanner-000] ERROR DirectoryOwnershipRecordMap.java (line 201) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:957babf1-e4cc-4d33-8488-40f196a4ac14, type: RT, keyHash: 3724, level: 0] 2017-05-10T19:44:01,842 [ZoneScanner-000] ERROR DirectoryOwnershipRecordMap.java (line 201) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:957babf1-e4cc-4d33-8488-40f196a4ac14, type: RT, keyHash: 3724, level: 0] 2017-05-10T19:44:28,847 [ZoneScanner-000] ERROR DirectoryOwnershipRecordMap.java (line 201) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:957babf1-e4cc-4d33-8488-40f196a4ac14, type: RT, keyHash: 3724, level: 0] ==> eventsvc-error.log <== 2017-05-10T20:06:50,433 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd36e5:5:2b0, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd36e5:5:2b0' 2017-05-10T20:06:55,435 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd36e5:5:2b2, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd36e5:5:2b2' 2017-05-10T20:07:00,437 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd36e5:5:2b4, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd36e5:5:2b4' 2017-05-10T20:07:05,439 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd36e5:5:2b6, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd36e5:5:2b6' 2017-05-10T20:07:05,439 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd36e5:5:2b6, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd36e5:5:2b6' mount: /.volumes/cdrom1: mount failed: Unknown error -1 _gen_hostname /etc/HOSTNAME: Error 1 _gen_ssh_user_key /home/svcuser/.ssh/id_dsa: Error 1 _gen_ssh_authkeys2 /home/svcuser/.ssh/authorized_keys2: Error 1 _gen_firewall_conf /etc/sysconfig: Error 2 ==> georeceiver-error.log <== 2017-05-10T20:06:20,016 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd2d19:5:2b7, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd2d19:5:2b7' 2017-05-10T20:06:25,018 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd2d19:5:2b9, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd2d19:5:2b9' 2017-05-10T20:06:30,019 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd2d19:5:2bb, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd2d19:5:2bb' 2017-05-10T20:06:35,021 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd2d19:5:2bd, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd2d19:5:2bd' 2017-05-10T20:06:40,022 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd2d19:5:2bf, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd2d19:5:2bf' 2017-05-10T20:06:45,024 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd2d19:5:2c1, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd2d19:5:2c1' 2017-05-10T20:06:50,025 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd2d19:5:2c3, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd2d19:5:2c3' 2017-05-10T20:06:55,027 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd2d19:5:2c5, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd2d19:5:2c5' 2017-05-10T20:07:00,029 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd2d19:5:2c7, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd2d19:5:2c7' 2017-05-10T20:07:05,030 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd2d19:5:2c9, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd2d19:5:2c9' 2017-05-10T20:07:00,029 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd2d19:5:2c7, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd2d19:5:2c7' 2017-05-10T20:07:05,030 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd2d19:5:2c9, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd2d19:5:2c9' ==> metering-error.log <== 2017-05-10T20:06:17,075 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3f04:5:2a2, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3f04:5:2a2' 2017-05-10T20:06:22,077 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3f04:5:2a4, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3f04:5:2a4' 2017-05-10T20:06:27,078 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3f04:5:2a6, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3f04:5:2a6' 2017-05-10T20:06:32,080 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3f04:5:2a8, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3f04:5:2a8' 2017-05-10T20:06:37,081 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3f04:5:2aa, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3f04:5:2aa' 2017-05-10T20:06:42,082 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3f04:5:2ac, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3f04:5:2ac' 2017-05-10T20:06:47,085 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3f04:5:2ae, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3f04:5:2ae' 2017-05-10T20:06:52,086 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3f04:5:2b0, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3f04:5:2b0' 2017-05-10T20:06:57,088 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3f04:5:2b2, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3f04:5:2b2' 2017-05-10T20:07:02,090 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3f04:5:2b4, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3f04:5:2b4' 2017-05-10T20:07:02,090 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3f04:5:2b4, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3f04:5:2b4' ==> nginx_error.log <== ==> provisionsvc-error.log <== 2017-05-10T20:06:24,612 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd17ec:5:2aa, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd17ec:5:2aa' 2017-05-10T20:06:29,613 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd17ec:5:2ac, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd17ec:5:2ac' 2017-05-10T20:06:34,615 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd17ec:5:2ae, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd17ec:5:2ae' 2017-05-10T20:06:39,617 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd17ec:5:2b0, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd17ec:5:2b0' 2017-05-10T20:06:44,618 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd17ec:5:2b2, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd17ec:5:2b2' 2017-05-10T20:06:46,220 [pool-52-thread-6-c0a8015a:15bf3dd211f:e9:0] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd17ec:18:58, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd17ec:18:58' 2017-05-10T20:06:49,620 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd17ec:5:2b4, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd17ec:5:2b4' 2017-05-10T20:06:54,621 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd17ec:5:2b6, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd17ec:5:2b6' 2017-05-10T20:06:59,622 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd17ec:5:2b8, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd17ec:5:2b8' 2017-05-10T20:07:04,624 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd17ec:5:2ba, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd17ec:5:2ba' 2017-05-10T20:06:59,622 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd17ec:5:2b8, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd17ec:5:2b8' 2017-05-10T20:07:04,624 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd17ec:5:2ba, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd17ec:5:2ba' ==> resourcesvc-error.log <== 2017-05-10T20:07:03,789 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-005] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd3e60:4:2b6 failed due to 2017-05-10T20:07:04,623 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-006] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14__RT_12_32_0: with request id c0a8015a:15bf3dd17ec:5:2ba failed due to 2017-05-10T20:07:05,030 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-007] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd2d19:5:2c9 failed due to 2017-05-10T20:07:05,388 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-136] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd3254:5:2b4 failed due to 2017-05-10T20:07:05,438 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-137] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14__RT_12_32_0: with request id c0a8015a:15bf3dd36e5:5:2b6 failed due to 2017-05-10T20:07:04,623 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-006] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd17ec:5:2ba failed due to 2017-05-10T20:07:05,030 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-007] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd2d19:5:2c9 failed due to 2017-05-10T20:07:05,388 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-136] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14__RT_12_32_0: with request id c0a8015a:15bf3dd3254:5:2b4 failed due to 2017-05-10T20:07:05,438 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-137] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd36e5:5:2b6 failed due to ==> rm-error.log <== 2017-05-10T20:06:20,375 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3254:5:2a2, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3254:5:2a2' 2017-05-10T20:06:25,377 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3254:5:2a4, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3254:5:2a4' 2017-05-10T20:06:30,378 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3254:5:2a6, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3254:5:2a6' 2017-05-10T20:06:35,380 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3254:5:2a8, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3254:5:2a8' 2017-05-10T20:06:40,381 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3254:5:2aa, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3254:5:2aa' 2017-05-10T20:06:45,382 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3254:5:2ac, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3254:5:2ac' 2017-05-10T20:06:50,384 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3254:5:2ae, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3254:5:2ae' 2017-05-10T20:06:55,385 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3254:5:2b0, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3254:5:2b0' 2017-05-10T20:07:00,386 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3254:5:2b2, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3254:5:2b2' 2017-05-10T20:07:05,388 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3254:5:2b4, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3254:5:2b4' 2017-05-10T20:07:05,388 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3254:5:2b4, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3254:5:2b4' ==> ssm-error.log <== 2017-05-10T20:06:17,356 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd22fb:5:2a2, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd22fb:5:2a2' 2017-05-10T20:06:22,357 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd22fb:5:2a4, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd22fb:5:2a4' 2017-05-10T20:06:27,359 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd22fb:5:2a6, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd22fb:5:2a6' 2017-05-10T20:06:32,360 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd22fb:5:2a8, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd22fb:5:2a8' 2017-05-10T20:06:37,362 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd22fb:5:2aa, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd22fb:5:2aa' 2017-05-10T20:06:42,363 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd22fb:5:2ac, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd22fb:5:2ac' 2017-05-10T20:06:47,367 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd22fb:5:2ae, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd22fb:5:2ae' 2017-05-10T20:06:52,368 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd22fb:5:2b0, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd22fb:5:2b0' 2017-05-10T20:06:57,369 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd22fb:5:2b2, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd22fb:5:2b2' 2017-05-10T20:07:02,371 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd22fb:5:2b4, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd22fb:5:2b4' 2017-05-10T20:07:02,371 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd22fb:5:2b4, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd22fb:5:2b4' ==> stat-error.log <== log4j:WARN Continuable parsing error 22 and column 23 ==> vnest-error.log <== 2017-05-10T19:37:12,647 [ReceiverSocketThread-192.168.1.90:37522-5] ERROR RemoteDataNodeCommunicator.java (line 1044) socket for 192.168.1.90 broken 2017-05-10T19:37:32,067 [pool-28-thread-1] ERROR KeepAliveMessageSender.java (line 203) Caught IOException (Is the node down?) trying to reset existing connection to 192.168.1.90:9230 2017-05-10T19:37:57,101 [pool-28-thread-1] ERROR KeepAliveMessageSender.java (line 203) Caught IOException (Is the node down?) trying to reset existing connection to 192.168.1.90:9230 2017-05-10T19:38:22,124 [pool-28-thread-1] ERROR KeepAliveMessageSender.java (line 203) Caught IOException (Is the node down?) trying to reset existing connection to 192.168.1.90:9230 2017-05-10T19:38:47,161 [pool-28-thread-1] ERROR KeepAliveMessageSender.java (line 203) Caught IOException (Is the node down?) trying to reset existing connection to 192.168.1.90:9230 2017-05-10T19:39:12,184 [pool-28-thread-1] ERROR KeepAliveMessageSender.java (line 203) Caught IOException (Is the node down?) trying to reset existing connection to 192.168.1.90:9230 2017-05-10T19:39:26,124 [SenderSocketThread-192.168.1.90:9230-5-000] ERROR CommunicatorSenderSocket.java (line 473) reInitializeSocketConnection - threw Interrupted exception for remote 192.168.1.90 port 9230 2017-05-10T19:39:37,210 [pool-28-thread-1] ERROR KeepAliveMessageSender.java (line 203) Caught IOException (Is the node down?) trying to reset existing connection to 192.168.1.90:9230 ==> cm-error.log <== 2017-05-10T20:07:06,076 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd324e:6:2b2, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd324e:6:2b2' 2017-05-10T20:07:06,076 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd324e:6:2b2, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd324e:6:2b2' ==> dtquery-error.log <== 2017-05-10T20:07:06,429 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3983:9:2b0, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3983:9:2b0' 2017-05-10T20:07:06,429 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3983:9:2b0, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3983:9:2b0' ==> resourcesvc-error.log <== 2017-05-10T20:07:06,075 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-202] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd324e:6:2b2 failed due to 2017-05-10T20:07:06,428 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-203] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14__RT_12_32_0: with request id c0a8015a:15bf3dd3983:9:2b0 failed due to 2017-05-10T20:07:06,075 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-202] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd324e:6:2b2 failed due to 2017-05-10T20:07:06,428 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-203] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd3983:9:2b0 failed due to ==> metering-error.log <== 2017-05-10T20:07:07,091 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3f04:5:2b6, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3f04:5:2b6' 2017-05-10T20:07:07,091 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3f04:5:2b6, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3f04:5:2b6' ==> resourcesvc-error.log <== 2017-05-10T20:07:06,892 [ZoneScanner-000] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd3a34:6:1ab failed due to 2017-05-10T20:07:06,893 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3a34:6:1ab, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3a34:6:1ab' 2017-05-10T20:07:07,091 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-043] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd3f04:5:2b6 failed due to 2017-05-10T20:07:07,372 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-044] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd22fb:5:2b6 failed due to 2017-05-10T20:07:06,892 [ZoneScanner-000] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd3a34:6:1ab failed due to 2017-05-10T20:07:06,893 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3a34:6:1ab, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3a34:6:1ab' 2017-05-10T20:07:07,091 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-043] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd3f04:5:2b6 failed due to 2017-05-10T20:07:07,372 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-044] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd22fb:5:2b6 failed due to ==> ssm-error.log <== 2017-05-10T20:07:07,372 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd22fb:5:2b6, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd22fb:5:2b6' 2017-05-10T20:07:07,372 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd22fb:5:2b6, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd22fb:5:2b6' ==> blobsvc-error.log <== 2017-05-10T20:07:07,989 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd383d:5:2b6, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd383d:5:2b6' 2017-05-10T20:07:07,989 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd383d:5:2b6, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd383d:5:2b6' ==> resourcesvc-error.log <== 2017-05-10T20:07:07,988 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-172] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14__RT_12_32_0: with request id c0a8015a:15bf3dd383d:5:2b6 failed due to 2017-05-10T20:07:07,988 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-172] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd383d:5:2b6 failed due to ==> dataheadsvc-error.log <== 2017-05-10T20:07:08,791 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3e60:4:2b8, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3e60:4:2b8' 2017-05-10T20:07:08,791 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3e60:4:2b8, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3e60:4:2b8' ==> provisionsvc-error.log <== 2017-05-10T20:07:09,625 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd17ec:5:2bc, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd17ec:5:2bc' 2017-05-10T20:07:09,625 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd17ec:5:2bc, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd17ec:5:2bc' ==> resourcesvc-error.log <== 2017-05-10T20:07:08,790 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-174] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd3e60:4:2b8 failed due to 2017-05-10T20:07:09,625 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-078] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14__RT_12_32_0: with request id c0a8015a:15bf3dd17ec:5:2bc failed due to 2017-05-10T20:07:08,790 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-174] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd3e60:4:2b8 failed due to 2017-05-10T20:07:09,625 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-078] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd17ec:5:2bc failed due to ==> eventsvc-error.log <== 2017-05-10T20:07:10,440 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd36e5:5:2b8, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd36e5:5:2b8' 2017-05-10T20:07:10,440 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd36e5:5:2b8, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd36e5:5:2b8' ==> georeceiver-error.log <== 2017-05-10T20:07:10,032 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd2d19:5:2cb, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd2d19:5:2cb' 2017-05-10T20:07:10,032 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd2d19:5:2cb, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd2d19:5:2cb' ==> resourcesvc-error.log <== 2017-05-10T20:07:10,031 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-142] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14__RT_12_32_0: with request id c0a8015a:15bf3dd2d19:5:2cb failed due to 2017-05-10T20:07:10,389 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-144] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd3254:5:2b6 failed due to 2017-05-10T20:07:10,440 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-145] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd36e5:5:2b8 failed due to 2017-05-10T20:07:10,031 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-142] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14__RT_12_32_0: with request id c0a8015a:15bf3dd2d19:5:2cb failed due to 2017-05-10T20:07:10,389 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-144] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd3254:5:2b6 failed due to 2017-05-10T20:07:10,440 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-145] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd36e5:5:2b8 failed due to ==> rm-error.log <== 2017-05-10T20:07:10,390 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3254:5:2b6, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3254:5:2b6' 2017-05-10T20:07:10,390 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3254:5:2b6, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3254:5:2b6' 2017-05-10T20:07:11,056 [RequestStats1] INFO RequestStatTracker.java (line 147) # of 500 and 503 errors in the last hour: 0 ==> cm-error.log <== 2017-05-10T20:07:11,077 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd324e:6:2b4, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd324e:6:2b4' 2017-05-10T20:07:11,077 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd324e:6:2b4, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd324e:6:2b4' ==> dtquery-error.log <== 2017-05-10T20:07:11,430 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3983:9:2b2, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3983:9:2b2' 2017-05-10T20:07:11,430 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3983:9:2b2, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3983:9:2b2' ==> resourcesvc-error.log <== 2017-05-10T20:07:11,077 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-147] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14__RT_12_32_0: with request id c0a8015a:15bf3dd324e:6:2b4 failed due to 2017-05-10T20:07:11,430 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-210] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd3983:9:2b2 failed due to 2017-05-10T20:07:11,077 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-147] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd324e:6:2b4 failed due to 2017-05-10T20:07:11,430 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-210] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14__RT_12_32_0: with request id c0a8015a:15bf3dd3983:9:2b2 failed due to ==> metering-error.log <== 2017-05-10T20:07:12,093 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3f04:5:2b8, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3f04:5:2b8' 2017-05-10T20:07:12,093 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3f04:5:2b8, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3f04:5:2b8' ==> resourcesvc-error.log <== 2017-05-10T20:07:11,893 [ZoneScanner-000] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd3a34:6:1ac failed due to 2017-05-10T20:07:11,893 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3a34:6:1ac, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3a34:6:1ac' 2017-05-10T20:07:12,092 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-114] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd3f04:5:2b8 failed due to 2017-05-10T20:07:12,373 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-116] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14__RT_12_32_0: with request id c0a8015a:15bf3dd22fb:5:2b8 failed due to 2017-05-10T20:07:11,893 [ZoneScanner-000] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd3a34:6:1ac failed due to 2017-05-10T20:07:11,893 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3a34:6:1ac, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3a34:6:1ac' 2017-05-10T20:07:12,092 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-114] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd3f04:5:2b8 failed due to 2017-05-10T20:07:12,373 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-116] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14__RT_12_32_0: with request id c0a8015a:15bf3dd22fb:5:2b8 failed due to ==> ssm-error.log <== 2017-05-10T20:07:12,374 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd22fb:5:2b8, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd22fb:5:2b8' 2017-05-10T20:07:12,374 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd22fb:5:2b8, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd22fb:5:2b8' ==> blobsvc-error.log <== 2017-05-10T20:07:12,990 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd383d:5:2b8, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd383d:5:2b8' 2017-05-10T20:07:12,990 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd383d:5:2b8, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd383d:5:2b8' ==> resourcesvc-error.log <== 2017-05-10T20:07:12,990 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-117] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd383d:5:2b8 failed due to 2017-05-10T20:07:12,990 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-117] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd383d:5:2b8 failed due to ==> dataheadsvc-error.log <== 2017-05-10T20:07:13,792 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3e60:4:2ba, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3e60:4:2ba' 2017-05-10T20:07:13,792 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3e60:4:2ba, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3e60:4:2ba' ==> provisionsvc-error.log <== 2017-05-10T20:07:14,626 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd17ec:5:2be, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd17ec:5:2be' 2017-05-10T20:07:14,626 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd17ec:5:2be, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd17ec:5:2be' ==> resourcesvc-error.log <== 2017-05-10T20:07:13,792 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-085] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14__RT_12_32_0: with request id c0a8015a:15bf3dd3e60:4:2ba failed due to 2017-05-10T20:07:14,626 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-086] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd17ec:5:2be failed due to 2017-05-10T20:07:13,792 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-085] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd3e60:4:2ba failed due to 2017-05-10T20:07:14,626 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-086] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14__RT_12_32_0: with request id c0a8015a:15bf3dd17ec:5:2be failed due to ==> eventsvc-error.log <== 2017-05-10T20:07:15,442 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd36e5:5:2ba, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd36e5:5:2ba' 2017-05-10T20:07:15,442 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd36e5:5:2ba, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd36e5:5:2ba' ==> georeceiver-error.log <== 2017-05-10T20:07:15,034 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd2d19:5:2cd, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd2d19:5:2cd' 2017-05-10T20:07:15,034 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd2d19:5:2cd, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd2d19:5:2cd' ==> resourcesvc-error.log <== 2017-05-10T20:07:15,033 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-088] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd2d19:5:2cd failed due to 2017-05-10T20:07:15,391 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-153] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd3254:5:2b8 failed due to 2017-05-10T20:07:15,441 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-154] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14__RT_12_32_0: with request id c0a8015a:15bf3dd36e5:5:2ba failed due to 2017-05-10T20:07:15,033 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-088] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd2d19:5:2cd failed due to 2017-05-10T20:07:15,391 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-153] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd3254:5:2b8 failed due to 2017-05-10T20:07:15,441 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-154] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14__RT_12_32_0: with request id c0a8015a:15bf3dd36e5:5:2ba failed due to ==> rm-error.log <== 2017-05-10T20:07:15,391 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3254:5:2b8, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3254:5:2b8' 2017-05-10T20:07:15,391 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3254:5:2b8, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3254:5:2b8' ==> cm-error.log <== 2017-05-10T20:07:16,078 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd324e:6:2b6, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd324e:6:2b6' 2017-05-10T20:07:16,078 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd324e:6:2b6, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd324e:6:2b6' ==> dtquery-error.log <== 2017-05-10T20:07:16,432 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3983:9:2b4, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3983:9:2b4' 2017-05-10T20:07:16,432 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3983:9:2b4, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3983:9:2b4' ==> resourcesvc-error.log <== 2017-05-10T20:07:16,078 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-058] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd324e:6:2b6 failed due to 2017-05-10T20:07:16,431 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-059] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd3983:9:2b4 failed due to 2017-05-10T20:07:16,078 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-058] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14__RT_12_32_0: with request id c0a8015a:15bf3dd324e:6:2b6 failed due to 2017-05-10T20:07:16,431 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-059] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd3983:9:2b4 failed due to ==> metering-error.log <== 2017-05-10T20:07:17,094 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3f04:5:2ba, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3f04:5:2ba' 2017-05-10T20:07:17,094 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3f04:5:2ba, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3f04:5:2ba' ==> provisionsvc-error.log <== 2017-05-10T20:07:16,829 [pool-52-thread-9-c0a8015a:15bf3dd211f:ea:0] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd17ec:30:7c, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd17ec:30:7c' 2017-05-10T20:07:16,829 [pool-52-thread-9-c0a8015a:15bf3dd211f:ea:0] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd17ec:30:7c, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd17ec:30:7c' ==> resourcesvc-error.log <== 2017-05-10T20:07:16,828 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-060-c0a8015a:15bf3dd211f:ea:0] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd17ec:30:7c failed due to 2017-05-10T20:07:16,894 [ZoneScanner-000] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd3a34:6:1ad failed due to 2017-05-10T20:07:16,894 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3a34:6:1ad, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3a34:6:1ad' 2017-05-10T20:07:17,093 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-062] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd3f04:5:2ba failed due to 2017-05-10T20:07:17,374 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-127] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd22fb:5:2ba failed due to 2017-05-10T20:07:16,828 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-060-c0a8015a:15bf3dd211f:ea:0] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd17ec:30:7c failed due to 2017-05-10T20:07:16,894 [ZoneScanner-000] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd3a34:6:1ad failed due to 2017-05-10T20:07:16,894 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd3a34:6:1ad, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd3a34:6:1ad' 2017-05-10T20:07:17,093 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-062] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd3f04:5:2ba failed due to 2017-05-10T20:07:17,374 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-127] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14RT_12_32_0: with request id c0a8015a:15bf3dd22fb:5:2ba failed due to ==> ssm-error.log <== 2017-05-10T20:07:17,375 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd22fb:5:2ba, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd22fb:5:2ba' 2017-05-10T20:07:17,375 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd22fb:5:2ba, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd22fb:5:2ba' ==> blobsvc-error.log <== 2017-05-10T20:07:17,992 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd383d:5:2ba, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd383d:5:2ba' 2017-05-10T20:07:17,992 [ZoneScanner-000] ERROR ResourceClient.java (line 325) request id c0a8015a:15bf3dd383d:5:2ba, command type REQUEST_RG_GET failed with error code ERROR_RG_NOT_FOUND and message 'rg schemaType REP_GROUP_KEY rgId urn:storageos:ReplicationGroupInfo:00000000-0000-0000-0000-000000000000:global not found, request id c0a8015a:15bf3dd383d:5:2ba' ==> eventsvc-error.log <== 2017-05-10T20:07:18,621 [TaskScheduler-EventService-DEFAULT_BACKGROUND_OPERATION-ScheduledExecutor-073] ERROR ConnectEmcTask.java (line 186) failed to process alert schemaType ALERT_KEY timestamp 1494445643583 alertId urn:storageos:Alert:70e044bd-a551-4efb-a43d-225f7047cf77 com.emc.storageos.data.object.exception.ObjectControllerException: ERROR_INTERNAL 2017-05-10T20:07:18,621 [TaskScheduler-EventService-DEFAULT_BACKGROUND_OPERATION-ScheduledExecutor-073] ERROR ConnectEmcTask.java (line 186) failed to process alert schemaType ALERT_KEY timestamp 1494445643583 alertId urn:storageos:Alert:70e044bd-a551-4efb-a43d-225f7047cf77 com.emc.storageos.data.object.exception.ObjectControllerException: ERROR_INTERNAL ==> resourcesvc-error.log <== 2017-05-10T20:07:17,991 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-029] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14__RT_12_32_0: with request id c0a8015a:15bf3dd383d:5:2ba failed due to 2017-05-10T20:07:17,991 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-029] ERROR AbstractResourceDirectoryRequestHandler.java (line 72) command REQUEST_RG_GET on DT urn:storageos:OwnershipInfo:957babf1-e4cc-4d33-8488-40f196a4ac14__RT_12_32_0: with request id c0a8015a:15bf3dd383d:5:2ba failed due to ^C

df -h View

[centos@ecstest emcvipr-object]$ df -h Filesystem Size Used Avail Use% Mounted on /dev/vda1 60G 2.9G 58G 5% / devtmpfs 7.8G 0 7.8G 0% /dev tmpfs 7.8G 0 7.8G 0% /dev/shm tmpfs 7.8G 17M 7.8G 1% /run tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup tmpfs 1.6G 0 1.6G 0% /run/user/1000 tmpfs 1.6G 0 1.6G 0% /run/user/0 /dev/vdb1 200G 191G 9.9G 96% /ecs/uuid-6eee34f0-1b83-4881-8690-4837af07c566

cblackuk commented 7 years ago

Additional info: cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core)

But I tried on CentOS 7.1 and 7.3 - same issues

########### Nmap stats ######### sudo nmap -sT -O localhost

Starting Nmap 6.40 ( http://nmap.org ) at 2017-05-10 20:24 UTC Nmap scan report for localhost (127.0.0.1) Host is up (0.000054s latency). Other addresses for localhost (not scanned): 127.0.0.1 Not shown: 993 closed ports PORT STATE SERVICE 22/tcp open ssh 25/tcp open smtp 111/tcp open rpcbind 4443/tcp open pharos 7443/tcp open oracleas-https 9010/tcp open sdr 9011/tcp open unknown No exact OS matches for host (If you know what OS is running on it, see http://nmap.org/submit/ ). TCP/IP fingerprint: OS:SCAN(V=6.40%E=4%D=5/10%OT=22%CT=1%CU=41369%PV=N%DS=0%DC=L%G=Y%TM=591376F OS:A%P=x86_64-redhat-linux-gnu)SEQ(SP=106%GCD=1%ISR=10B%TI=Z%CI=I%II=I%TS=A OS:)OPS(O1=MFFD7ST11NW7%O2=MFFD7ST11NW7%O3=MFFD7NNT11NW7%O4=MFFD7ST11NW7%O5 OS:=MFFD7ST11NW7%O6=MFFD7ST11)WIN(W1=AAAA%W2=AAAA%W3=AAAA%W4=AAAA%W5=AAAA%W OS:6=AAAA)ECN(R=Y%DF=Y%T=40%W=AAAA%O=MFFD7NNSNW7%CC=Y%Q=)T1(R=Y%DF=Y%T=40%S OS:=O%A=S+%F=AS%RD=0%Q=)T2(R=N)T3(R=N)T4(R=Y%DF=Y%T=40%W=0%S=A%A=Z%F=R%O=%R OS:D=0%Q=)T5(R=Y%DF=Y%T=40%W=0%S=Z%A=S+%F=AR%O=%RD=0%Q=)T6(R=Y%DF=Y%T=40%W= OS:0%S=A%A=Z%F=R%O=%RD=0%Q=)T7(R=Y%DF=Y%T=40%W=0%S=Z%A=S+%F=AR%O=%RD=0%Q=)U OS:1(R=Y%DF=N%T=40%IPL=164%UN=0%RIPL=G%RID=G%RIPCK=G%RUCK=G%RUD=G)IE(R=Y%DF OS:I=N%T=40%CD=S)

Network Distance: 0 hops

OS detection performed. Please report any incorrect results at http://nmap.org/submit/ . Nmap done: 1 IP address (1 host up) scanned in 11.85 seconds

############# DT Check ###########

sudo curl -X GET "http://192.168.1.90:9101/stats/dt/DTInitStat" ?xml version="1.0" encoding="UTF-8" standalone="yes"?>38400[centos@ecstest emcvipr-object]$

cblackuk commented 7 years ago

Error logs from MultiNode Install:

[centos@ecs00001 emcvipr-object]$ tail -f * | grep -i error tail: cannot open ‘audit’ for reading: Permission denied tail: cannot open ‘btmp’ for reading: Permission denied ==> blobsvc-error.log <== ==> btree-gc-error.log <== 2017-05-10T14:45:02,452 [main] ERROR BTreeGC.java (line 219) No root record found. Exiting tail: error reading ‘cores’: Is a directory tail: cores: cannot follow end of this type of file; giving up on this name ==> cm-error.log <== 2017-05-10T15:18:31,032 [TaskScheduler-ChunkManager-ParallelExecutor-133] ERROR Allocator.java (line 505) Allocating Blocks for chunk: f11f3490-3c19-4dc6-9355-0ded20d6b485 , hit error: java.util.concurrent.ExecutionException: com.emc.storageos.data.object.exception.ObjectControllerException: allocate block failed for SSID 192.168.1.86, chunk ID f11f3490-3c19-4dc6-9355-0ded20d6b485, request id c0a80153:15bf2e5e574:1a:1e, serverIp 192.168.1.84 2017-05-10T15:18:31,163 [TaskScheduler-ChunkManager-ParallelExecutor-133] ERROR Allocator.java (line 407) allocate block for f11f3490-3c19-4dc6-9355-0ded20d6b485 get error, retry # 1, com.emc.storageos.data.object.exception.ObjectControllerException: allocate block get error for chunk f11f3490-3c19-4dc6-9355-0ded20d6b485, hit exception when allocate blocks, chunkId:f11f3490-3c19-4dc6-9355-0ded20d6b485, error: java.util.concurrent.ExecutionException: com.emc.storageos.data.object.exception.ObjectControllerException: allocate block failed for SSID 192.168.1.86, chunk ID f11f3490-3c19-4dc6-9355-0ded20d6b485, request id c0a80153:15bf2e5e574:1a:1e, serverIp 192.168.1.84 ==> dataheadsvc-error.log <== 2017-05-10T15:38:43,593 [TaskScheduler-DataHead-OWNERSHIP-ParallelExecutor-009] ERROR DirectoryOwner.java (line 2385) lease expired, lastSuccessfulCheck = 0, currentTime = 4201249784248, livenessMonitorTimeoutNanos = 15000000000 2017-05-10T15:38:43,593 [TaskScheduler-DataHead-OWNERSHIP-ParallelExecutor-022] ERROR DirectoryOwner.java (line 2385) lease expired, lastSuccessfulCheck = 0, currentTime = 4201250215390, livenessMonitorTimeoutNanos = 15000000000 2017-05-10T15:38:43,594 [TaskScheduler-DataHead-OWNERSHIP-ParallelExecutor-022] ERROR DirectoryOwner.java (line 2385) lease expired, lastSuccessfulCheck = 0, currentTime = 4201250639474, livenessMonitorTimeoutNanos = 15000000000 tail: cannot open ‘lastlog’ for reading2017-05-10T15:38:43,594 [TaskScheduler-DataHead-OWNERSHIP-ParallelExecutor-002] ERROR DirectoryOwner.java (line 2385) lease expired, lastSuccessfulCheck = 0, currentTime = 4201250707447, livenessMonitorTimeoutNanos = 15000000000 : Permission denied 2017-05-10T15:38:43,594 [TaskScheduler-DataHead-OWNERSHIP-ParallelExecutor-022] ERROR DirectoryOwner.java (line 2385) lease expired, lastSuccessfulCheck = 0, currentTime = 4201250804015, livenessMonitorTimeoutNanos = 15000000000 2017-05-10T15:38:43,594 [TaskScheduler-DataHead-OWNERSHIP-ParallelExecutor-002] ERROR DirectoryOwner.java (line 2385) lease expired, lastSuccessfulCheck = 0, currentTime = 4201250925950, livenessMonitorTimeoutNanos = 15000000000 2017-05-10T15:38:43,594 [TaskScheduler-DataHead-OWNERSHIP-ParallelExecutor-023] ERROR DirectoryOwner.java (line 2385) lease expired, lastSuccessfulCheck = 0, currentTime = 4201251159646, livenessMonitorTimeoutNanos = 15000000000 2017-05-10T15:38:43,593 [TaskScheduler-DataHead-OWNERSHIP-ParallelExecutor-017] ERROR DirectoryOwner.java (line 2385) lease expired, lastSuccessfulCheck = 0, currentTime = 4201249787887, livenessMonitorTimeoutNanos = 15000000000 2017-05-10T15:38:43,595 [TaskScheduler-DataHead-OWNERSHIP-ParallelExecutor-021] ERROR DirectoryOwner.java (line 2385) lease expired, lastSuccessfulCheck = 0, currentTime = 4201251929298, livenessMonitorTimeoutNanos = 15000000000 2017-05-10T15:38:43,595 [TaskScheduler-DataHead-OWNERSHIP-ParallelExecutor-001] ERROR DirectoryOwner.java (line 2385) lease expired, lastSuccessfulCheck = 0, currentTime = 4201252161118, livenessMonitorTimeoutNanos = 15000000000 ==> dtquery-error.log <== May 10, 2017 3:07:43 PM com.sun.jersey.spi.inject.Errors processErrorMessages "alertsNumUnackError" : [ { ==> eventsvc-error.log <== mount: /.volumes/cdrom1: mount failed: Unknown error -1 _gen_hostname /etc/HOSTNAME: Error 1 _gen_ssh_user_key /home/svcuser/.ssh/id_dsa: Error 1 _gen_ssh_authkeys2 /home/svcuser/.ssh/authorized_keys2: Error 1 _gen_firewall_conf /etc/sysconfig: Error 2 ==> georeceiver-error.log <== 2017-05-10T15:19:17,686 [main-StartScanner] ERROR DirectoryOwner.java (line 2385) lease expired, lastSuccessfulCheck = 0, currentTime = 3035343100316, livenessMonitorTimeoutNanos = 15000000000 2017-05-10T15:19:17,698 [main-StartScanner] ERROR DirectoryOwner.java (line 2385) lease expired, lastSuccessfulCheck = 0, currentTime = 3035354659380, livenessMonitorTimeoutNanos = 15000000000 2017-05-10T15:19:17,711 [main-StartScanner] ERROR DirectoryOwner.java (line 2385) lease expired, lastSuccessfulCheck = 0, currentTime = 3035368253055, livenessMonitorTimeoutNanos = 15000000000 2017-05-10T15:19:17,727 [main-StartScanner] ERROR DirectoryOwner.java (line 2385) lease expired, lastSuccessfulCheck = 0, currentTime = 3035384395713, livenessMonitorTimeoutNanos = 15000000000 2017-05-10T15:19:17,737 [main-StartScanner] ERROR DirectoryOwner.java (line 2385) lease expired, lastSuccessfulCheck = 0, currentTime = 3035393913769, livenessMonitorTimeoutNanos = 15000000000 2017-05-10T15:19:17,747 [main-StartScanner] ERROR DirectoryOwner.java (line 2385) lease expired, lastSuccessfulCheck = 0, currentTime = 3035404319840, livenessMonitorTimeoutNanos = 15000000000 2017-05-10T15:19:17,757 [main-StartScanner] ERROR DirectoryOwner.java (line 2385) lease expired, lastSuccessfulCheck = 0, currentTime = 3035414034747, livenessMonitorTimeoutNanos = 15000000000 2017-05-10T15:19:17,768 [main-StartScanner] ERROR DirectoryOwner.java (line 2385) lease expired, lastSuccessfulCheck = 0, currentTime = 3035424505019, livenessMonitorTimeoutNanos = 15000000000 2017-05-10T15:25:17,975 [main-StartScanner] ERROR DirectoryOwner.java (line 2385) lease expired, lastSuccessfulCheck = 0, currentTime = 3395631599053, livenessMonitorTimeoutNanos = 15000000000 2017-05-10T15:25:18,011 [main-StartScanner] ERROR DirectoryOwner.java (line 2385) lease expired, lastSuccessfulCheck = 0, currentTime = 3395667561720, livenessMonitorTimeoutNanos = 15000000000 ==> metering-error.log <== 2017-05-10T15:09:39,436 [TaskScheduler-MeteringManager-BFW-ParallelExecutor-008] ERROR ChunkClientImpl.java (line 257) chunk creation failed 620fcc0e-dd45-4d2f-aac9-1af6a1b21adb, write context: [writerId: 192.168.1.83-9203-DIRECTORY_TABLE-BFW-1-writer-0, cos: urn:storageos:VirtualArray:71d60f2f-c361-4ffe-9307-94952917ad44, level: 0, type: DIRECTORY_TABLE, skipChunkSequenceCounter: false, repoMergeChunkId: null], error com.emc.storageos.data.object.exception.ObjectControllerException: remote hit internal error, request c0a80153:15bf2e5e229:11f:0, ip 192.168.1.86:9091 sleep 5000 millis before retry 2017-05-10T15:09:42,545 [TaskScheduler-MeteringManager-BFW-ParallelExecutor-009] ERROR ChunkClientImpl.java (line 257) chunk creation failed 9aedd82d-9ebd-4d2b-86b0-20214c3fd560, write context: [writerId: 192.168.1.83-9203-BPLUSTREE-BFW-2-writer-0, cos: urn:storageos:VirtualArray:71d60f2f-c361-4ffe-9307-94952917ad44, level: 0, type: BPLUSTREE, skipChunkSequenceCounter: false, repoMergeChunkId: null], error com.emc.storageos.data.object.exception.ObjectControllerException: create chunk 9aedd82d-9ebd-4d2b-86b0-20214c3fd560 failed, request id c0a80153:15bf2e5e229:121:0, serverIp 192.168.1.85, status ERROR_NO_STORAGE_DEVICE_FOUND sleep 5000 millis before retry 2017-05-10T15:09:44,502 [TaskScheduler-MeteringManager-BFW-ParallelExecutor-008] ERROR ChunkClientImpl.java (line 257) chunk creation failed cc794110-211e-45da-9e6d-b7c9e017d011, write context: [writerId: 192.168.1.83-9203-DIRECTORY_TABLE-BFW-1-writer-0, cos: urn:storageos:VirtualArray:71d60f2f-c361-4ffe-9307-94952917ad44, level: 0, type: DIRECTORY_TABLE, skipChunkSequenceCounter: false, repoMergeChunkId: null], error com.emc.storageos.data.object.exception.ObjectControllerException: create chunk cc794110-211e-45da-9e6d-b7c9e017d011 failed, request id c0a80153:15bf2e5e229:11f:4, serverIp 192.168.1.83, status ERROR_NO_STORAGE_DEVICE_FOUND sleep 5000 millis before retry 2017-05-10T15:09:44,806 [TaskScheduler-MeteringManager-BFW-ParallelExecutor-010] ERROR ChunkClientImpl.java (line 257) chunk creation failed 2187996d-919f-44be-bf66-2a0f8d171081, write context: [writerId: 192.168.1.83-9203-BPLUSTREE-BFW-2-writer-1, cos: urn:storageos:VirtualArray:71d60f2f-c361-4ffe-9307-94952917ad44, level: 0, type: BPLUSTREE, skipChunkSequenceCounter: false, repoMergeChunkId: null], error com.emc.storageos.data.object.exception.ObjectControllerException: create chunk 2187996d-919f-44be-bf66-2a0f8d171081 failed, request id c0a80153:15bf2e5e229:12a:0, serverIp 192.168.1.85, status ERROR_NO_STORAGE_DEVICE_FOUND sleep 5000 millis before retry 2017-05-10T15:09:47,580 [TaskScheduler-MeteringManager-BFW-ParallelExecutor-009] ERROR ChunkClientImpl.java (line 257) chunk creation failed 789e633f-ff63-4d82-8238-ec6e944fbda8, write context: [writerId: 192.168.1.83-9203-BPLUSTREE-BFW-2-writer-0, cos: urn:storageos:VirtualArray:71d60f2f-c361-4ffe-9307-94952917ad44, level: 0, type: BPLUSTREE, skipChunkSequenceCounter: false, repoMergeChunkId: null], error com.emc.storageos.data.object.exception.ObjectControllerException: create chunk 789e633f-ff63-4d82-8238-ec6e944fbda8 failed, request id c0a80153:15bf2e5e229:121:4, serverIp 192.168.1.85, status ERROR_NO_STORAGE_DEVICE_FOUND sleep 5000 millis before retry 2017-05-10T15:09:49,510 [TaskScheduler-MeteringManager-BFW-ParallelExecutor-008] ERROR ChunkClientImpl.java (line 257) chunk creation failed 2c296aeb-49c3-4610-9c30-4525ba35b125, write context: [writerId: 192.168.1.83-9203-DIRECTORY_TABLE-BFW-1-writer-0, cos: urn:storageos:VirtualArray:71d60f2f-c361-4ffe-9307-94952917ad44, level: 0, type: DIRECTORY_TABLE, skipChunkSequenceCounter: false, repoMergeChunkId: null], error com.emc.storageos.data.object.exception.ObjectControllerException: create chunk 2c296aeb-49c3-4610-9c30-4525ba35b125 failed, request id c0a80153:15bf2e5e229:11f:8, serverIp 192.168.1.85, status ERROR_NO_STORAGE_DEVICE_FOUND sleep 5000 millis before retry 2017-05-10T15:09:49,832 [TaskScheduler-MeteringManager-BFW-ParallelExecutor-010] ERROR ChunkClientImpl.java (line 257) chunk creation failed 0901fb6b-7a85-45b7-9d18-3abcc7856227, write context: [writerId: 192.168.1.83-9203-BPLUSTREE-BFW-2-writer-1, cos: urn:storageos:VirtualArray:71d60f2f-c361-4ffe-9307-94952917ad44, level: 0, type: BPLUSTREE, skipChunkSequenceCounter: false, repoMergeChunkId: null], error com.emc.storageos.data.object.exception.ObjectControllerException: create chunk 0901fb6b-7a85-45b7-9d18-3abcc7856227 failed, request id c0a80153:15bf2e5e229:12a:4, serverIp 192.168.1.85, status ERROR_NO_STORAGE_DEVICE_FOUND sleep 5000 millis before retry 2017-05-10T15:09:52,589 [TaskScheduler-MeteringManager-BFW-ParallelExecutor-009] ERROR ChunkClientImpl.java (line 257) chunk creation failed 00ad0be1-8336-420e-9aad-a06b5923f800, write context: [writerId: 192.168.1.83-9203-BPLUSTREE-BFW-2-writer-0, cos: urn:storageos:VirtualArray:71d60f2f-c361-4ffe-9307-94952917ad44, level: 0, type: BPLUSTREE, skipChunkSequenceCounter: false, repoMergeChunkId: null], error com.emc.storageos.data.object.exception.ObjectControllerException: create chunk 00ad0be1-8336-420e-9aad-a06b5923f800 failed, request id c0a80153:15bf2e5e229:121:8, serverIp 192.168.1.83, status ERROR_NO_STORAGE_DEVICE_FOUND sleep 5000 millis before retry ==> nginx_error.log <== 2017/05/10 14:35:17 [error] 13779#0: 1 no live upstreams while connecting to upstream, client: 192.168.1.83, server: 192.168.1.83, request: "GET /login HTTP/1.1", upstream: "https://authsvc/login", host: "192.168.1.83:4443" 2017/05/10 15:07:26 [error] 13335#0: 147 no live upstreams while connecting to upstream, client: 127.0.0.1, server: 192.168.1.83, request: "GET /user/whoami HTTP/1.1", upstream: "https://authsvc/user/whoami", host: "localhost:4443" 2017/05/10 15:07:29 [error] 13335#0: 163 no live upstreams while connecting to upstream, client: 127.0.0.1, server: 192.168.1.83, request: "GET /dashboard/zones/localzone?category=alerts HTTP/1.1", upstream: "http://dashboardsvc/dashboard/zones/localzone?category=alerts", host: "localhost:4443" 2017/05/10 15:07:30 [error] 13335#0: 162 no live upstreams while connecting to upstream, client: 127.0.0.1, server: 192.168.1.83, request: "GET /login?service=https%3A%2F%2Flocalhost%3A4443%2Fvdc%2Fdata-services%2Fvarrays HTTP/1.1", upstream: "https://authsvc/login?service=https%3A%2F%2Flocalhost%3A4443%2Fvdc%2Fdata-services%2Fvarrays", host: "localhost:4443" 2017/05/10 15:07:30 [error] 13335#0: 163 no live upstreams while connecting to upstream, client: 127.0.0.1, server: 192.168.1.83, request: "GET /login?service=https%3A%2F%2Flocalhost%3A4443%2Ffeature%2FServerSideEncryption HTTP/1.1", upstream: "https://authsvc/login?service=https%3A%2F%2Flocalhost%3A4443%2Ffeature%2FServerSideEncryption", host: "localhost:4443" 2017/05/10 15:07:30 [error] 13335#0: 147 no live upstreams while connecting to upstream, client: 127.0.0.1, server: 192.168.1.83, request: "GET /login?service=https%3A%2F%2Flocalhost%3A4443%2Fvdc%2Fdata-services%2Fvarrays HTTP/1.1", upstream: "https://authsvc/login?service=https%3A%2F%2Flocalhost%3A4443%2Fvdc%2Fdata-services%2Fvarrays", host: "localhost:4443" ==> provisionsvc-error.log <== tail: error reading ‘statistics’: Is a directory tail: statistics: cannot follow end of this type of file; giving up on this name ==> resourcesvc-error.log <== 2017-05-10T15:25:14,482 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-193] ERROR RemoteRequestProcessorResourceSvc.java (line 75) handleRequest, id c0a80153:15bf2e5e31c:12b:31f type REQUEST_RG_LIST, error com.emc.storageos.data.object.exception.ObjectControllerException: Not the owner for key urn:storageos:OwnershipInfo:71d60f2f-c361-4ffe-9307-94952917ad44RT_18_32_0: 2017-05-10T15:25:14,773 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-211] ERROR RemoteRequestProcessorResourceSvc.java (line 75) handleRequest, id c0a80154:15bf2e5e401:51:2b6 type REQUEST_RG_LIST, error com.emc.storageos.data.object.exception.ObjectControllerException: Not the owner for key urn:storageos:OwnershipInfo:71d60f2f-c361-4ffe-9307-94952917ad44__RT_22_32_0: 2017-05-10T15:25:14,810 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-220] ERROR RemoteRequestProcessorResourceSvc.java (line 75) handleRequest, id c0a80154:15bf2e5e401:51:2e1 type REQUEST_RG_LIST, error com.emc.storageos.data.object.exception.ObjectControllerException: Not the owner for key urn:storageos:OwnershipInfo:71d60f2f-c361-4ffe-9307-94952917ad44RT_18_32_0: tail: cannot open ‘wtmp’ for reading: Permission denied2017-05-10T15:25:17,636 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-007] ERROR RemoteRequestProcessorResourceSvc.java (line 75) handleRequest, id c0a80155:15bf2e5f970:11c:347 type REQUEST_RG_LIST, error com.emc.storageos.data.object.exception.ObjectControllerException: Not the owner for key urn:storageos:OwnershipInfo:71d60f2f-c361-4ffe-9307-94952917ad44__RT_22_32_0:

2017-05-10T15:25:17,671 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-008] ERROR RemoteRequestProcessorResourceSvc.java (line 75) handleRequest, id c0a80155:15bf2e5f970:11c:374 type REQUEST_RG_LIST, error com.emc.storageos.data.object.exception.ObjectControllerException: Not the owner for key urn:storageos:OwnershipInfo:71d60f2f-c361-4ffe-9307-94952917ad44RT_18_32_0: 2017-05-10T15:25:17,972 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-031] ERROR RemoteRequestProcessorResourceSvc.java (line 75) handleRequest, id c0a80153:15bf2e5cad7:4b:2b6 type REQUEST_RG_LIST, error com.emc.storageos.data.object.exception.ObjectControllerException: Not the owner for key urn:storageos:OwnershipInfo:71d60f2f-c361-4ffe-9307-94952917ad44__RT_22_32_0: 2017-05-10T15:25:18,008 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-034] ERROR RemoteRequestProcessorResourceSvc.java (line 75) handleRequest, id c0a80153:15bf2e5cad7:4b:2e1 type REQUEST_RG_LIST, error com.emc.storageos.data.object.exception.ObjectControllerException: Not the owner for key urn:storageos:OwnershipInfo:71d60f2f-c361-4ffe-9307-94952917ad44RT_18_32_0: 2017-05-10T15:25:19,557 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-049] ERROR RemoteRequestProcessorResourceSvc.java (line 75) handleRequest, id c0a80155:15bf2e5ef52:1bc:302 type REQUEST_RG_LIST, error com.emc.storageos.data.object.exception.ObjectControllerException: Not the owner for key urn:storageos:OwnershipInfo:71d60f2f-c361-4ffe-9307-94952917ad44RT_18_32_0: 2017-05-10T15:25:20,720 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-067] ERROR RemoteRequestProcessorResourceSvc.java (line 75) handleRequest, id c0a80156:15bf2e6d2c1:fe:346 type REQUEST_RG_LIST, error com.emc.storageos.data.object.exception.ObjectControllerException: Not the owner for key urn:storageos:OwnershipInfo:71d60f2f-c361-4ffe-9307-94952917ad44__RT_22_32_0: 2017-05-10T15:25:20,756 [TaskScheduler-ResourceSvc-COMMUNICATOR-ParallelExecutor-075] ERROR RemoteRequestProcessorResourceSvc.java (line 75) handleRequest, id c0a80156:15bf2e6d2c1:fe:373 type REQUEST_RG_LIST, error com.emc.storageos.data.object.exception.ObjectControllerException: Not the owner for key urn:storageos:OwnershipInfo:71d60f2f-c361-4ffe-9307-94952917ad44RT_18_32_0: ==> rm-error.log <== 2017-05-10T15:09:43,297 [TaskScheduler-RecordManager-ParallelExecutor-093] ERROR ChunkClientImpl.java (line 257) chunk creation failed 0f3ed079-a4e4-4e05-bc7b-958b1234af28, write context: [writerId: 192.168.1.83-1096-BPLUSTREE-BFW-1-writer-1, cos: urn:storageos:VirtualArray:71d60f2f-c361-4ffe-9307-94952917ad44, level: 1, type: BPLUSTREE, skipChunkSequenceCounter: false, repoMergeChunkId: null], error com.emc.storageos.data.object.exception.ObjectControllerException: remote hit internal error, request c0a80153:15bf2e5e040:e6:d, ip 192.168.1.84:9091 sleep 5000 millis before retry 2017-05-10T15:09:48,311 [TaskScheduler-RecordManager-ParallelExecutor-093] ERROR ChunkClientImpl.java (line 257) chunk creation failed 183f1255-2d02-4d1b-a8fe-572dc238d10f, write context: [writerId: 192.168.1.83-1096-BPLUSTREE-BFW-1-writer-1, cos: urn:storageos:VirtualArray:71d60f2f-c361-4ffe-9307-94952917ad44, level: 1, type: BPLUSTREE, skipChunkSequenceCounter: false, repoMergeChunkId: null], error com.emc.storageos.data.object.exception.ObjectControllerException: create chunk 183f1255-2d02-4d1b-a8fe-572dc238d10f failed, request id c0a80153:15bf2e5e040:e6:f, serverIp 192.168.1.86, status ERROR_NO_STORAGE_DEVICE_FOUND sleep 5000 millis before retry ==> ssm-error.log <== 2017-05-10T15:30:21,225 [TaskScheduler-SSManager-ParallelExecutor-065] ERROR SSManagerRequestProcessor.java (line 658) verifyBlockAssignment busyblock chunk mismatch for device:192.168.1.83 partition:48dd713e-0e9b-449e-9faa-42c5a25f44c5 blockbin:0003 offset:268435200 chunkid:fb205a44-f714-4b40-b4db-54a8d5bb3f47 size:134217600 found chunkid:a4deccdb-f54d-4501-8438-f498cbd4220f offset:134217600 size:134217600 state:BLOCK_ALLOCATED ==> stat-error.log <== 2017-05-10T15:06:50,076 [SenderSocketThread-192.168.1.85:9204-3-000] ERROR CommunicatorSenderSocket.java (line 473) reInitializeSocketConnection - threw Interrupted exception for remote 192.168.1.85 port 9204 2017-05-10T15:06:50,078 [SenderSocketThread-192.168.1.84:9204-4-000] ERROR CommunicatorSenderSocket.java (line 473) reInitializeSocketConnection - threw Interrupted exception for remote 192.168.1.84 port 9204 2017-05-10T15:07:46,180 [ReceiverSocketThread-192.168.1.83:53787-15] ERROR RemoteDataNodeCommunicator.java (line 1044) socket for 192.168.1.83 broken ==> storageserver-error.log <== 2017-May-10 20:52:09.379535 20771724590 ERROR 11613 0x7f7940005dc0 emc:storageserver src/storageserver/storageserver.cpp:1213 getAgentHealthFromAgent invalid uri /host/data/agent.json/v1/agent/mode check configuration 2017-May-10 20:52:14.380755 20776725810 ERROR 11584 0x7f791c0057c0 emc:storageserver src/storageserver/storageserver.cpp:1213 getAgentHealthFromAgent invalid uri /host/data/agent.json/v1/agent/mode check configuration 2017-May-10 20:52:19.381772 20781726827 ERROR 11607 0x7f793c014c80 emc:storageserver src/storageserver/storageserver.cpp:1213 getAgentHealthFromAgent invalid uri /host/data/agent.json/v1/agent/mode check configuration 2017-May-10 20:52:24.382394 20786727449 ERROR 11595 0x7f792c005bf0 emc:storageserver src/storageserver/storageserver.cpp:1213 getAgentHealthFromAgent invalid uri /host/data/agent.json/v1/agent/mode check configuration 2017-May-10 20:52:29.383273 20791728327 ERROR 11594 0x7f7920006b60 emc:storageserver src/storageserver/storageserver.cpp:1213 getAgentHealthFromAgent invalid uri /host/data/agent.json/v1/agent/mode check configuration 2017-May-10 20:52:34.383874 20796728929 ERROR 11607 0x7f793c014c80 emc:storageserver src/storageserver/storageserver.cpp:1213 getAgentHealthFromAgent invalid uri /host/data/agent.json/v1/agent/mode check configuration 2017-May-10 20:52:39.385376 20801730431 ERROR 11564 0x7f78f0005a90 emc:storageserver src/storageserver/storageserver.cpp:1213 getAgentHealthFromAgent invalid uri /host/data/agent.json/v1/agent/mode check configuration 2017-May-10 20:52:44.386101 20806731156 ERROR 11582 0x7f7910005d00 emc:storageserver src/storageserver/storageserver.cpp:1213 getAgentHealthFromAgent invalid uri /host/data/agent.json/v1/agent/mode check configuration 2017-May-10 20:52:49.386949 20811732004 ERROR 11598 0x7f79280071d0 emc:storageserver src/storageserver/storageserver.cpp:1213 getAgentHealthFromAgent invalid uri /host/data/agent.json/v1/agent/mode check configuration 2017-May-10 20:52:54.387730 20816732784 ERROR 11582 0x7f7910005d00 emc:storageserver src/storageserver/storageserver.cpp:1213 getAgentHealthFromAgent invalid uri /host/data/agent.json/v1/agent/mode check configuration 2017-May-10 20:52:54.387730 20816732784 ERROR 11582 0x7f7910005d00 emc:storageserver src/storageserver/storageserver.cpp:1213 getAgentHealthFromAgent invalid uri /host/data/agent.json/v1/agent/mode check configuration ==> vnest-error.log <== 2017-05-10T15:09:39,797 [pool-28-thread-1] ERROR KeepAliveMessageSender.java (line 203) Caught IOException (Is the node down?) trying to reset existing connection to 192.168.1.83:9230 2017-05-10T15:09:48,567 [SenderSocketThread-192.168.1.85:9230-33-000] ERROR CommunicatorSenderSocket.java (line 473) reInitializeSocketConnection - threw Interrupted exception for remote 192.168.1.85 port 9230 2017-05-10T15:10:04,803 [pool-28-thread-1] ERROR KeepAliveMessageSender.java (line 203) Caught IOException (Is the node down?) trying to reset existing connection to 192.168.1.85:9230 2017-05-10T15:10:08,707 [SenderSocketThread-192.168.1.84:9230-20-000] ERROR CommunicatorSenderSocket.java (line 473) reInitializeSocketConnection - threw Interrupted exception for remote 192.168.1.84 port 9230 2017-05-10T15:10:24,831 [pool-28-thread-1] ERROR KeepAliveMessageSender.java (line 203) Caught IOException (Is the node down?) trying to reset existing connection to 192.168.1.84:9230 2017-05-10T15:10:28,750 [SenderSocketThread-192.168.1.83:9230-4-000] ERROR CommunicatorSenderSocket.java (line 473) reInitializeSocketConnection - threw Interrupted exception for remote 192.168.1.83 port 9230 2017-05-10T15:10:44,899 [pool-28-thread-1] ERROR KeepAliveMessageSender.java (line 203) Caught IOException (Is the node down?) trying to reset existing connection to 192.168.1.86:9230 2017-05-10T15:11:10,533 [pool-28-thread-1] ERROR KeepAliveMessageSender.java (line 203) Caught IOException (Is the node down?) trying to reset existing connection to 192.168.1.86:9230 2017-05-10T15:11:27,831 [SenderSocketThread-192.168.1.86:9230-70-000] ERROR CommunicatorSenderSocket.java (line 473) reInitializeSocketConnection - threw Interrupted exception for remote 192.168.1.86 port 9230 2017-05-10T15:11:36,157 [pool-28-thread-1] ERROR KeepAliveMessageSender.java (line 203) Caught IOException (Is the node down?) trying to reset existing connection to 192.168.1.86:9230 ==> storageserver-error.log <== 2017-May-10 20:52:59.388646 20821733701 ERROR 11590 0x7f79240095e0 emc:storageserver src/storageserver/storageserver.cpp:1213 getAgentHealthFromAgent invalid uri /host/data/agent.json/v1/agent/mode check configuration 2017-May-10 20:52:59.388646 20821733701 ERROR 11590 0x7f79240095e0 emc:storageserver src/storageserver/storageserver.cpp:1213 getAgentHealthFromAgent invalid uri /host/data/agent.json/v1/agent/mode check configuration ==> eventsvc-error.log <== 2017-05-10T20:52:59,906 [TaskScheduler-EventService-DEFAULT_BACKGROUND_OPERATION-ScheduledExecutor-088] ERROR ConnectEmcTask.java (line 186) failed to process alert schemaType ALERT_KEY timestamp 1494427520623 alertId urn:storageos:Alert:55d65bcc-3b31-44fd-88ce-445b9651435f com.emc.storageos.data.object.exception.ObjectControllerException: ERROR_INTERNAL 2017-05-10T20:53:00,044 [TaskScheduler-EventService-DEFAULT_BACKGROUND_OPERATION-ScheduledExecutor-011] ERROR ConnectEmcTask.java (line 186) failed to process alert schemaType ALERT_KEY timestamp 1494427521047 alertId urn:storageos:Alert:733aa833-e4d0-4ee0-877e-9ac22f9d1432 com.emc.storageos.data.object.exception.ObjectControllerException: ERROR_INTERNAL 2017-05-10T20:53:00,063 [TaskScheduler-EventService-DEFAULT_BACKGROUND_OPERATION-ScheduledExecutor-004] ERROR ConnectEmcTask.java (line 186) failed to process alert schemaType ALERT_KEY timestamp 1494427519782 alertId urn:storageos:Alert:3068146c-0b3a-4b0a-ae0e-3404b7a80702 com.emc.storageos.data.object.exception.ObjectControllerException: ERROR_INTERNAL 2017-05-10T20:52:59,906 [TaskScheduler-EventService-DEFAULT_BACKGROUND_OPERATION-ScheduledExecutor-088] ERROR ConnectEmcTask.java (line 186) failed to process alert schemaType ALERT_KEY timestamp 1494427520623 alertId urn:storageos:Alert:55d65bcc-3b31-44fd-88ce-445b9651435f com.emc.storageos.data.object.exception.ObjectControllerException: ERROR_INTERNAL 2017-05-10T20:53:00,044 [TaskScheduler-EventService-DEFAULT_BACKGROUND_OPERATION-ScheduledExecutor-011] ERROR ConnectEmcTask.java (line 186) failed to process alert schemaType ALERT_KEY timestamp 1494427521047 alertId urn:storageos:Alert:733aa833-e4d0-4ee0-877e-9ac22f9d1432 com.emc.storageos.data.object.exception.ObjectControllerException: ERROR_INTERNAL 2017-05-10T20:53:00,063 [TaskScheduler-EventService-DEFAULT_BACKGROUND_OPERATION-ScheduledExecutor-004] ERROR ConnectEmcTask.java (line 186) failed to process alert schemaType ALERT_KEY timestamp 1494427519782 alertId urn:storageos:Alert:3068146c-0b3a-4b0a-ae0e-3404b7a80702 com.emc.storageos.data.object.exception.ObjectControllerException: ERROR_INTERNAL ^C

padthaitofuhot commented 7 years ago

Hi @cblackuk. If you can store objects in buckets without issue, then it's likely that your storage pools and data stores are fully initialized. ECS Community Edition is a bit weird in that there are some UI/display issues with storage pools showing "Not Ready" and data stores showing "initializing" even after they have become fully initialized. If you can create VDCs, replication groups, namespaces, and buckets, then your storage pools are certainly initialized as those higher level abstractions require a working storage pool. Please let me know if I can help with anything else.

cblackuk commented 7 years ago

Thanks for getting back to me. How about the thousands of error messages that constantly being logged? Nothing to worry about?

padthaitofuhot commented 7 years ago

Nah, those are normal. Most of those have to do with elements of the SKU build of ECS not being present in the CE build. If you're having trouble using object storage APIs, that's when we'll want to look at certain logs.

cblackuk commented 7 years ago

Wish I knew that before I wasted so much time on it. We stood up the same environment a few months ago and it is showing ready but I was trying a new one (on multiple platforms) and never got it in READY state.

Having said that, I just created two buckets one per environment and uploaded images to them via Cyberduck so I guess it is working?

padthaitofuhot commented 7 years ago

Yes, seems like it is working. There was a version change between a few months ago and a few weeks ago, so perhaps the display issue only showed up in the new version. For the sake of record-keeping and upstream bug reporting, could you login to both environments and paste output of the following command for each here?

# sudo docker exec -it  ecsstandalone cat /etc/ecs_release

The output should look something like this:

[admin@luna ~]$ sudo docker exec -it ecs-storageos cat /etc/ecs_release
3.0.0.0.86239.1c9e5ec
[admin@luna ~]$
cblackuk commented 7 years ago

Sure:

SingleNode

sudo docker exec -it ecsmultinode cat /etc/ecs_release 3.0.0.0.86239.1c9e5ec

MultiNode

sudo docker exec -it ecsstandalone cat /etc/ecs_release 3.0.0.0.86239.1c9e5ec

Also I will get the output from the version that is working when I get to the office.

padthaitofuhot commented 7 years ago

@cblackuk did you happen to get that output from the working version?

cblackuk commented 7 years ago

I have no SSH access to the working system, just the ECS GUI, I am trying to get access but it has not happened yet.

ksteinfeldt commented 7 years ago

http://ecsce.readthedocs.io/en/latest/troubleshooting/FAQ.html