search

EMCECS / ECS-CommunityEdition

ECS Community Edition "Free & Frictionless"
https://community.emc.com/community/products/ecs
Other
131 stars 94 forks source link

Error deploying #532

Closed cybojanek closed 2 years ago

cybojanek commented 3 years ago

Expected Behavior

step1 does not have python errors step2 finishes

Actual Behavior

step1 has python errors step2 never finnishes

Steps to Reproduce Behavior

  1. Allocate OpenStack VM
  2. Run the following commands:
# OpenStack
# CentOS-7-x86_64-GenericCloud-1811.qcow2
# 16 cores, 80 gigabytes RAM, 384 GB root disk (vda), 256 GB ecs disk (vdb)
# [centos@ecs ~]$ lsblk
# NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
# vda    253:0    0  384G  0 disk
# `-vda1 253:1    0  384G  0 part /
# vdb    253:16   0  256G  0 disk
# Security Group: allow all traffic from 10.0.0.0/8

# setup passwordless login
ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

# Fix passwordless sudo
sudo bash -c 'echo "%wheel  ALL=(ALL)       NOPASSWD: ALL" >> /etc/sudoers'
sudo bash -c 'echo "Defaults verifypw = any" >> /etc/sudoers'

# Install docker
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo yum install -y docker-ce docker-ce-cli containerd.io
sudo systemctl enable docker
sudo systemctl start docker

# Install firewalld
sudo yum -y install firewalld
sudo systemctl enable firewalld
sudo systemctl start firewalld

# Disable systemd rpcbind because it conflicts with ECS
sudo systemctl stop rpcbind
sudo systemctl disable rpcbind
sudo systemctl mask rpcbind
sudo systemctl stop rpcbind.socket
sudo systemctl disable rpcbind.socket

# disable ipv6
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1

sudo bash -c 'echo "net.ipv6.conf.all.disable_ipv6 = 1" >> /etc/sysctl.conf'
sudo bash -c 'echo "net.ipv6.conf.default.disable_ipv6 = 1" >> /etc/sysctl.conf'

# Install git and patch
sudo yum install -y git patch

# clone ecs
git clone https://github.com/EMCECS/ECS-CommunityEdition
cd ECS-CommunityEdition
git checkout 2.9.1-3.4.0.0
cp docs/design/reference.deploy.yml deploy.yml

# Modify settings
patch -p1 << EOF
--- a/deploy.yml  2021-02-17 19:02:18.834295654 +0000
+++ b/deploy.yml  2021-02-17 19:05:16.658361406 +0000
@@ -6,7 +6,7 @@
 # contained in the license.txt file included with this software
 # distribution.
 licensing:
-  license_accepted: false
+  license_accepted: true

 #autonames:
 #  custom:
@@ -28,7 +28,7 @@
   # NOTE: if the install node is to be migrated into an island environment,
   #       the hostname or IP address listed here should be the one in the
   #       island environment.
-  install_node: 192.168.2.200
+  install_node: 10.0.0.114

   # [Required]
   # IPs of machines that will be whitelisted in the firewall and allowed
@@ -44,14 +44,14 @@
   ssh_defaults:
     # [Required]
     # Username to use when logging in to nodes
-    ssh_username: admin
+    ssh_username: centos
     # [Required]
     # Password to use with SSH login
     # *** Set to same value as ssh_username to enable SSH public key authentication ***
-    ssh_password: ChangeMe
+    ssh_password: centos
     # [Required when enabling SSH public key authentication]
     # Password to give to sudo when gaining root access.
-    ansible_become_pass: ChangeMe
+    ansible_become_pass:
     # [Required]
     # Select the type of crypto to use when dealing with ssh public key
     # authentication. Valid values here are:
@@ -64,9 +64,9 @@
   node_defaults:
     dns_domain: local
     dns_servers:
-      - 192.168.2.2
+      - 8.8.8.8
     ntp_servers:
-      - 192.168.2.2
+      - 216.239.35.0
     #
     # [Optional]
     # VFS path to source of randomness
@@ -106,13 +106,13 @@
   storage_pools:
     - name: sp1
       members:
-        - 192.168.2.200
+        - 10.0.0.114
       options:
         is_protected: false
         is_cold_storage_enabled: false
         description: My First SP
         ecs_block_devices:
-          - /dev/vda
+          - /dev/vdb

   # [Optional]
   # VDC defaults. Configure to your liking.
@@ -159,11 +159,11 @@
   # Management Users
   management_users:
     - username: admin1
-      password: ChangeMe
+      password: eYQk0x2LnvJf6V0HfrxM4xS1tFlT96vfuxcaUpuD
       options:
         is_system_admin: true
     - username: monitor1
-      password: ChangeMe
+      password: 83Ecujcse9RgmDmTVUlJtcNPLrmwdk8eWze7wksL
       options:
         is_system_monitor: true

@@ -203,13 +203,13 @@
         swift_groups_list:
           - admin
           - users
-        s3_secret_key: ChangeMeChangeMeChangeMeChangeMeChangeMe
+        s3_secret_key: rXu8LD396E5tBpNLjXvVGdge30uRghQR03iAbWwL
         s3_expiry_time: 2592000
     - username: object_user1
       namespace: ns1
       options:
-        swift_password: ChangeMe
-        s3_secret_key: ChangeMeChangeMeChangeMeChangeMeChangeMe
+        swift_password: 666673483fbef0b7e90b467b
+        s3_secret_key: 9VllJJ2Bnrr1jjGyMkY7frTrjuqpb9pb9P7Ojr9e

   # [Optional]
   # Bucket defaults
EOF

# bootstrap
./bootstrap.sh -c deploy.yml --ssh-private-key ~/.ssh/id_rsa --ssh-public-key ~/.ssh/id_rsa.pub

# after reboot
cd ECS-CommunityEdition
step1

# Fails with:
# [centos@ecs ECS-CommunityEdition]$ step1
# ecs-install> Initializing data container, one moment ..DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
#   ERROR: Command errored out with exit status 1:
#    command: /usr/bin/python2 /usr/lib/python2.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-pbZ5tR/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.6.0' wheel 'cffi>=1.12; platform_python_implementation != '"'"'PyPy'"'"''
#        cwd: None
#   Complete output (67 lines):
#   DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
#   Collecting setuptools>=40.6.0
#     Using cached setuptools-44.1.1-py2.py3-none-any.whl (583 kB)
#   Collecting wheel
#     Using cached wheel-0.36.2-py2.py3-none-any.whl (35 kB)
#   Collecting cffi>=1.12
#     Downloading cffi-1.14.5.tar.gz (475 kB)
#   Collecting pycparser
#     Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
#   Using legacy 'setup.py install' for cffi, since package 'wheel' is not installed.
#   Installing collected packages: setuptools, wheel, pycparser, cffi
#       Running setup.py install for cffi: started
#       Running setup.py install for cffi: finished with status 'error'
#       ERROR: Command errored out with exit status 1:
#        command: /usr/bin/python2 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-exwiQT/cffi/setup.py'"'"'; __file__='"'"'/tmp/pip-install-exwiQT/cffi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-RSMofJ/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-pbZ5tR/overlay --compile --install-headers /tmp/pip-build-env-pbZ5tR/overlay/include/python2.7/cffi
#            cwd: /tmp/pip-install-exwiQT/cffi/
#       Complete output (48 lines):
#       unable to execute 'gcc': No such file or directory
#       unable to execute 'gcc': No such file or directory
#
#           No working compiler found, or bogus compiler options passed to
#           the compiler from Python's standard "distutils" module.  See
#           the error messages above.  Likely, the problem is not related
#           to CFFI but generic to the setup.py of any Python package that
#           tries to compile C code.  (Hints: on OS/X 10.8, for errors about
#           -mno-fused-madd see http://stackoverflow.com/questions/22313407/
#           Otherwise, see https://wiki.python.org/moin/CompLangPython or
#           the IRC channel #python on irc.freenode.net.)
#
#           Trying to continue anyway.  If you are trying to install CFFI from
#           a build done in a different context, you can ignore this warning.
#
#       running install
#       running build
#       running build_py
#       creating build
#       creating build/lib.linux-x86_64-2.7
#       creating build/lib.linux-x86_64-2.7/cffi
#       copying cffi/__init__.py -> build/lib.linux-x86_64-2.7/cffi
#       copying cffi/api.py -> build/lib.linux-x86_64-2.7/cffi
#       copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-2.7/cffi
#       copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-2.7/cffi
#       copying cffi/commontypes.py -> build/lib.linux-x86_64-2.7/cffi
#       copying cffi/cparser.py -> build/lib.linux-x86_64-2.7/cffi
#       copying cffi/error.py -> build/lib.linux-x86_64-2.7/cffi
#       copying cffi/ffiplatform.py -> build/lib.linux-x86_64-2.7/cffi
#       copying cffi/lock.py -> build/lib.linux-x86_64-2.7/cffi
#       copying cffi/model.py -> build/lib.linux-x86_64-2.7/cffi
#       copying cffi/pkgconfig.py -> build/lib.linux-x86_64-2.7/cffi
#       copying cffi/recompiler.py -> build/lib.linux-x86_64-2.7/cffi
#       copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-2.7/cffi
#       copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-2.7/cffi
#       copying cffi/vengine_gen.py -> build/lib.linux-x86_64-2.7/cffi
#       copying cffi/verifier.py -> build/lib.linux-x86_64-2.7/cffi
#       copying cffi/_cffi_include.h -> build/lib.linux-x86_64-2.7/cffi
#       copying cffi/parse_c_type.h -> build/lib.linux-x86_64-2.7/cffi
#       copying cffi/_embedding.h -> build/lib.linux-x86_64-2.7/cffi
#       copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-2.7/cffi
#       running build_ext
#       building '_cffi_backend' extension
#       creating build/temp.linux-x86_64-2.7
#       creating build/temp.linux-x86_64-2.7/c
#       gcc -fno-strict-aliasing -Os -fomit-frame-pointer -g -DNDEBUG -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-x86_64-2.7/c/_cffi_backend.o
#       unable to execute 'gcc': No such file or directory
#       error: command 'gcc' failed with exit status 1
#       ----------------------------------------
#   ERROR: Command errored out with exit status 1: /usr/bin/python2 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-exwiQT/cffi/setup.py'"'"'; __file__='"'"'/tmp/pip-install-exwiQT/cffi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-RSMofJ/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-pbZ5tR/overlay --compile --install-headers /tmp/pip-build-env-pbZ5tR/overlay/include/python2.7/cffi Check the logs for full command output.
#   ----------------------------------------
# ERROR: Command errored out with exit status 1: /usr/bin/python2 /usr/lib/python2.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-pbZ5tR/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.6.0' wheel 'cffi>=1.12; platform_python_implementation != '"'"'PyPy'"'"'' Check the logs for full command output.
# ecs-install> ui.tgz failed to build.
# . OK
# ecs-install> Applying and validating deploy.yml...
# /usr/local/bin/entrypoint.sh: line 167: ecsdeploy: not found
# /usr/local/bin/entrypoint.sh: line 232: ecsdeploy: not found
#
# [centos@ecs ECS-CommunityEdition]$

# To get past this step, I did the following:
# Start docker container
sudo docker run --rm -i -t --privileged --net=host \
    --volumes-from ecs-install-data \
    -v /opt/emc/ecs-install:/opt \
    -v /opt/emc/ecs-install/ssh:/root/.ssh \
    -v /opt/emc/ecs-install/ssl:/usr/local/share/ca-certificates \
    -v /var/log/emc:/var/log \
    -v /var/cache:/var/cache \
    --entrypoint /bin/sh emccorp/ecs-install:latest

# Add missing python build dependencies
apk add build-base python2-dev libffi-dev openssl-dev

# Install non-breaking package versions - not sure if these are correct
pip install cryptography==2.8 "pyopenssl<20.0.0" "pykwalify<1.8.0" "ruamel.yaml<0.15.83"

# Exit container
exit

# Run step1 again
step1
# NOTE: Still has some warnings:
# ecs-install> Applying and validating deploy.yml...
# /usr/lib/python2.7/site-packages/ruamel/yaml/constructor.py:281: DuplicateKeyFutureWarning: while constructing a mapping
#   in "/usr/local/src/ui/etc/schema.yml", line 28, column 7
# found duplicate key "object_user_defaults" with value "{}" (original value: "{}")
#   in "/usr/local/src/ui/etc/schema.yml", line 388, column 7
#
# To suppress this check see:
#     http://yaml.readthedocs.io/en/latest/api.html#duplicate-keys
#
# Duplicate keys will become an error in future releases, and are errors
# by default when using the new API.
#
#   warnings.warn(DuplicateKeyFutureWarning(*args))

# Run step2
step2
# Stuck at:
> Pinging endpoint 10.0.0.114... (CTRL-C to break)
> WAIT: ECS API timed out.  If ECS services are still bootstrapping, this is likely temporary.
> WAIT: ECS API timed out.  If ECS services are still bootstrapping, this is likely temporary.

Relevant Output and Logs

# Output and Logs go here
# Step2 output:
> Pinging endpoint 10.0.0.114... (CTRL-C to break)
> WAIT: ECS API timed out.  If ECS services are still bootstrapping, this is likely temporary.
> WAIT: ECS API timed out.  If ECS services are still bootstrapping, this is likely temporary.
> WAIT: ECS API timed out.  If ECS services are still bootstrapping, this is likely temporary.

DTInitStat:
[centos@luna ~]$ curl -X GET "http://10.0.0.114:9101/stats/dt/DTInitStat"
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><result><entry><total_dt_num>192</total_dt_num><unready_dt_num>0</unready_dt_num
[centos@luna ~]$

Errors in logs:
docker exec -it ecs-storageos /bin/bash
tail -f /var/log/*.log | grep -F ERROR --line-buffered
# Snippet of error logs
2021-02-17T20:32:05,410 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:08,414 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:14,420 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:23,425 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:31:56,972 [TaskScheduler-BlobService-DefaultScheduler-007] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:00,402 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:00,406 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:05,410 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:08,414 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:14,420 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:23,425 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:20,240 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:21:17,788 [pool-4-thread-1] ERROR  ActivityLog.java (line 106) ZooKeeper Server: ZooKeeper JMX enabled by default
2021-02-17T20:21:17,790 [pool-4-thread-1] ERROR  ActivityLog.java (line 106) ZooKeeper Server: Using config: /data/zk/bin/../conf/zoo.cfg
2021-02-17T20:32:00,755 [TaskScheduler-DataHead-DefaultScheduler-008] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:05,732 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:08,736 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:14,741 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:31:20,332 [TaskScheduler-DtQuery-DefaultScheduler-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:31:20,419 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:31:56,337 [TaskScheduler-DtQuery-DefaultScheduler-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:31:59,423 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:31:59,428 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:04,434 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:07,439 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:13,444 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:22,449 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:31:16,227 [TaskScheduler-EventService-DefaultScheduler-009] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:31:17,268 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:31:52,233 [TaskScheduler-EventService-DefaultScheduler-009] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:31:56,273 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:31:56,277 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:01,284 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:04,289 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:10,295 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:19,300 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:09,965 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:12,970 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:13,529 [TaskScheduler-GeoReceiver-DefaultScheduler-005] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:18,976 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:18,976 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:03,257 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:06,261 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:12,268 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:21,272 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:06,261 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:12,268 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:21,272 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:03,764 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:04,592 [TaskScheduler-ObjControlService-DefaultScheduler-008] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 6908, level: 0]
2021-02-17T20:32:06,769 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:07,129 [ProviderConfigUpdater] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3519, level: 0]
2021-02-17T20:32:09,465 [qtp313945225-263] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: ET, keyHash: 169, level: 0]
2021-02-17T20:32:12,775 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:14,474 [qtp313945225-376] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: ET, keyHash: 889, level: 0]
2021-02-17T20:32:18,471 [qtp313945225-263] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: ET, keyHash: 169, level: 0]
2021-02-17T20:32:21,779 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:22,134 [ProviderConfigUpdater] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3519, level: 0]
2021-02-17T20:31:53,301 [TaskScheduler-ResourceSvc-DefaultScheduler-004] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:31:57,229 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:31:57,233 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:02,240 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:05,244 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:11,249 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:20,254 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:31:58,940 [TaskScheduler-RecordManager-ParallelExecutor-001-0a000072:177b1a588c3:33:0] ERROR  ChunkClientImpl.java (line 265) chunk creation failed a4e4cf92-09d7-4424-962a-b1c5f5250bea, write context: [writerId: 10.0.0.114-1096-DIRECTORY_TABLE-BFW-0-writer-0, cos: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, level: 1, type: DIRECTORY_TABLE, skipChunkSequenceCounter: false, repoMergeChunkId: null], error com.emc.storageos.data.object.exception.ObjectControllerException: create chunk a4e4cf92-09d7-4424-962a-b1c5f5250bea failed, request id 0a000072:177b1a780b6:64:10a, serverIp 10.0.0.114, status ERROR_NO_STORAGE_DEVICE_FOUND sleep 5000 millis before retry
2021-02-17T20:32:03,949 [TaskScheduler-RecordManager-ParallelExecutor-001-0a000072:177b1a588c3:33:0] ERROR  ChunkClientImpl.java (line 265) chunk creation failed dba5129b-d55e-40b5-907e-a898e38b4dc9, write context: [writerId: 10.0.0.114-1096-DIRECTORY_TABLE-BFW-0-writer-0, cos: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, level: 1, type: DIRECTORY_TABLE, skipChunkSequenceCounter: false, repoMergeChunkId: null], error com.emc.storageos.data.object.exception.ObjectControllerException: create chunk dba5129b-d55e-40b5-907e-a898e38b4dc9 failed, request id 0a000072:177b1a780b6:64:10c, serverIp 10.0.0.114, status ERROR_NO_STORAGE_DEVICE_FOUND sleep 5000 millis before retry
2021-02-17T20:32:08,960 [TaskScheduler-RecordManager-ParallelExecutor-001-0a000072:177b1a588c3:33:0] ERROR  ChunkClientImpl.java (line 265) chunk creation failed 6c4e2e91-59fc-4721-a65d-3775d12bd917, write context: [writerId: 10.0.0.114-1096-DIRECTORY_TABLE-BFW-0-writer-0, cos: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, level: 1, type: DIRECTORY_TABLE, skipChunkSequenceCounter: false, repoMergeChunkId: null], error com.emc.storageos.data.object.exception.ObjectControllerException: create chunk 6c4e2e91-59fc-4721-a65d-3775d12bd917 failed, request id 0a000072:177b1a780b6:64:10e, serverIp 10.0.0.114, status ERROR_NO_STORAGE_DEVICE_FOUND sleep 5000 millis before retry
2021-02-17T20:32:13,969 [TaskScheduler-RecordManager-ParallelExecutor-001-0a000072:177b1a588c3:33:0] ERROR  ChunkClientImpl.java (line 265) chunk creation failed 0c281af9-7a3c-4d96-a7c4-1d121a23110f, write context: [writerId: 10.0.0.114-1096-DIRECTORY_TABLE-BFW-0-writer-0, cos: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, level: 1, type: DIRECTORY_TABLE, skipChunkSequenceCounter: false, repoMergeChunkId: null], error com.emc.storageos.data.object.exception.ObjectControllerException: create chunk 0c281af9-7a3c-4d96-a7c4-1d121a23110f failed, request id 0a000072:177b1a780b6:64:110, serverIp 10.0.0.114, status ERROR_NO_STORAGE_DEVICE_FOUND sleep 5000 millis before retry
2021-02-17T20:32:18,978 [TaskScheduler-RecordManager-ParallelExecutor-001-0a000072:177b1a588c3:33:0] ERROR  ChunkClientImpl.java (line 265) chunk creation failed 84f18079-8700-4686-ad71-6e7b10b951f2, write context: [writerId: 10.0.0.114-1096-DIRECTORY_TABLE-BFW-0-writer-0, cos: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, level: 1, type: DIRECTORY_TABLE, skipChunkSequenceCounter: false, repoMergeChunkId: null], error com.emc.storageos.data.object.exception.ObjectControllerException: create chunk 84f18079-8700-4686-ad71-6e7b10b951f2 failed, request id 0a000072:177b1a780b6:64:112, serverIp 10.0.0.114, status ERROR_NO_STORAGE_DEVICE_FOUND sleep 5000 millis before retry
2021-02-17T20:32:10,585 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:19,589 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:31:59,291 [TaskScheduler-SrManager-DefaultScheduler-008] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:03,298 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:06,304 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:12,309 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:21,313 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:31:58,290 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:31:58,293 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:31:59,291 [TaskScheduler-SrManager-DefaultScheduler-008] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:03,298 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:06,304 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:12,309 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:21,313 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:23:54,848 [TaskScheduler-SSManager-DEFAULT_BACKGROUND_OPERATION-ScheduledExecutor-017] ERROR  DirectoryTable.java (line 1923) dt urn:storageos:OwnershipInfo:4b52e9ea-d19f-4566-aad6-4a4a328d22aa__SS_29_32_1:96c52fe4-5b9a-401e-ae5c-39a89c864421 prefix schemaType SSTABLE_TASK_KEY type BLOCK_RELINQUISH_TASK device  partition  blockBin  offset -1 token null, ex com.emc.storageos.data.object.exception.ObjectControllerException: no effective owner - dt urn:storageos:OwnershipInfo:4b52e9ea-d19f-4566-aad6-4a4a328d22aa__SS_29_32_1:96c52fe4-5b9a-401e-ae5c-39a89c864421
2021-02-17T20:23:54,848 [TaskScheduler-SSManager-DEFAULT_BACKGROUND_OPERATION-ScheduledExecutor-038] ERROR  DirectoryTable.java (line 1923) dt urn:storageos:OwnershipInfo:4b52e9ea-d19f-4566-aad6-4a4a328d22aa__SS_29_32_1:96c52fe4-5b9a-401e-ae5c-39a89c864421 prefix schemaType SSTABLE_TASK_KEY type BLOCK_FREE_TASK device  partition  blockBin  offset -1 token null, ex com.emc.storageos.data.object.exception.ObjectControllerException: no effective owner - dt urn:storageos:OwnershipInfo:4b52e9ea-d19f-4566-aad6-4a4a328d22aa__SS_29_32_1:96c52fe4-5b9a-401e-ae5c-39a89c864421
2021-02-17T20:23:54,939 [TaskScheduler-SSManager-DEFAULT_BACKGROUND_OPERATION-ScheduledExecutor-008]  INFO  DirectoryTable.java (line 790) dt id urn:storageos:OwnershipInfo:4b52e9ea-d19f-4566-aad6-4a4a328d22aa__SS_30_32_1:, status ERROR: No effective owner for directory
2021-02-17T20:23:54,939 [TaskScheduler-SSManager-DEFAULT_BACKGROUND_OPERATION-ScheduledExecutor-040]  INFO  DirectoryTable.java (line 790) dt id urn:storageos:OwnershipInfo:4b52e9ea-d19f-4566-aad6-4a4a328d22aa__SS_30_32_1:, status ERROR: No effective owner for directory
2021-02-17T20:23:54,939 [TaskScheduler-SSManager-DEFAULT_BACKGROUND_OPERATION-ScheduledExecutor-008] ERROR  DirectoryTable.java (line 1923) dt urn:storageos:OwnershipInfo:4b52e9ea-d19f-4566-aad6-4a4a328d22aa__SS_30_32_1:63ce0a4a-fcba-43a2-a193-d87ddba51470 prefix schemaType SSTABLE_TASK_KEY type BLOCK_RELINQUISH_TASK device  partition  blockBin  offset -1 token null, ex com.emc.storageos.data.object.exception.ObjectControllerException: no effective owner - dt urn:storageos:OwnershipInfo:4b52e9ea-d19f-4566-aad6-4a4a328d22aa__SS_30_32_1:63ce0a4a-fcba-43a2-a193-d87ddba51470
2021-02-17T20:23:54,939 [TaskScheduler-SSManager-DEFAULT_BACKGROUND_OPERATION-ScheduledExecutor-040] ERROR  DirectoryTable.java (line 1923) dt urn:storageos:OwnershipInfo:4b52e9ea-d19f-4566-aad6-4a4a328d22aa__SS_30_32_1:63ce0a4a-fcba-43a2-a193-d87ddba51470 prefix schemaType SSTABLE_TASK_KEY type BLOCK_FREE_TASK device  partition  blockBin  offset -1 token null, ex com.emc.storageos.data.object.exception.ObjectControllerException: no effective owner - dt urn:storageos:OwnershipInfo:4b52e9ea-d19f-4566-aad6-4a4a328d22aa__SS_30_32_1:63ce0a4a-fcba-43a2-a193-d87ddba51470
2021-02-17T20:23:55,030 [TaskScheduler-SSManager-DEFAULT_BACKGROUND_OPERATION-ScheduledExecutor-044]  INFO  DirectoryTable.java (line 790) dt id urn:storageos:OwnershipInfo:4b52e9ea-d19f-4566-aad6-4a4a328d22aa__SS_31_32_1:, status ERROR: No effective owner for directory
2021-02-17T20:23:55,030 [TaskScheduler-SSManager-DEFAULT_BACKGROUND_OPERATION-ScheduledExecutor-047]  INFO  DirectoryTable.java (line 790) dt id urn:storageos:OwnershipInfo:4b52e9ea-d19f-4566-aad6-4a4a328d22aa__SS_31_32_1:, status ERROR: No effective owner for directory
2021-02-17T20:23:55,030 [TaskScheduler-SSManager-DEFAULT_BACKGROUND_OPERATION-ScheduledExecutor-044] ERROR  DirectoryTable.java (line 1923) dt urn:storageos:OwnershipInfo:4b52e9ea-d19f-4566-aad6-4a4a328d22aa__SS_31_32_1:14b18154-1a9a-4c0c-a592-e266233df1ab prefix schemaType SSTABLE_TASK_KEY type BLOCK_RELINQUISH_TASK device  partition  blockBin  offset -1 token null, ex com.emc.storageos.data.object.exception.ObjectControllerException: no effective owner - dt urn:storageos:OwnershipInfo:4b52e9ea-d19f-4566-aad6-4a4a328d22aa__SS_31_32_1:14b18154-1a9a-4c0c-a592-e266233df1ab
2021-02-17T20:23:55,030 [TaskScheduler-SSManager-DEFAULT_BACKGROUND_OPERATION-ScheduledExecutor-047] ERROR  DirectoryTable.java (line 1923) dt urn:storageos:OwnershipInfo:4b52e9ea-d19f-4566-aad6-4a4a328d22aa__SS_31_32_1:14b18154-1a9a-4c0c-a592-e266233df1ab prefix schemaType SSTABLE_TASK_KEY type BLOCK_FREE_TASK device  partition  blockBin  offset -1 token null, ex com.emc.storageos.data.object.exception.ObjectControllerException: no effective owner - dt urn:storageos:OwnershipInfo:4b52e9ea-d19f-4566-aad6-4a4a328d22aa__SS_31_32_1:14b18154-1a9a-4c0c-a592-e266233df1ab
2021-02-17T20:32:02,238 [TaskScheduler-SSManager-ParallelExecutor-017-0a000072:177b1a588c3:33:0] ERROR  ChunkClientImpl.java (line 265) chunk creation failed 7fce4efa-57da-4101-b0b6-81c05331d9e6, write context: [writerId: 10.0.0.114-1095-DIRECTORY_TABLE-BFW-0-writer-0, cos: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, level: 1, type: DIRECTORY_TABLE, skipChunkSequenceCounter: false, repoMergeChunkId: null], error com.emc.storageos.data.object.exception.ObjectControllerException: create chunk 7fce4efa-57da-4101-b0b6-81c05331d9e6 failed, request id 0a000072:177b1a78c86:121:102, serverIp 10.0.0.114, status ERROR_NO_STORAGE_DEVICE_FOUND sleep 5000 millis before retry
2021-02-17T20:32:07,249 [TaskScheduler-SSManager-ParallelExecutor-017-0a000072:177b1a588c3:33:0] ERROR  ChunkClientImpl.java (line 265) chunk creation failed 41fbd6f5-b8c2-4f6e-bad3-9e64418a1a9b, write context: [writerId: 10.0.0.114-1095-DIRECTORY_TABLE-BFW-0-writer-0, cos: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, level: 1, type: DIRECTORY_TABLE, skipChunkSequenceCounter: false, repoMergeChunkId: null], error com.emc.storageos.data.object.exception.ObjectControllerException: create chunk 41fbd6f5-b8c2-4f6e-bad3-9e64418a1a9b failed, request id 0a000072:177b1a78c86:121:104, serverIp 10.0.0.114, status ERROR_NO_STORAGE_DEVICE_FOUND sleep 5000 millis before retry
2021-02-17T20:32:12,261 [TaskScheduler-SSManager-ParallelExecutor-017-0a000072:177b1a588c3:33:0] ERROR  ChunkClientImpl.java (line 265) chunk creation failed 07f926b5-b69c-4a7a-b556-81746e27aabe, write context: [writerId: 10.0.0.114-1095-DIRECTORY_TABLE-BFW-0-writer-0, cos: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, level: 1, type: DIRECTORY_TABLE, skipChunkSequenceCounter: false, repoMergeChunkId: null], error com.emc.storageos.data.object.exception.ObjectControllerException: create chunk 07f926b5-b69c-4a7a-b556-81746e27aabe failed, request id 0a000072:177b1a78c86:121:108, serverIp 10.0.0.114, status ERROR_NO_STORAGE_DEVICE_FOUND sleep 5000 millis before retry
2021-02-17T20:32:17,270 [TaskScheduler-SSManager-ParallelExecutor-017-0a000072:177b1a588c3:33:0] ERROR  ChunkClientImpl.java (line 265) chunk creation failed b130458d-7dca-432a-b418-2e15e399680b, write context: [writerId: 10.0.0.114-1095-DIRECTORY_TABLE-BFW-0-writer-0, cos: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, level: 1, type: DIRECTORY_TABLE, skipChunkSequenceCounter: false, repoMergeChunkId: null], error com.emc.storageos.data.object.exception.ObjectControllerException: create chunk b130458d-7dca-432a-b418-2e15e399680b failed, request id 0a000072:177b1a78c86:121:10a, serverIp 10.0.0.114, status ERROR_NO_STORAGE_DEVICE_FOUND sleep 5000 millis before retry
2021-02-17T20:32:22,278 [TaskScheduler-SSManager-ParallelExecutor-017-0a000072:177b1a588c3:33:0] ERROR  ChunkClientImpl.java (line 265) chunk creation failed 036f230e-5c1b-49ae-8679-bc6d1c45e8dc, write context: [writerId: 10.0.0.114-1095-DIRECTORY_TABLE-BFW-0-writer-0, cos: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, level: 1, type: DIRECTORY_TABLE, skipChunkSequenceCounter: false, repoMergeChunkId: null], error com.emc.storageos.data.object.exception.ObjectControllerException: create chunk 036f230e-5c1b-49ae-8679-bc6d1c45e8dc failed, request id 0a000072:177b1a78c86:121:10c, serverIp 10.0.0.114, status ERROR_NO_STORAGE_DEVICE_FOUND sleep 5000 millis before retry
2021-02-17T20:32:05,889 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:11,894 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:20,898 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:21,182 [ChunkProgressCacheImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: CT, keyHash: 0, level: 1]
2021-02-17T20:21:36,139 [pool-33-thread-1] ERROR  StatClientBackupper.java (line 102) Failed to read statistics from /opt/storageos/logs/clientstatistics/stat-statistics.json
2021-02-17T20:30:26,099 [pool-62-thread-1] ERROR  StatClientBackupper  Failed to read statistics from /opt/storageos/logs/clientstatistics/transformsvc-statistics.json
2021-02-17T20:31:30,211 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:31:37,361 [TaskScheduler-transform-DefaultScheduler-009] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:31:51,216 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:31:55,366 [TaskScheduler-transform-DefaultScheduler-009] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:15,221 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:16,370 [TaskScheduler-transform-DefaultScheduler-009] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:21:23,619 [ForkJoinPool-COMMUNICATOR-worker-427] ERROR  PaxosClient.java (line 546) key schemaType VIPR_NEST_KEY  path /dynamicconfig key DYNAMIC_CONFIG_KEY VALUE 0 com.emc.ecs.chunk.ss.cache.journal.enabled already exists
2021-02-17T20:21:23,619 [ForkJoinPool-COMMUNICATOR-worker-427] ERROR  PaxosClient.java (line 546) key schemaType VIPR_NEST_KEY  path /dynamicconfig key DYNAMIC_CONFIG_KEY VALUE 0 com.emc.ecs.chunk.ss.cache.repo.enabled already exists
2021-02-17T20:21:29,024 [ReceiverSocketThread-10.0.0.114:41471-10] ERROR  RemoteDataNodeCommunicator.java (line 1095) socket for 10.0.0.114 broken
2021-02-17T20:21:49,244 [pool-33-thread-1] ERROR  KeepAliveMessageSender.java (line 214) Caught IOException (Is the node down?) trying to reset existing connection to 10.0.0.114:9230
2021-02-17T20:22:14,277 [pool-33-thread-1] ERROR  KeepAliveMessageSender.java (line 214) Caught IOException (Is the node down?) trying to reset existing connection to 10.0.0.114:9230
2021-02-17T20:22:39,332 [pool-33-thread-1] ERROR  KeepAliveMessageSender.java (line 214) Caught IOException (Is the node down?) trying to reset existing connection to 10.0.0.114:9230
2021-02-17T20:23:04,361 [pool-33-thread-1] ERROR  KeepAliveMessageSender.java (line 214) Caught IOException (Is the node down?) trying to reset existing connection to 10.0.0.114:9230
2021-02-17T20:23:29,390 [pool-33-thread-1] ERROR  KeepAliveMessageSender.java (line 214) Caught IOException (Is the node down?) trying to reset existing connection to 10.0.0.114:9230
2021-02-17T20:23:44,351 [SenderSocketThread-10.0.0.114:9230-12-000] ERROR  CommunicatorSenderSocket.java (line 500) reInitializeSocketConnection - threw Interrupted exception for remote 10.0.0.114 port 9230
2021-02-17T20:23:54,418 [pool-33-thread-1] ERROR  KeepAliveMessageSender.java (line 214) Caught IOException (Is the node down?) trying to reset existing connection to 10.0.0.114:9230
2021-02-17T20:32:23,745 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:23,745 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:23,985 [TaskScheduler-ChunkManager-ParallelExecutor-024-0a000072:177b1a588c3:33:0] ERROR  SsSelector.java (line 600) Not enough resources for chunk 5279adf7-16f0-4c47-b6ef-6361dc2b3425: cos urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa level 2 to place 1 segments with type SYSTEM
2021-02-17T20:32:23,985 [TaskScheduler-ChunkManager-ParallelExecutor-024-0a000072:177b1a588c3:33:0] ERROR  SsSelector.java (line 600) Not enough resources for chunk 5279adf7-16f0-4c47-b6ef-6361dc2b3425: cos urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa level 2 to place 1 segments with type SYSTEM
2021-02-17T20:32:23,985 [TaskScheduler-ChunkManager-ParallelExecutor-024-0a000072:177b1a588c3:33:0] ERROR  Allocator.java (line 571) allocate block for 5279adf7-16f0-4c47-b6ef-6361dc2b3425 get error, retry # 1
2021-02-17T20:32:23,985 [TaskScheduler-ChunkManager-ParallelExecutor-024-0a000072:177b1a588c3:33:0] ERROR  Allocator.java (line 571) allocate block for 5279adf7-16f0-4c47-b6ef-6361dc2b3425 get error, retry # 1
2021-02-17T20:32:23,986 [TaskScheduler-ChunkManager-ParallelExecutor-024-0a000072:177b1a588c3:33:0] ERROR  RequestProcessor.java (line 160) hit exception to create chunk 5279adf7-16f0-4c47-b6ef-6361dc2b3425, request id 0a000072:177b1a780b6:64:114
2021-02-17T20:32:23,986 [TaskScheduler-ChunkManager-ParallelExecutor-024-0a000072:177b1a588c3:33:0] ERROR  RequestProcessor.java (line 160) hit exception to create chunk 5279adf7-16f0-4c47-b6ef-6361dc2b3425, request id 0a000072:177b1a780b6:64:114
2021-02-17T20:32:23,987 [TaskScheduler-RecordManager-ParallelExecutor-001-0a000072:177b1a588c3:33:0] ERROR  ChunkClientImpl.java (line 265) chunk creation failed 5279adf7-16f0-4c47-b6ef-6361dc2b3425, write context: [writerId: 10.0.0.114-1096-DIRECTORY_TABLE-BFW-0-writer-0, cos: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, level: 1, type: DIRECTORY_TABLE, skipChunkSequenceCounter: false, repoMergeChunkId: null], error com.emc.storageos.data.object.exception.ObjectControllerException: create chunk 5279adf7-16f0-4c47-b6ef-6361dc2b3425 failed, request id 0a000072:177b1a780b6:64:114, serverIp 10.0.0.114, status ERROR_NO_STORAGE_DEVICE_FOUND sleep 5000 millis before retry
2021-02-17T20:32:23,987 [TaskScheduler-RecordManager-ParallelExecutor-001-0a000072:177b1a588c3:33:0] ERROR  ChunkClientImpl.java (line 265) chunk creation failed 5279adf7-16f0-4c47-b6ef-6361dc2b3425, write context: [writerId: 10.0.0.114-1096-DIRECTORY_TABLE-BFW-0-writer-0, cos: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, level: 1, type: DIRECTORY_TABLE, skipChunkSequenceCounter: false, repoMergeChunkId: null], error com.emc.storageos.data.object.exception.ObjectControllerException: create chunk 5279adf7-16f0-4c47-b6ef-6361dc2b3425 failed, request id 0a000072:177b1a780b6:64:114, serverIp 10.0.0.114, status ERROR_NO_STORAGE_DEVICE_FOUND sleep 5000 millis before retry
2021-02-17T20:32:24,341 [qtp313945225-373] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: ET, keyHash: 962, level: 0]
2021-02-17T20:32:25,373 [qtp313945225-371] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: ET, keyHash: 3054, level: 0]
2021-02-17T20:32:27,287 [TaskScheduler-ChunkManager-ParallelExecutor-097-0a000072:177b1a588c3:33:0] ERROR  SsSelector.java (line 600) Not enough resources for chunk 1e5ab73d-4a46-4a6c-943b-e5ff74d654b6: cos urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa level 2 to place 1 segments with type SYSTEM
2021-02-17T20:32:27,287 [TaskScheduler-ChunkManager-ParallelExecutor-097-0a000072:177b1a588c3:33:0] ERROR  SsSelector.java (line 600) Not enough resources for chunk 1e5ab73d-4a46-4a6c-943b-e5ff74d654b6: cos urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa level 2 to place 1 segments with type SYSTEM
2021-02-17T20:32:27,287 [TaskScheduler-ChunkManager-ParallelExecutor-097-0a000072:177b1a588c3:33:0] ERROR  Allocator.java (line 571) allocate block for 1e5ab73d-4a46-4a6c-943b-e5ff74d654b6 get error, retry # 1
2021-02-17T20:32:27,287 [TaskScheduler-ChunkManager-ParallelExecutor-097-0a000072:177b1a588c3:33:0] ERROR  Allocator.java (line 571) allocate block for 1e5ab73d-4a46-4a6c-943b-e5ff74d654b6 get error, retry # 1
2021-02-17T20:32:27,288 [TaskScheduler-ChunkManager-ParallelExecutor-097-0a000072:177b1a588c3:33:0] ERROR  RequestProcessor.java (line 160) hit exception to create chunk 1e5ab73d-4a46-4a6c-943b-e5ff74d654b6, request id 0a000072:177b1a78c86:121:10e
2021-02-17T20:32:27,288 [TaskScheduler-ChunkManager-ParallelExecutor-097-0a000072:177b1a588c3:33:0] ERROR  RequestProcessor.java (line 160) hit exception to create chunk 1e5ab73d-4a46-4a6c-943b-e5ff74d654b6, request id 0a000072:177b1a78c86:121:10e
2021-02-17T20:32:27,289 [TaskScheduler-SSManager-ParallelExecutor-017-0a000072:177b1a588c3:33:0] ERROR  ChunkClientImpl.java (line 265) chunk creation failed 1e5ab73d-4a46-4a6c-943b-e5ff74d654b6, write context: [writerId: 10.0.0.114-1095-DIRECTORY_TABLE-BFW-0-writer-0, cos: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, level: 1, type: DIRECTORY_TABLE, skipChunkSequenceCounter: false, repoMergeChunkId: null], error com.emc.storageos.data.object.exception.ObjectControllerException: create chunk 1e5ab73d-4a46-4a6c-943b-e5ff74d654b6 failed, request id 0a000072:177b1a78c86:121:10e, serverIp 10.0.0.114, status ERROR_NO_STORAGE_DEVICE_FOUND sleep 5000 millis before retry
2021-02-17T20:32:27,289 [TaskScheduler-SSManager-ParallelExecutor-017-0a000072:177b1a588c3:33:0] ERROR  ChunkClientImpl.java (line 265) chunk creation failed 1e5ab73d-4a46-4a6c-943b-e5ff74d654b6, write context: [writerId: 10.0.0.114-1095-DIRECTORY_TABLE-BFW-0-writer-0, cos: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, level: 1, type: DIRECTORY_TABLE, skipChunkSequenceCounter: false, repoMergeChunkId: null], error com.emc.storageos.data.object.exception.ObjectControllerException: create chunk 1e5ab73d-4a46-4a6c-943b-e5ff74d654b6 failed, request id 0a000072:177b1a78c86:121:10e, serverIp 10.0.0.114, status ERROR_NO_STORAGE_DEVICE_FOUND sleep 5000 millis before retry
2021-02-17T20:32:27,980 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:27,980 [ZoneScannerImpl-000] ERROR  DirectoryOwnershipRecordMapImpl.java (line 215) Cannot find directory for key: [CoS: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, type: RT, keyHash: 3724, level: 0]
2021-02-17T20:32:28,995 [TaskScheduler-ChunkManager-ParallelExecutor-053-0a000072:177b1a588c3:33:0] ERROR  SsSelector.java (line 600) Not enough resources for chunk 6d941b2a-2bba-4def-a59f-254f5048206e: cos urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa level 2 to place 1 segments with type SYSTEM
2021-02-17T20:32:28,995 [TaskScheduler-ChunkManager-ParallelExecutor-053-0a000072:177b1a588c3:33:0] ERROR  SsSelector.java (line 600) Not enough resources for chunk 6d941b2a-2bba-4def-a59f-254f5048206e: cos urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa level 2 to place 1 segments with type SYSTEM
2021-02-17T20:32:28,995 [TaskScheduler-ChunkManager-ParallelExecutor-053-0a000072:177b1a588c3:33:0] ERROR  Allocator.java (line 571) allocate block for 6d941b2a-2bba-4def-a59f-254f5048206e get error, retry # 1
2021-02-17T20:32:28,995 [TaskScheduler-ChunkManager-ParallelExecutor-053-0a000072:177b1a588c3:33:0] ERROR  Allocator.java (line 571) allocate block for 6d941b2a-2bba-4def-a59f-254f5048206e get error, retry # 1
2021-02-17T20:32:28,996 [TaskScheduler-ChunkManager-ParallelExecutor-053-0a000072:177b1a588c3:33:0] ERROR  RequestProcessor.java (line 160) hit exception to create chunk 6d941b2a-2bba-4def-a59f-254f5048206e, request id 0a000072:177b1a780b6:64:116
2021-02-17T20:32:28,996 [TaskScheduler-ChunkManager-ParallelExecutor-053-0a000072:177b1a588c3:33:0] ERROR  RequestProcessor.java (line 160) hit exception to create chunk 6d941b2a-2bba-4def-a59f-254f5048206e, request id 0a000072:177b1a780b6:64:116
2021-02-17T20:32:28,997 [TaskScheduler-RecordManager-ParallelExecutor-001-0a000072:177b1a588c3:33:0] ERROR  ChunkClientImpl.java (line 265) chunk creation failed 6d941b2a-2bba-4def-a59f-254f5048206e, write context: [writerId: 10.0.0.114-1096-DIRECTORY_TABLE-BFW-0-writer-0, cos: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, level: 1, type: DIRECTORY_TABLE, skipChunkSequenceCounter: false, repoMergeChunkId: null], error com.emc.storageos.data.object.exception.ObjectControllerException: create chunk 6d941b2a-2bba-4def-a59f-254f5048206e failed, request id 0a000072:177b1a780b6:64:116, serverIp 10.0.0.114, status ERROR_NO_STORAGE_DEVICE_FOUND sleep 5000 millis before retry
2021-02-17T20:32:28,997 [TaskScheduler-RecordManager-ParallelExecutor-001-0a000072:177b1a588c3:33:0] ERROR  ChunkClientImpl.java (line 265) chunk creation failed 6d941b2a-2bba-4def-a59f-254f5048206e, write context: [writerId: 10.0.0.114-1096-DIRECTORY_TABLE-BFW-0-writer-0, cos: urn:storageos:VirtualArray:4b52e9ea-d19f-4566-aad6-4a4a328d22aa, level: 1, type: DIRECTORY_TABLE, skipChunkSequenceCounter: false, repoMergeChunkId: null], error com.emc.storageos.data.object.exception.ObjectControllerException: create chunk 6d941b2a-2bba-4def-a59f-254f5048206e failed, request id 0a000072:177b1a780b6:64:116, serverIp 10.0.0.114, status ERROR_NO_STORAGE_DEVICE_FOUND sleep 5000 millis before retry

Notifies: @Oohdark30 @padthaitofuhot

drude-dell commented 3 years ago

What is the fix to address this failure???

mpolitze commented 3 years ago

sadly same here. do you have any new insights on this failure?

(also heappens when building my own containers, see workaround described here https://github.com/EMCECS/ECS-CommunityEdition/issues/534#issuecomment-869845571)

Best I can do is get the management endpoints running commit 9717ae3ebfc00bc1ff97c3bd2c97868faf6203ff but then I cannot get after a storage pool as that is stuck in initializing state.

nikhil-vr commented 2 years ago

Please use the latest version and report if issue persists