Closed infa-rammekal closed 2 years ago
oalbrigt
commented
2 years ago You can skip OCF_CHECK_LEVEL, as that's not used by the agent.
You can disable or create the resource with --disabled, and then run it manually with pcs resource debug-start --full aws_eip.
That way you'll be able to see the commands being run, and then run them manually to see why it's failing.
infa-rammekal
commented
2 years ago here is debug output:
`$ sudo pcs resource create aws_eip ocf:heartbeat:awseip elastic_ip=35.86.13.200 allocation_id=eipalloc- >052c7f5348fc5d26c --group aws_group op monitor timeout="30s" interval="60s" OCF_CHECK_LEVEL="10" --disabled
[rmekala@db25.or2 ~]$ sudo pcs resource debug-start --full aws_eip warning: unpack_config: Blind faith: not fencing unseen nodes warning: unpack_rsc_op_failure: Processing failed probe of aws_eip on db26.or2.omniture.com: unknown error | rc=1 Operation start for aws_eip (ocf:heartbeat:awseip) returned: 'ok' (0) stderr: +++ 08:40:07: ocf_start_trace:999: echo stderr: +++ 08:40:07: ocf_start_trace:999: printenv stderr: +++ 08:40:07: ocf_start_trace:999: sort stderr: ++ 08:40:07: ocf_start_trace:999: env=' stderr: HA_debug=1 stderr: HA_logfacility=none stderr: HOME=/root stderr: HOSTNAME=db25.or2.omniture.com stderr: LC_ALL=C stderr: LOGNAME=root stderr: LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:.tar=38;5;9:.tgz=38;5;9:.arc=38;5;9:.arj=38;5;9:.taz=38;5;9:.lha=38;5;9:.lz4=38;5;9:.lzh=38;5;9:.lzma=38;5;9:.tlz=38;5;9:.txz=38;5;9:.tzo=38;5;9:.t7z=38;5;9:.zip=38;5;9:.z=38;5;9:.Z=38;5;9:.dz=38;5;9:.gz=38;5;9:.lrz=38;5;9:.lz=38;5;9:.lzo=38;5;9:.xz=38;5;9:.bz2=38;5;9:.bz=38;5;9:.tbz=38;5;9:.tbz2=38;5;9:.tz=38;5;9:.deb=38;5;9:.rpm=38;5;9:.jar=38;5;9:.war=38;5;9:.ear=38;5;9:.sar=38;5;9:.rar=38;5;9:.alz=38;5;9:.ace=38;5;9:.zoo=38;5;9:.cpio=38;5;9:.7z=38;5;9:.rz=38;5;9:.cab=38;5;9:.jpg=38;5;13:.jpeg=38;5;13:.gif=38;5;13:.bmp=38;5;13:.pbm=38;5;13:.pgm=38;5;13:.ppm=38;5;13:.tga=38;5;13:.xbm=38;5;13:.xpm=38;5;13:.tif=38;5;13:.tiff=38;5;13:.png=38;5;13:.svg=38;5;13:.svgz=38;5;13:.mng=38;5;13:.pcx=38;5;13:.mov=38;5;13:.mpg=38;5;13:.mpeg=38;5;13:.m2v=38;5;13:.mkv=38;5;13:.webm=38;5;13:.ogm=38;5;13:.mp4=38;5;13:.m4v=38;5;13:.mp4v=38;5;13:.vob=38;5;13:.qt=38;5;13:.nuv=38;5;13:.wmv=38;5;13:.asf=38;5;13:.rm=38;5;13:.rmvb=38;5;13:.flc=38;5;13:.avi=38;5;13:.fli=38;5;13:.flv=38;5;13:.gl=38;5;13:.dl=38;5;13:.xcf=38;5;13:.xwd=38;5;13:.yuv=38;5;13:.cgm=38;5;13:.emf=38;5;13:.axv=38;5;13:.anx=38;5;13:.ogv=38;5;13:.ogx=38;5;13:.aac=38;5;45:.au=38;5;45:.flac=38;5;45:.mid=38;5;45:.midi=38;5;45:.mka=38;5;45:.mp3=38;5;45:.mpc=38;5;45:.ogg=38;5;45:.ra=38;5;45:.wav=38;5;45:.axa=38;5;45:.oga=38;5;45:.spx=38;5;45:.xspf=38;5;45: stderr: MAIL=/var/mail/root stderr: OCF_EXIT_REASON_PREFIX=ocf-exit-reason: stderr: OCF_RA_VERSION_MAJOR=1 stderr: OCF_RA_VERSION_MINOR=0 stderr: OCF_RESKEY_CRM_meta_class=ocf stderr: OCF_RESKEY_CRM_meta_id=aws_eip stderr: OCF_RESKEY_CRM_meta_provider=heartbeat stderr: OCF_RESKEY_CRM_meta_resource_stickiness=100 stderr: OCF_RESKEY_CRM_meta_target_role=Stopped stderr: OCF_RESKEY_CRM_meta_timeout=30000 stderr: OCF_RESKEY_CRM_meta_type=awseip stderr: OCF_RESKEY_allocation_id=eipalloc-052c7f5348fc5d26c stderr: OCF_RESKEY_crm_feature_set=3.0.14 stderr: OCF_RESKEY_elastic_ip=35.86.13.200 stderr: OCF_RESOURCE_INSTANCE=aws_eip stderr: OCF_RESOURCE_PROVIDER=heartbeat stderr: OCF_RESOURCE_TYPE=awseip stderr: OCF_ROOT=/usr/lib/ocf stderr: OCF_TRACE_FILE=/dev/stderr stderr: OCF_TRACE_RA=1 stderr: PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/rmekala/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/ucb stderr: PCMK_logfacility=none stderr: PCMK_service=crm_resource stderr: PWD=/home/rmekala stderr: SHELL=/bin/bash stderr: SHLVL=1 stderr: SUDO_COMMAND=/usr/sbin/pcs resource debug-start --full aws_eip stderr: SUDO_GID=100 stderr: SUDO_UID=122820 stderr: SUDO_USER=rmekala stderr: TERM=xterm-256color stderr: USER=root stderr: USERNAME=root stderr: XDG_SESSIONID=21377 stderr: =/usr/bin/printenv stderr: OCF_TRC_DEST=/dev/stderr stderr: OCF_TRC_MANAGE=' stderr: ++ 08:40:07: 1053: ocf_is_true '' stderr: ++ 08:40:07: ocf_is_true:103: case "$1" in stderr: ++ 08:40:07: ocf_is_true:103: case "$1" in stderr: ++ 08:40:07: ocf_is_true:105: false stderr: + 08:40:07: 45: OCF_RESKEY_awscli_default=/usr/bin/aws stderr: + 08:40:07: 46: OCF_RESKEY_profile_default=default stderr: + 08:40:07: 47: OCF_RESKEY_api_delay_default=3 stderr: + 08:40:07: 49: : /usr/bin/aws stderr: + 08:40:07: 50: : default stderr: + 08:40:07: 51: : 3 stderr: + 08:40:07: 235: case $OCF_ACTION in stderr: + 08:40:07: 242: AWSCLI=/usr/bin/aws stderr: + 08:40:07: 243: ELASTIC_IP=35.86.13.200 stderr: + 08:40:07: 244: ALLOCATION_ID=eipalloc-052c7f5348fc5d26c stderr: + 08:40:07: 245: PRIVATE_IP_ADDRESS= stderr: ++ 08:40:07: 246: curl -X PUT http://169.254.169.254/latest/api/token -H 'X-aws-ec2-metadata-token-ttl-seconds: 21600' stderr: % Total % Received % Xferd Average Speed Time Time Time Current stderr: Dload Upload Total Spent Left Speed 100 56 100 56 0 0 32332 0 --:--:-- --:--:-- --:--:-- 56000 stderr: + 08:40:07: 246: TOKEN=xxxxxxxxxxxxxxxxxxxx stderr: ++ 08:40:07: 247: curl -s http://169.254.169.254/latest/meta-data/instance-id -H 'X-aws-ec2-metadata-token: xxxxxxxxxxxxxxxxxxxxxxx' stderr: + 08:40:07: 247: INSTANCE_ID=i-07b8432a9dc874ef4 stderr: + 08:40:07: 249: case $OCF_ACTION in stderr: + 08:40:07: 251: awseip_validate stderr: + 08:40:07: awseip_validate:220: check_binary /usr/bin/aws stderr: + 08:40:07: check_binary:55: have_binary /usr/bin/aws stderr: + 08:40:07: have_binary:67: '[' '' = 1 ']' stderr: ++ 08:40:07: have_binary:70: echo /usr/bin/aws stderr: ++ 08:40:07: have_binary:70: sed -e 's/ -.//' stderr: + 08:40:07: have_binary:70: local bin=/usr/bin/aws stderr: ++ 08:40:07: have_binary:71: which /usr/bin/aws stderr: + 08:40:07: have_binary:71: test -x /usr/bin/aws stderr: + 08:40:07: awseip_validate:222: '[' -z default ']' stderr: + 08:40:07: awseip_validate:227: '[' -z i-07b8432a9dc874ef4 ']' stderr: + 08:40:07: awseip_validate:232: return 0 stderr: + 08:40:07: 252: awseip_start stderr: + 08:40:07: awseip_start:148: awseip_monitor stderr: + 08:40:07: awseip_monitor:210: /usr/bin/aws --profile default ec2 describe-instances --instance-id i-07b8432a9dc874ef4 stderr: + 08:40:07: awseip_monitor:210: grep -q 35.86.13.200 stderr: + 08:40:08: awseip_monitor:211: RET=0 stderr: + 08:40:08: awseip_monitor:213: '[' 0 -ne 0 ']' stderr: + 08:40:08: awseip_monitor:216: return 0 stderr: + 08:40:08: awseip_start:148: return 0 stderr: + 08:40:08: 284: rc=0 stderr: + 08:40:08: 285: ocf_log debug 'aws_eip start : 0' stderr: + 08:40:08: ocf_log:323: '[' 2 -lt 2 ']' stderr: + 08:40:08: ocf_log:327: OCF_PRIO=debug stderr: + 08:40:08: ocf_log:328: shift stderr: + 08:40:08: ocf_log:329: __OCF_MSG='aws_eip start : 0' stderr: + 08:40:08: ocf_log:331: case "${OCF_PRIO}" in stderr: + 08:40:08: ocf_log:336: __OCF_PRIO=DEBUG stderr: + 08:40:08: ocf_log:340: '[' DEBUG = DEBUG ']' stderr: + 08:40:08: ocf_log:341: ha_debug 'DEBUG: aws_eip start : 0' stderr: + 08:40:08: ha_debug:260: '[' x1 = x0 ']' stderr: + 08:40:08: ha_debug:260: '[' -z 1 ']' stderr: + 08:40:08: ha_debug:263: tty stderr: + 08:40:08: ha_debug:272: set_logtag stderr: + 08:40:08: set_logtag:177: '[' -z '' ']' stderr: + 08:40:08: set_logtag:178: '[' -n aws_eip ']' stderr: + 08:40:08: set_logtag:179: HA_LOGTAG='awseip(aws_eip)[4341]' stderr: + 08:40:08: ha_debug:274: '[' x = xyes ']' stderr: + 08:40:08: ha_debug:281: '[' none = '' ']' stderr: + 08:40:08: ha_debug:284: '[' -n '' ']' stderr: + 08:40:08: ha_debug:290: '[' -n /dev/null ']' stderr: + 08:40:08: ha_debug:292: : appending to /dev/null stderr: ++ 08:40:08: ha_debug:293: hadate stderr: ++ 08:40:08: hadate:173: date '+%b %d %T ' stderr: + 08:40:08: ha_debug:293: echo 'awseip(aws_eip)[4341]: Jul' 25 08:40:08 'DEBUG: aws_eip start : 0' stderr: + 08:40:08: ha_debug:296: '[' -z '' -a -z /dev/null ']' stderr: + 08:40:08: 286: exit 0`
oalbrigt
commented
2 years ago It seems like it's the stop-action that fails in your tests, so try running debug-stop --full to see what happens. I dont see any issues in the start-action.
infa-rammekal
commented
2 years ago Here is output:
$ sudo pcs resource create aws_eip ocf:heartbeat:awseip elastic_ip=35.86.13.200 allocation_id=eipalloc-052c7f5348fc5d26c --group aws_group op monitor timeout="30s" interval="60s" OCF_CHECK_LEVEL="10" --disabled [rmekala@db25.or2 /usr/local/ccc]$ sudo pcs resource debug-stop --full aws_eip warning: unpack_config: Blind faith: not fencing unseen nodes Operation stop for aws_eip (ocf:heartbeat:awseip) returned: 'ok' (0)
stderr: +++ 07:37:54: ocf_start_trace:999: echo stderr: +++ 07:37:54: ocf_start_trace:999: sort stderr: +++ 07:37:54: ocf_start_trace:999: printenv stderr: ++ 07:37:54: ocf_start_trace:999: env=' stderr: HA_debug=1 stderr: HA_logfacility=none stderr: HOME=/root stderr: HOSTNAME=db25.or2.omniture.com stderr: LC_ALL=C stderr: LOGNAME=root stderr: LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:.tar=38;5;9:.tgz=38;5;9:.arc=38;5;9:.arj=38;5;9:.taz=38;5;9:.lha=38;5;9:.lz4=38;5;9:.lzh=38;5;9:.lzma=38;5;9:.tlz=38;5;9:.txz=38;5;9:.tzo=38;5;9:.t7z=38;5;9:.zip=38;5;9:.z=38;5;9:.Z=38;5;9:.dz=38;5;9:.gz=38;5;9:.lrz=38;5;9:.lz=38;5;9:.lzo=38;5;9:.xz=38;5;9:.bz2=38;5;9:.bz=38;5;9:.tbz=38;5;9:.tbz2=38;5;9:.tz=38;5;9:.deb=38;5;9:.rpm=38;5;9:.jar=38;5;9:.war=38;5;9:.ear=38;5;9:.sar=38;5;9:.rar=38;5;9:.alz=38;5;9:.ace=38;5;9:.zoo=38;5;9:.cpio=38;5;9:.7z=38;5;9:.rz=38;5;9:.cab=38;5;9:.jpg=38;5;13:.jpeg=38;5;13:.gif=38;5;13:.bmp=38;5;13:.pbm=38;5;13:.pgm=38;5;13:.ppm=38;5;13:.tga=38;5;13:.xbm=38;5;13:.xpm=38;5;13:.tif=38;5;13:.tiff=38;5;13:.png=38;5;13:.svg=38;5;13:.svgz=38;5;13:.mng=38;5;13:.pcx=38;5;13:.mov=38;5;13:.mpg=38;5;13:.mpeg=38;5;13:.m2v=38;5;13:.mkv=38;5;13:.webm=38;5;13:.ogm=38;5;13:.mp4=38;5;13:.m4v=38;5;13:.mp4v=38;5;13:.vob=38;5;13:.qt=38;5;13:.nuv=38;5;13:.wmv=38;5;13:.asf=38;5;13:.rm=38;5;13:.rmvb=38;5;13:.flc=38;5;13:.avi=38;5;13:.fli=38;5;13:.flv=38;5;13:.gl=38;5;13:.dl=38;5;13:.xcf=38;5;13:.xwd=38;5;13:.yuv=38;5;13:.cgm=38;5;13:.emf=38;5;13:.axv=38;5;13:.anx=38;5;13:.ogv=38;5;13:.ogx=38;5;13:.aac=38;5;45:.au=38;5;45:.flac=38;5;45:.mid=38;5;45:.midi=38;5;45:.mka=38;5;45:.mp3=38;5;45:.mpc=38;5;45:.ogg=38;5;45:.ra=38;5;45:.wav=38;5;45:.axa=38;5;45:.oga=38;5;45:.spx=38;5;45:*.xspf=38;5;45: stderr: MAIL=/var/mail/root stderr: OCF_EXIT_REASON_PREFIX=ocf-exit-reason: stderr: OCF_RA_VERSION_MAJOR=1 stderr: OCF_RA_VERSION_MINOR=0 stderr: OCF_RESKEY_CRM_meta_class=ocf stderr: OCF_RESKEY_CRM_meta_id=aws_eip stderr: OCF_RESKEY_CRM_meta_provider=heartbeat stderr: OCF_RESKEY_CRM_meta_resource_stickiness=100 stderr: OCF_RESKEY_CRM_meta_target_role=Stopped stderr: OCF_RESKEY_CRM_meta_timeout=30000 stderr: OCF_RESKEY_CRM_meta_type=awseip stderr: OCF_RESKEY_allocation_id=eipalloc-052c7f5348fc5d26c stderr: OCF_RESKEY_crm_feature_set=3.0.14 stderr: OCF_RESKEY_elastic_ip=35.86.13.200 stderr: OCF_RESOURCE_INSTANCE=aws_eip stderr: OCF_RESOURCE_PROVIDER=heartbeat stderr: OCF_RESOURCE_TYPE=awseip stderr: OCF_ROOT=/usr/lib/ocf stderr: OCF_TRACE_FILE=/dev/stderr stderr: OCF_TRACE_RA=1 stderr: PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/rmekala/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/ucb stderr: PCMK_logfacility=none stderr: PCMK_service=crm_resource stderr: PWD=/usr/local/ccc stderr: SHELL=/bin/bash stderr: SHLVL=1 stderr: SUDO_COMMAND=/usr/sbin/pcs resource debug-stop --full aws_eip stderr: SUDO_GID=100 stderr: SUDO_UID=122820 stderr: SUDO_USER=rmekala stderr: TERM=xterm-256color stderr: USER=root stderr: USERNAME=root stderr: XDG_SESSIONID=24318 stderr: =/usr/bin/printenv stderr: OCF_TRC_DEST=/dev/stderr stderr: OCF_TRC_MANAGE=' stderr: ++ 07:37:54: 1053: ocf_is_true '' stderr: ++ 07:37:54: ocf_is_true:103: case "$1" in stderr: ++ 07:37:54: ocf_is_true:103: case "$1" in stderr: ++ 07:37:54: ocf_is_true:105: false stderr: + 07:37:54: 45: OCF_RESKEY_awscli_default=/usr/bin/aws stderr: + 07:37:54: 46: OCF_RESKEY_profile_default=default stderr: + 07:37:54: 47: OCF_RESKEY_api_delay_default=3 stderr: + 07:37:54: 49: : /usr/bin/aws stderr: + 07:37:54: 50: : default stderr: + 07:37:54: 51: : 3 stderr: + 07:37:54: 235: case $OCF_ACTION in stderr: + 07:37:54: 242: AWSCLI=/usr/bin/aws stderr: + 07:37:54: 243: ELASTIC_IP=35.86.13.200 stderr: + 07:37:54: 244: ALLOCATION_ID=eipalloc-052c7f5348fc5d26c stderr: + 07:37:54: 245: PRIVATE_IP_ADDRESS= stderr: ++ 07:37:54: 246: curl -X PUT http://169.254.169.254/latest/api/token -H 'X-aws-ec2-metadata-token-ttl-seconds: 21600' stderr: % Total % Received % Xferd Average Speed Time Time Time Current stderr: Dload Upload Total Spent Left Speed 100 56 100 56 0 0 13251 0 --:--:-- --:--:-- --:--:-- 56000 stderr: + 07:37:54: 246: TOKEN=AQAEAF3fVArE_aLjIkHzsl5m7Sa7JhYP_B8cYkqpfoAhm1Z9fml08w== stderr: ++ 07:37:54: 247: curl -s http://169.254.169.254/latest/meta-data/instance-id -H 'X-aws-ec2-metadata-token: xxxxxxxxxxx' stderr: + 07:37:54: 247: INSTANCE_ID=i-07b8432a9dc874ef4 stderr: + 07:37:54: 249: case $OCF_ACTION in stderr: + 07:37:54: 255: awseip_stop stderr: + 07:37:54: awseip_stop:183: awseip_monitor stderr: + 07:37:54: awseip_monitor:210: /usr/bin/aws --profile default ec2 describe-instances --instance-id i-07b8432a9dc874ef4 stderr: + 07:37:54: awseip_monitor:210: grep -q 35.86.13.200 stderr: + 07:37:55: awseip_monitor:211: RET=0 stderr: + 07:37:55: awseip_monitor:213: '[' 0 -ne 0 ']' stderr: + 07:37:55: awseip_monitor:216: return 0 stderr: ++ 07:37:55: awseip_stop:186: /usr/bin/aws --profile default --output json ec2 describe-addresses --allocation-id eipalloc-052c7f5348fc5d26c stderr: ++ 07:37:55: awseip_stop:186: grep -m 1 AssociationId stderr: ++ 07:37:55: awseip_stop:186: awk '-F"' '{print$4}' stderr: + 07:37:55: awseip_stop:186: ASSOCIATION_ID=eipassoc-0ca34af008b64f4d5 stderr: + 07:37:55: awseip_stop:188: '[' -z eipassoc-0ca34af008b64f4d5 ']' stderr: + 07:37:55: awseip_stop:193: /usr/bin/aws --profile default ec2 disassociate-address --association-id eipassoc-0ca34af008b64f4d5 stderr: + 07:37:57: awseip_stop:196: RET=0 stderr: + 07:37:57: awseip_stop:199: sleep 3 stderr: + 07:38:00: awseip_stop:201: '[' 0 -ne 0 ']' stderr: + 07:38:00: awseip_stop:205: ocf_log info 'elastic_ip has been successfully brought down (35.86.13.200)' stderr: + 07:38:00: ocf_log:323: '[' 2 -lt 2 ']' stderr: + 07:38:00: ocf_log:327: OCF_PRIO=info stderr: + 07:38:00: ocf_log:328: shift stderr: + 07:38:00: ocf_log:329: __OCF_MSG='elastic_ip has been successfully brought down (35.86.13.200)' stderr: + 07:38:00: ocf_log:331: case "${OCF_PRIO}" in stderr: + 07:38:00: ocf_log:335: OCF_PRIO=INFO stderr: + 07:38:00: ocf_log:340: '[' INFO = DEBUG ']' stderr: + 07:38:00: ocf_log:343: ha_log 'INFO: elastic_ip has been successfully brought down (35.86.13.200)' stderr: + 07:38:00: ha_log:255: ha_log 'INFO: elastic_ip has been successfully brought down (35.86.13.200)' stderr: + 07:38:00: ha_log:187: local ignore_stderr=false stderr: + 07:38:00: ha_log:188: local loglevel stderr: + 07:38:00: ha_log:190: '[' 'xINFO: elastic_ip has been successfully brought down (35.86.13.200)' = x--ignore-stderr ']' stderr: + 07:38:00: ha_log:192: '[' none = '' ']' stderr: + 07:38:00: ha_log:194: tty stderr: + 07:38:00: ha_log:209: set_logtag stderr: + 07:38:00: set_logtag:177: '[' -z '' ']' stderr: + 07:38:00: set_logtag:178: '[' -n aws_eip ']' stderr: + 07:38:00: set_logtag:179: HA_LOGTAG='awseip(aws_eip)[15683]' stderr: + 07:38:00: ha_log:211: '[' x = xyes ']' stderr: + 07:38:00: ha_log:219: '[' -n '' ']' stderr: + 07:38:00: ha_log:232: '[' -n '' ']' stderr: + 07:38:00: __ha_log:238: '[' -z '' -a -z '' ']' stderr: + 07:38:00: ha_log:238: '[' false = true ']' stderr: + 07:38:00: ha_log:240: : appending to stderr stderr: ++ 07:38:00: __ha_log:241: hadate stderr: ++ 07:38:00: hadate:173: date '+%b %d %T ' stderr: + 07:38:00: ha_log:241: echo Jul 26 07:38:00 'INFO: elastic_ip has been successfully brought down (35.86.13.200)' stderr: Jul 26 07:38:00 INFO: elastic_ip has been successfully brought down (35.86.13.200) stderr: + 07:38:00: ha_log:244: '[' -n /dev/null ']' stderr: + 07:38:00: __ha_log:246: : appending to /dev/null stderr: + 07:38:00: ha_log:247: '[' x '!=' /dev/nullx ']' stderr: ++ 07:38:00: ha_log:248: hadate stderr: ++ 07:38:00: hadate:173: date '+%b %d %T ' stderr: + 07:38:00: ha_log:248: echo 'awseip(aws_eip)[15683]: Jul' 26 07:38:00 'INFO: elastic_ip has been successfully brought down (35.86.13.200)' stderr: + 07:38:00: awseip_stop:206: return 0 stderr: + 07:38:00: 284: rc=0 stderr: + 07:38:00: 285: ocf_log debug 'aws_eip stop : 0' stderr: + 07:38:00: ocf_log:323: '[' 2 -lt 2 ']' stderr: + 07:38:00: ocf_log:327: OCF_PRIO=debug stderr: + 07:38:00: ocf_log:328: shift stderr: + 07:38:00: ocf_log:329: OCF_MSG='aws_eip stop : 0' stderr: + 07:38:00: ocf_log:331: case "${OCF_PRIO}" in stderr: + 07:38:00: ocf_log:336: OCF_PRIO=DEBUG stderr: + 07:38:00: ocf_log:340: '[' DEBUG = DEBUG ']' stderr: + 07:38:00: ocf_log:341: ha_debug 'DEBUG: aws_eip stop : 0' stderr: + 07:38:00: ha_debug:260: '[' x1 = x0 ']' stderr: + 07:38:00: ha_debug:260: '[' -z 1 ']' stderr: + 07:38:00: ha_debug:263: tty stderr: + 07:38:00: ha_debug:272: set_logtag stderr: + 07:38:00: set_logtag:177: '[' -z 'awseip(aws_eip)[15683]' ']' stderr: + 07:38:00: ha_debug:274: '[' x = xyes ']' stderr: + 07:38:00: ha_debug:281: '[' none = '' ']' stderr: + 07:38:00: ha_debug:284: '[' -n '' ']' stderr: + 07:38:00: ha_debug:290: '[' -n /dev/null ']' stderr: + 07:38:00: ha_debug:292: : appending to /dev/null stderr: ++ 07:38:00: ha_debug:293: hadate stderr: ++ 07:38:00: hadate:173: date '+%b %d %T ' stderr: + 07:38:00: ha_debug:293: echo 'awseip(aws_eip)[15683]: Jul' 26 07:38:00 'DEBUG: aws_eip stop : 0' stderr: + 07:38:00: ha_debug:296: '[' -z '' -a -z /dev/null ']' stderr: + 07:38:00: 286: exit 0
still seeing the error: Stack: corosync Current DC: db26.or2.omniture.com (version 1.1.23-1.el7_9.1-9acf116022) - partition with quorum Last updated: Tue Jul 26 07:43:28 2022 Last change: Tue Jul 26 07:40:00 2022 by hacluster via crmd on db26.or2.omniture.com
2 nodes configured 4 resource instances configured (1 DISABLED, 1 BLOCKED from further action due to failure)
Online: [ db25.or2.omniture.com db26.or2.omniture.com ]
Active resources:
Master/Slave Set: ms_mysql [p_mysql] Masters: [ db25.or2.omniture.com ] Slaves: [ db26.or2.omniture.com ] Resource Group: aws_group aws_eip (ocf::heartbeat:awseip): FAILED (disabled, blocked)[ db25.or2.omniture.com db26.or2.omniture.com ]
Failed Resource Actions:
infa-rammekal
commented
2 years ago if it is ok. we can have a quick zoom session.
oalbrigt
commented
2 years ago Strange. Try adding trace_ra=1 to your pcs resource create line.
Then you should be able to find the output in /var/lib/heartbeat/trace_ra/ for the specific actions, so in this case the stop-action.
infa-rammekal
commented
2 years ago sudo pcs resource create aws_eip ocf:heartbeat:awseip trace_ra=1 trace_file=/tmp/trace elastic_ip=35.86.13.200 allocation_id=eipalloc-052c7f5348fc5d26c --group aws_group op monitor timeout="30s" interval="60s" OCF_CHECK_LEVEL="10" $ sudo pcs resource debug-stop --full aws_eip warning: unpack_config: Blind faith: not fencing unseen nodes Operation stop for aws_eip (ocf:heartbeat:awseip) returned: 'ok' (0)
stderr: +++ 08:14:36: ocf_start_trace:999: echo stderr: +++ 08:14:36: ocf_start_trace:999: printenv stderr: +++ 08:14:36: ocf_start_trace:999: sort stderr: ++ 08:14:36: ocf_start_trace:999: env=' stderr: HA_debug=1 stderr: HA_logfacility=none stderr: HOME=/root stderr: HOSTNAME=db25.or2.omniture.com stderr: LC_ALL=C stderr: LOGNAME=root stderr: LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:.tar=38;5;9:.tgz=38;5;9:.arc=38;5;9:.arj=38;5;9:.taz=38;5;9:.lha=38;5;9:.lz4=38;5;9:.lzh=38;5;9:.lzma=38;5;9:.tlz=38;5;9:.txz=38;5;9:.tzo=38;5;9:.t7z=38;5;9:.zip=38;5;9:.z=38;5;9:.Z=38;5;9:.dz=38;5;9:.gz=38;5;9:.lrz=38;5;9:.lz=38;5;9:.lzo=38;5;9:.xz=38;5;9:.bz2=38;5;9:.bz=38;5;9:.tbz=38;5;9:.tbz2=38;5;9:.tz=38;5;9:.deb=38;5;9:.rpm=38;5;9:.jar=38;5;9:.war=38;5;9:.ear=38;5;9:.sar=38;5;9:.rar=38;5;9:.alz=38;5;9:.ace=38;5;9:.zoo=38;5;9:.cpio=38;5;9:.7z=38;5;9:.rz=38;5;9:.cab=38;5;9:.jpg=38;5;13:.jpeg=38;5;13:.gif=38;5;13:.bmp=38;5;13:.pbm=38;5;13:.pgm=38;5;13:.ppm=38;5;13:.tga=38;5;13:.xbm=38;5;13:.xpm=38;5;13:.tif=38;5;13:.tiff=38;5;13:.png=38;5;13:.svg=38;5;13:.svgz=38;5;13:.mng=38;5;13:.pcx=38;5;13:.mov=38;5;13:.mpg=38;5;13:.mpeg=38;5;13:.m2v=38;5;13:.mkv=38;5;13:.webm=38;5;13:.ogm=38;5;13:.mp4=38;5;13:.m4v=38;5;13:.mp4v=38;5;13:.vob=38;5;13:.qt=38;5;13:.nuv=38;5;13:.wmv=38;5;13:.asf=38;5;13:.rm=38;5;13:.rmvb=38;5;13:.flc=38;5;13:.avi=38;5;13:.fli=38;5;13:.flv=38;5;13:.gl=38;5;13:.dl=38;5;13:.xcf=38;5;13:.xwd=38;5;13:.yuv=38;5;13:.cgm=38;5;13:.emf=38;5;13:.axv=38;5;13:.anx=38;5;13:.ogv=38;5;13:.ogx=38;5;13:.aac=38;5;45:.au=38;5;45:.flac=38;5;45:.mid=38;5;45:.midi=38;5;45:.mka=38;5;45:.mp3=38;5;45:.mpc=38;5;45:.ogg=38;5;45:.ra=38;5;45:.wav=38;5;45:.axa=38;5;45:.oga=38;5;45:.spx=38;5;45:*.xspf=38;5;45: stderr: MAIL=/var/mail/root stderr: OCF_EXIT_REASON_PREFIX=ocf-exit-reason: stderr: OCF_RA_VERSION_MAJOR=1 stderr: OCF_RA_VERSION_MINOR=0 stderr: OCF_RESKEY_CRM_meta_class=ocf stderr: OCF_RESKEY_CRM_meta_id=aws_eip stderr: OCF_RESKEY_CRM_meta_provider=heartbeat stderr: OCF_RESKEY_CRM_meta_resource_stickiness=100 stderr: OCF_RESKEY_CRM_meta_timeout=30000 stderr: OCF_RESKEY_CRM_meta_type=awseip stderr: OCF_RESKEY_allocation_id=eipalloc-052c7f5348fc5d26c stderr: OCF_RESKEY_crm_feature_set=3.0.14 stderr: OCF_RESKEY_elastic_ip=35.86.13.200 stderr: OCF_RESKEY_trace_file=/tmp/trace stderr: OCF_RESKEY_trace_ra=1 stderr: OCF_RESOURCE_INSTANCE=aws_eip stderr: OCF_RESOURCE_PROVIDER=heartbeat stderr: OCF_RESOURCE_TYPE=awseip stderr: OCF_ROOT=/usr/lib/ocf stderr: OCF_TRACE_FILE=/dev/stderr stderr: OCF_TRACE_RA=1 stderr: PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/rmekala/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/ucb stderr: PCMK_logfacility=none stderr: PCMK_service=crm_resource stderr: PWD=/usr/local/ccc stderr: SHELL=/bin/bash stderr: SHLVL=1 stderr: SUDO_COMMAND=/usr/sbin/pcs resource debug-stop --full aws_eip stderr: SUDO_GID=100 stderr: SUDO_UID=122820 stderr: SUDO_USER=rmekala stderr: TERM=xterm-256color stderr: USER=root stderr: USERNAME=root stderr: XDG_SESSIONID=24318 stderr: =/usr/bin/printenv stderr: OCF_TRC_DEST=/dev/stderr stderr: OCF_TRC_MANAGE=' stderr: ++ 08:14:36: 1053: ocf_is_true '' stderr: ++ 08:14:36: ocf_is_true:103: case "$1" in stderr: ++ 08:14:36: ocf_is_true:103: case "$1" in stderr: ++ 08:14:36: ocf_is_true:105: false stderr: + 08:14:36: 45: OCF_RESKEY_awscli_default=/usr/bin/aws stderr: + 08:14:36: 46: OCF_RESKEY_profile_default=default stderr: + 08:14:36: 47: OCF_RESKEY_api_delay_default=3 stderr: + 08:14:36: 49: : /usr/bin/aws stderr: + 08:14:36: 50: : default stderr: + 08:14:36: 51: : 3 stderr: + 08:14:36: 235: case $OCF_ACTION in stderr: + 08:14:36: 242: AWSCLI=/usr/bin/aws stderr: + 08:14:36: 243: ELASTIC_IP=35.86.13.200 stderr: + 08:14:36: 244: ALLOCATION_ID=eipalloc-052c7f5348fc5d26c stderr: + 08:14:36: 245: PRIVATE_IP_ADDRESS= stderr: ++ 08:14:36: 246: curl -X PUT http://169.254.169.254/latest/api/token -H 'X-aws-ec2-metadata-token-ttl-seconds: 21600' stderr: % Total % Received % Xferd Average Speed Time Time Time Current stderr: Dload Upload Total Spent Left Speed 100 56 100 56 0 0 59637 0 --:--:-- --:--:-- --:--:-- 56000 stderr: + 08:14:36: 246: TOKEN=AQAEAF3fVApMwJnyZHsz2VJHRj-5B-BDZo9h2xHPBPaZ3s2gQgAdxQ== stderr: ++ 08:14:36: 247: curl -s http://169.254.169.254/latest/meta-data/instance-id -H 'X-aws-ec2-metadata-token: xxxxxxxx' stderr: + 08:14:36: 247: INSTANCE_ID=i-07b8432a9dc874ef4 stderr: + 08:14:36: 249: case $OCF_ACTION in stderr: + 08:14:36: 255: awseip_stop stderr: + 08:14:36: awseip_stop:183: awseip_monitor stderr: + 08:14:36: awseip_monitor:210: grep -q 35.86.13.200 stderr: + 08:14:36: awseip_monitor:210: /usr/bin/aws --profile default ec2 describe-instances --instance-id i-07b8432a9dc874ef4 stderr: + 08:14:37: awseip_monitor:211: RET=0 stderr: + 08:14:37: awseip_monitor:213: '[' 0 -ne 0 ']' stderr: + 08:14:37: awseip_monitor:216: return 0 stderr: ++ 08:14:37: awseip_stop:186: grep -m 1 AssociationId stderr: ++ 08:14:37: awseip_stop:186: awk '-F"' '{print$4}' stderr: ++ 08:14:37: awseip_stop:186: /usr/bin/aws --profile default --output json ec2 describe-addresses --allocation-id eipalloc-052c7f5348fc5d26c stderr: + 08:14:38: awseip_stop:186: ASSOCIATION_ID=eipassoc-0be60f080840dc96e stderr: + 08:14:38: awseip_stop:188: '[' -z eipassoc-0be60f080840dc96e ']' stderr: + 08:14:38: awseip_stop:193: /usr/bin/aws --profile default ec2 disassociate-address --association-id eipassoc-0be60f080840dc96e stderr: + 08:14:40: awseip_stop:196: RET=0 stderr: + 08:14:40: awseip_stop:199: sleep 3 stderr: + 08:14:43: awseip_stop:201: '[' 0 -ne 0 ']' stderr: + 08:14:43: awseip_stop:205: ocf_log info 'elastic_ip has been successfully brought down (35.86.13.200)' stderr: + 08:14:43: ocf_log:323: '[' 2 -lt 2 ']' stderr: + 08:14:43: ocf_log:327: OCF_PRIO=info stderr: + 08:14:43: ocf_log:328: shift stderr: + 08:14:43: ocf_log:329: __OCF_MSG='elastic_ip has been successfully brought down (35.86.13.200)' stderr: + 08:14:43: ocf_log:331: case "${OCF_PRIO}" in stderr: + 08:14:43: ocf_log:335: OCF_PRIO=INFO stderr: + 08:14:43: ocf_log:340: '[' INFO = DEBUG ']' stderr: + 08:14:43: ocf_log:343: ha_log 'INFO: elastic_ip has been successfully brought down (35.86.13.200)' stderr: + 08:14:43: ha_log:255: ha_log 'INFO: elastic_ip has been successfully brought down (35.86.13.200)' stderr: + 08:14:43: ha_log:187: local ignore_stderr=false stderr: + 08:14:43: ha_log:188: local loglevel stderr: + 08:14:43: ha_log:190: '[' 'xINFO: elastic_ip has been successfully brought down (35.86.13.200)' = x--ignore-stderr ']' stderr: + 08:14:43: ha_log:192: '[' none = '' ']' stderr: + 08:14:43: ha_log:194: tty stderr: + 08:14:43: ha_log:209: set_logtag stderr: + 08:14:43: set_logtag:177: '[' -z '' ']' stderr: + 08:14:43: set_logtag:178: '[' -n aws_eip ']' stderr: + 08:14:43: set_logtag:179: HA_LOGTAG='awseip(aws_eip)[29027]' stderr: + 08:14:43: ha_log:211: '[' x = xyes ']' stderr: + 08:14:43: ha_log:219: '[' -n '' ']' stderr: + 08:14:43: ha_log:232: '[' -n '' ']' stderr: + 08:14:43: __ha_log:238: '[' -z '' -a -z '' ']' stderr: + 08:14:43: ha_log:238: '[' false = true ']' stderr: + 08:14:43: ha_log:240: : appending to stderr stderr: ++ 08:14:43: __ha_log:241: hadate stderr: ++ 08:14:43: hadate:173: date '+%b %d %T ' stderr: + 08:14:43: ha_log:241: echo Jul 26 08:14:43 'INFO: elastic_ip has been successfully brought down (35.86.13.200)' stderr: Jul 26 08:14:43 INFO: elastic_ip has been successfully brought down (35.86.13.200) stderr: + 08:14:43: ha_log:244: '[' -n /dev/null ']' stderr: + 08:14:43: __ha_log:246: : appending to /dev/null stderr: + 08:14:43: ha_log:247: '[' x '!=' /dev/nullx ']' stderr: ++ 08:14:43: ha_log:248: hadate stderr: ++ 08:14:43: hadate:173: date '+%b %d %T ' stderr: + 08:14:43: ha_log:248: echo 'awseip(aws_eip)[29027]: Jul' 26 08:14:43 'INFO: elastic_ip has been successfully brought down (35.86.13.200)' stderr: + 08:14:43: awseip_stop:206: return 0 stderr: + 08:14:43: 284: rc=0 stderr: + 08:14:43: 285: ocf_log debug 'aws_eip stop : 0' stderr: + 08:14:43: ocf_log:323: '[' 2 -lt 2 ']' stderr: + 08:14:43: ocf_log:327: OCF_PRIO=debug stderr: + 08:14:43: ocf_log:328: shift stderr: + 08:14:43: ocf_log:329: OCF_MSG='aws_eip stop : 0' stderr: + 08:14:43: ocf_log:331: case "${OCF_PRIO}" in stderr: + 08:14:43: ocf_log:336: OCF_PRIO=DEBUG stderr: + 08:14:43: ocf_log:340: '[' DEBUG = DEBUG ']' stderr: + 08:14:43: ocf_log:341: ha_debug 'DEBUG: aws_eip stop : 0' stderr: + 08:14:43: ha_debug:260: '[' x1 = x0 ']' stderr: + 08:14:43: ha_debug:260: '[' -z 1 ']' stderr: + 08:14:43: ha_debug:263: tty stderr: + 08:14:43: ha_debug:272: set_logtag stderr: + 08:14:43: set_logtag:177: '[' -z 'awseip(aws_eip)[29027]' ']' stderr: + 08:14:43: ha_debug:274: '[' x = xyes ']' stderr: + 08:14:43: ha_debug:281: '[' none = '' ']' stderr: + 08:14:43: ha_debug:284: '[' -n '' ']' stderr: + 08:14:43: ha_debug:290: '[' -n /dev/null ']' stderr: + 08:14:43: ha_debug:292: : appending to /dev/null stderr: ++ 08:14:43: ha_debug:293: hadate stderr: ++ 08:14:43: hadate:173: date '+%b %d %T ' stderr: + 08:14:43: ha_debug:293: echo 'awseip(aws_eip)[29027]: Jul' 26 08:14:43 'DEBUG: aws_eip stop : 0' stderr: + 08:14:43: ha_debug:296: '[' -z '' -a -z /dev/null ']' stderr: + 08:14:43: 286: exit 0
infa-rammekal
commented
2 years ago $ sudo pcs resource debug-start --full aws_eip warning: unpack_config: Blind faith: not fencing unseen nodes warning: unpack_rsc_op_failure: Processing failed stop of aws_eip on db25.or2.omniture.com: unknown error | rc=1 warning: unpack_rsc_op_failure: Processing failed stop of aws_eip on db25.or2.omniture.com: unknown error | rc=1 warning: unpack_rsc_op_failure: Processing failed stop of aws_eip on db26.or2.omniture.com: unknown error | rc=1 warning: unpack_rsc_op_failure: Processing failed stop of aws_eip on db26.or2.omniture.com: unknown error | rc=1 warning: action_synced_wait: aws_eip_start_0:29973 - timed out after 30000ms Operation start for aws_eip (ocf:heartbeat:awseip) failed: 'Timed Out' (2)
stderr: +++ 08:16:09: ocf_start_trace:999: echo stderr: +++ 08:16:09: ocf_start_trace:999: printenv stderr: +++ 08:16:09: ocf_start_trace:999: sort stderr: ++ 08:16:09: ocf_start_trace:999: env=' stderr: HA_debug=1 stderr: HA_logfacility=none stderr: HOME=/root stderr: HOSTNAME=db25.or2.omniture.com stderr: LC_ALL=C stderr: LOGNAME=root stderr: LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:.tar=38;5;9:.tgz=38;5;9:.arc=38;5;9:.arj=38;5;9:.taz=38;5;9:.lha=38;5;9:.lz4=38;5;9:.lzh=38;5;9:.lzma=38;5;9:.tlz=38;5;9:.txz=38;5;9:.tzo=38;5;9:.t7z=38;5;9:.zip=38;5;9:.z=38;5;9:.Z=38;5;9:.dz=38;5;9:.gz=38;5;9:.lrz=38;5;9:.lz=38;5;9:.lzo=38;5;9:.xz=38;5;9:.bz2=38;5;9:.bz=38;5;9:.tbz=38;5;9:.tbz2=38;5;9:.tz=38;5;9:.deb=38;5;9:.rpm=38;5;9:.jar=38;5;9:.war=38;5;9:.ear=38;5;9:.sar=38;5;9:.rar=38;5;9:.alz=38;5;9:.ace=38;5;9:.zoo=38;5;9:.cpio=38;5;9:.7z=38;5;9:.rz=38;5;9:.cab=38;5;9:.jpg=38;5;13:.jpeg=38;5;13:.gif=38;5;13:.bmp=38;5;13:.pbm=38;5;13:.pgm=38;5;13:.ppm=38;5;13:.tga=38;5;13:.xbm=38;5;13:.xpm=38;5;13:.tif=38;5;13:.tiff=38;5;13:.png=38;5;13:.svg=38;5;13:.svgz=38;5;13:.mng=38;5;13:.pcx=38;5;13:.mov=38;5;13:.mpg=38;5;13:.mpeg=38;5;13:.m2v=38;5;13:.mkv=38;5;13:.webm=38;5;13:.ogm=38;5;13:.mp4=38;5;13:.m4v=38;5;13:.mp4v=38;5;13:.vob=38;5;13:.qt=38;5;13:.nuv=38;5;13:.wmv=38;5;13:.asf=38;5;13:.rm=38;5;13:.rmvb=38;5;13:.flc=38;5;13:.avi=38;5;13:.fli=38;5;13:.flv=38;5;13:.gl=38;5;13:.dl=38;5;13:.xcf=38;5;13:.xwd=38;5;13:.yuv=38;5;13:.cgm=38;5;13:.emf=38;5;13:.axv=38;5;13:.anx=38;5;13:.ogv=38;5;13:.ogx=38;5;13:.aac=38;5;45:.au=38;5;45:.flac=38;5;45:.mid=38;5;45:.midi=38;5;45:.mka=38;5;45:.mp3=38;5;45:.mpc=38;5;45:.ogg=38;5;45:.ra=38;5;45:.wav=38;5;45:.axa=38;5;45:.oga=38;5;45:.spx=38;5;45:.xspf=38;5;45: stderr: MAIL=/var/mail/root stderr: OCF_EXIT_REASON_PREFIX=ocf-exit-reason: stderr: OCF_RA_VERSION_MAJOR=1 stderr: OCF_RA_VERSION_MINOR=0 stderr: OCF_RESKEY_CRM_meta_class=ocf stderr: OCF_RESKEY_CRM_meta_id=aws_eip stderr: OCF_RESKEY_CRM_meta_provider=heartbeat stderr: OCF_RESKEY_CRM_meta_resource_stickiness=100 stderr: OCF_RESKEY_CRM_meta_timeout=30000 stderr: OCF_RESKEY_CRM_meta_type=awseip stderr: OCF_RESKEY_allocation_id=eipalloc-052c7f5348fc5d26c stderr: OCF_RESKEY_crm_feature_set=3.0.14 stderr: OCF_RESKEY_elastic_ip=35.86.13.200 stderr: OCF_RESKEY_trace_file=/tmp/trace stderr: OCF_RESKEY_trace_ra=1 stderr: OCF_RESOURCE_INSTANCE=aws_eip stderr: OCF_RESOURCE_PROVIDER=heartbeat stderr: OCF_RESOURCE_TYPE=awseip stderr: OCF_ROOT=/usr/lib/ocf stderr: OCF_TRACE_FILE=/dev/stderr stderr: OCF_TRACE_RA=1 stderr: PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/rmekala/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/ucb stderr: PCMK_logfacility=none stderr: PCMK_service=crm_resource stderr: PWD=/usr/local/ccc stderr: SHELL=/bin/bash stderr: SHLVL=1 stderr: SUDO_COMMAND=/usr/sbin/pcs resource debug-start --full aws_eip stderr: SUDO_GID=100 stderr: SUDO_UID=122820 stderr: SUDO_USER=rmekala stderr: TERM=xterm-256color stderr: USER=root stderr: USERNAME=root stderr: XDG_SESSIONID=24318 stderr: =/usr/bin/printenv stderr: OCF_TRC_DEST=/dev/stderr stderr: OCF_TRC_MANAGE=' stderr: ++ 08:16:09: 1053: ocf_is_true '' stderr: ++ 08:16:09: ocf_is_true:103: case "$1" in stderr: ++ 08:16:09: ocf_is_true:103: case "$1" in stderr: ++ 08:16:09: ocf_is_true:105: false stderr: + 08:16:09: 45: OCF_RESKEY_awscli_default=/usr/bin/aws stderr: + 08:16:09: 46: OCF_RESKEY_profile_default=default stderr: + 08:16:09: 47: OCF_RESKEY_api_delay_default=3 stderr: + 08:16:09: 49: : /usr/bin/aws stderr: + 08:16:09: 50: : default stderr: + 08:16:09: 51: : 3 stderr: + 08:16:09: 235: case $OCF_ACTION in stderr: + 08:16:09: 242: AWSCLI=/usr/bin/aws stderr: + 08:16:09: 243: ELASTIC_IP=35.86.13.200 stderr: + 08:16:09: 244: ALLOCATION_ID=eipalloc-052c7f5348fc5d26c stderr: + 08:16:09: 245: PRIVATE_IP_ADDRESS= stderr: ++ 08:16:09: 246: curl -X PUT http://169.254.169.254/latest/api/token -H 'X-aws-ec2-metadata-token-ttl-seconds: 21600' stderr: % Total % Received % Xferd Average Speed Time Time Time Current stderr: Dload Upload Total Spent Left Speed 100 56 100 56 0 0 26192 0 --:--:-- --:--:-- --:--:-- 56000 stderr: + 08:16:09: 246: TOKEN=AQAEAF3fVAr6CK1OQ-2BfRqsvMrVCVLsmiZTqnu0qR3fPforK3ur0w== stderr: ++ 08:16:09: 247: curl -s http://169.254.169.254/latest/meta-data/instance-id -H 'X-aws-ec2-metadata-token: AQAEAF3fVAr6CK1OQ-2BfRqsvMrVCVLsmiZTqnu0qR3fPforK3ur0w==' stderr: + 08:16:09: 247: INSTANCE_ID=i-07b8432a9dc874ef4 stderr: + 08:16:09: 249: case $OCF_ACTION in stderr: + 08:16:09: 251: awseip_validate stderr: + 08:16:09: awseip_validate:220: check_binary /usr/bin/aws stderr: + 08:16:09: check_binary:55: have_binary /usr/bin/aws stderr: + 08:16:09: have_binary:67: '[' '' = 1 ']' stderr: ++ 08:16:09: have_binary:70: sed -e 's/ -.//' stderr: ++ 08:16:09: have_binary:70: echo /usr/bin/aws stderr: + 08:16:09: have_binary:70: local bin=/usr/bin/aws stderr: ++ 08:16:09: have_binary:71: which /usr/bin/aws stderr: + 08:16:09: have_binary:71: test -x /usr/bin/aws stderr: + 08:16:09: awseip_validate:222: '[' -z default ']' stderr: + 08:16:09: awseip_validate:227: '[' -z i-07b8432a9dc874ef4 ']' stderr: + 08:16:09: awseip_validate:232: return 0 stderr: + 08:16:09: 252: awseip_start stderr: + 08:16:09: awseip_start:148: awseip_monitor stderr: + 08:16:09: awseip_monitor:210: grep -q 35.86.13.200 stderr: + 08:16:09: awseip_monitor:210: /usr/bin/aws --profile default ec2 describe-instances --instance-id i-07b8432a9dc874ef4
infa-rammekal
commented
2 years ago Stack: corosync Current DC: db26.or2.omniture.com (version 1.1.23-1.el7_9.1-9acf116022) - partition with quorum Last updated: Tue Jul 26 08:16:45 2022 Last change: Tue Jul 26 08:14:14 2022 by root via cibadmin on db25.or2.omniture.com
2 nodes configured 4 resource instances configured (1 BLOCKED from further action due to failure)
Online: [ db25.or2.omniture.com db26.or2.omniture.com ]
Active resources:
Master/Slave Set: ms_mysql [p_mysql] Masters: [ db25.or2.omniture.com ] Slaves: [ db26.or2.omniture.com ] Resource Group: aws_group aws_eip (ocf::heartbeat:awseip): FAILED (blocked)[ db25.or2.omniture.com db26.or2.omniture.com ]
Failed Resource Actions:
infa-rammekal
commented
2 years ago /var/lib/heartbeat/trace_ra/awseip # ls -ltr total 8 -rw-r-----. 1 root root 3493 Jul 26 08:14 aws_eip.monitor.2022-07-26.08:14:14 -rw-r-----. 1 root root 2886 Jul 26 08:14 aws_eip.stop.2022-07-26.08:14:44 [root@db25.or2|cluster1] P,S: db25,db26, W,R: , /var/lib/heartbeat/trace_ra/awseip # cat aws_eip.stop.2022-07-26.08:14:44 +++ 08:14:44: ocf_start_trace:999: echo +++ 08:14:44: ocf_start_trace:999: printenv +++ 08:14:44: ocf_start_trace:999: sort ++ 08:14:44: ocf_start_trace:999: env=' HA_LOGD=no HA_LOGFACILITY=daemon HA_LOGFILE=/var/log/cluster/corosync.log HA_cluster_type=corosync HA_debug=0 HA_logfacility=daemon HA_logfile=/var/log/cluster/corosync.log HA_mcp=true HA_quorum_type=corosync HA_use_logd=off LC_ALL=C OCF_EXIT_REASON_PREFIX=ocf-exit-reason: OCF_RA_VERSION_MAJOR=1 OCF_RA_VERSION_MINOR=0 OCF_RESKEY_CRM_meta_name=stop OCF_RESKEY_CRM_meta_on_node=db25.or2.omniture.com OCF_RESKEY_CRM_meta_on_node_uuid=1 OCF_RESKEY_CRM_meta_timeout=30000 OCF_RESKEY_allocation_id=eipalloc-052c7f5348fc5d26c OCF_RESKEY_crm_feature_set=3.0.14 OCF_RESKEY_elastic_ip=35.86.13.200 OCF_RESKEY_trace_file=/tmp/trace OCF_RESKEY_trace_ra=1 OCF_RESOURCE_INSTANCE=aws_eip OCF_RESOURCE_PROVIDER=heartbeat OCF_RESOURCE_TYPE=awseip OCF_ROOT=/usr/lib/ocf PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/ucb PCMK_cluster_type=corosync PCMK_debug=0 PCMK_logfacility=daemon PCMK_logfile=/var/log/cluster/corosync.log PCMK_mcp=true PCMK_quorum_type=corosync PCMK_service=lrmd PCMK_use_logd=off PCMK_watchdog=false PWD=/var/lib/pacemaker/cores SHLVL=1 VALGRINDOPTS=--leak-check=full --trace-children=no --vgdb=no --num-callers=25 --log-file=/var/lib/pacemaker/valgrind-%p --suppressions=/usr/share/pacemaker/tests/valgrind-pcmk.suppressions --gen-suppressions=all =/usr/bin/printenv OCF_TRC_DEST=/var/lib/heartbeat/trace_ra/awseip/aws_eip.stop.2022-07-26.08:14:44 OCF_TRC_MANAGE=1' ++ 08:14:44: 1053: ocf_is_true off ++ 08:14:44: ocf_is_true:103: case "$1" in ++ 08:14:44: ocf_is_true:103: case "$1" in ++ 08:14:44: ocf_is_true:105: false
infa-rammekal
commented
2 years ago when I run debug on other node I got some errors initially.. rest is same as above.
db26.or2 ~]$ sudo pcs resource debug-stop --full aws_eip warning: unpack_config: Blind faith: not fencing unseen nodes warning: unpack_rsc_op_failure: Processing failed stop of aws_eip on db25.or2.omniture.com: unknown error | rc=1 warning: unpack_rsc_op_failure: Processing failed stop of aws_eip on db25.or2.omniture.com: unknown error | rc=1 warning: unpack_rsc_op_failure: Processing failed stop of aws_eip on db26.or2.omniture.com: unknown error | rc=1 warning: unpack_rsc_op_failure: Processing failed stop of aws_eip on db26.or2.omniture.com: unknown error | rc=1 warning: action_synced_wait: aws_eip_stop_0:26758 - timed out after 30000ms Operation stop for aws_eip (ocf:heartbeat:awseip) failed: 'Timed Out' (2)
db26.or2 ~]$ sudo pcs resource debug-start --full aws_eip warning: unpack_config: Blind faith: not fencing unseen nodes warning: unpack_rsc_op_failure: Processing failed probe of aws_eip on db25.or2.omniture.com: unknown error | rc=1 warning: unpack_rsc_op_failure: Processing failed probe of aws_eip on db26.or2.omniture.com: unknown error | rc=1 warning: action_synced_wait: aws_eip_start_0:27422 - timed out after 30000ms Operation start for aws_eip (ocf:heartbeat:awseip) failed: 'Timed Out' (2)
stderr: +++ 08:23:16: ocf_start_trace:999: echo stderr: +++ 08:23:16: ocf_start_trace:999: printenv stderr: +++ 08:23:16: ocf_start_trace:999: sort
infa-rammekal
commented
2 years ago I found below error in /var/log/message on second node.
Jul 26 08:55:22 db26.or2.omniture.com crmd[20837]: notice: State transition S_IDLE -> S_POLICY_ENGINE Jul 26 08:55:22 db26.or2.omniture.com pengine[20836]: notice: On loss of CCM Quorum: Ignore Jul 26 08:55:22 db26.or2.omniture.com pengine[20836]: warning: Processing failed stop of aws_eip on db26.or2.omniture.com: unknown error Jul 26 08:55:22 db26.or2.omniture.com pengine[20836]: warning: Processing failed stop of aws_eip on db26.or2.omniture.com: unknown error Jul 26 08:55:22 db26.or2.omniture.com pengine[20836]: warning: Forcing aws_eip away from db26.or2.omniture.com after 1000000 failures (max=1000000) Jul 26 08:55:22 db26.or2.omniture.com pengine[20836]: notice: Calculated transition 692, saving inputs in /var/lib/pacemaker/pengine/pe-input-207.bz2 Jul 26 08:55:22 db26.or2.omniture.com crmd[20837]: notice: Transition 692 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-207.bz2): Complete Jul 26 08:55:22 db26.or2.omniture.com crmd[20837]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE
infa-rammekal
commented
2 years ago Jul 26 08:39:21 db26.or2.omniture.com crmd[20837]: notice: State transition S_IDLE -> S_POLICY_ENGINE Jul 26 08:39:22 db26.or2.omniture.com pengine[20836]: notice: On loss of CCM Quorum: Ignore Jul 26 08:39:22 db26.or2.omniture.com pengine[20836]: notice: Start aws_eip ( db25.or2.omniture.com ) Jul 26 08:39:22 db26.or2.omniture.com pengine[20836]: notice: Calculated transition 688, saving inputs in /var/lib/pacemaker/pengine/pe-input-204.bz2 Jul 26 08:39:22 db26.or2.omniture.com crmd[20837]: notice: Initiating monitor operation aws_eip_monitor_0 locally on db26.or2.omniture.com Jul 26 08:39:32 db26.or2.omniture.com systemd: Started Verify Quick. Jul 26 08:39:52 db26.or2.omniture.com lrmd[20834]: warning: aws_eip_monitor_0 process (PID 1880) timed out Jul 26 08:39:52 db26.or2.omniture.com lrmd[20834]: warning: aws_eip_monitor_0:1880 - timed out after 30000ms Jul 26 08:39:52 db26.or2.omniture.com lrmd[20834]: notice: aws_eip_monitor_0:1880:stderr [ % Total % Received % Xferd Average Speed Time Time Time Current ] Jul 26 08:39:52 db26.or2.omniture.com lrmd[20834]: notice: aws_eip_monitor_0:1880:stderr [ Dload Upload Total Spent Left Speed ] Jul 26 08:39:52 db26.or2.omniture.com lrmd[20834]: notice: aws_eip_monitor_0:1880:stderr [ #015 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0#015100 56 100 56 0 0 21952 0 --:--:-- --:--:-- --:--:-- 56000 ] Jul 26 08:39:52 db26.or2.omniture.com crmd[20837]: error: Result of probe operation for aws_eip on db26.or2.omniture.com: Timed Out Jul 26 08:39:52 db26.or2.omniture.com crmd[20837]: notice: db26.or2.omniture.com-aws_eip_monitor_0:640 [ % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 56 100 56 0 0 21952 0 --:--:-- --:--:-- --:--:-- 56000\n ] Jul 26 08:39:52 db26.or2.omniture.com crmd[20837]: warning: Action 3 (aws_eip_monitor_0) on db26.or2.omniture.com failed (target: 7 vs. rc: 1): Error Jul 26 08:39:52 db26.or2.omniture.com crmd[20837]: notice: Transition aborted by operation aws_eip_monitor_0 'modify' on db26.or2.omniture.com: Event failed Jul 26 08:39:52 db26.or2.omniture.com crmd[20837]: notice: Transition 688 (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=3, Source=/var/lib/pacemaker/pengine/pe-input-204.bz2): Complete Jul 26 08:39:52 db26.or2.omniture.com pengine[20836]: notice: On loss of CCM Quorum: Ignore Jul 26 08:39:52 db26.or2.omniture.com pengine[20836]: warning: Processing failed probe of aws_eip on db26.or2.omniture.com: unknown error Jul 26 08:39:52 db26.or2.omniture.com pengine[20836]: notice: If it is not possible for aws_eip to run on db26.or2.omniture.com, see the resource-discovery option for location constraints Jul 26 08:39:52 db26.or2.omniture.com pengine[20836]: warning: Processing failed probe of aws_eip on db26.or2.omniture.com: unknown error Jul 26 08:39:52 db26.or2.omniture.com pengine[20836]: notice: If it is not possible for aws_eip to run on db26.or2.omniture.com, see the resource-discovery option for location constraints Jul 26 08:39:52 db26.or2.omniture.com pengine[20836]: notice: Recover aws_eip ( db26.or2.omniture.com -> db25.or2.omniture.com ) Jul 26 08:39:52 db26.or2.omniture.com pengine[20836]: notice: Calculated transition 689, saving inputs in /var/lib/pacemaker/pengine/pe-input-205.bz2 Jul 26 08:39:52 db26.or2.omniture.com crmd[20837]: notice: Initiating stop operation aws_eip_stop_0 locally on db26.or2.omniture.com
oalbrigt
commented
2 years ago Check what the monitor-action at the time of the failed "probe" says in the trace-directory.
infa-rammekal
commented
2 years ago I started againg this morning with trace on. here is /var/log/messages & trace
Jul 27 06:29:01 db26.or2.omniture.com lrmd[20834]: warning: aws_eip_monitor_0 process (PID 6133) timed out
Jul 27 06:29:01 db26.or2.omniture.com lrmd[20834]: warning: aws_eip_monitor_0:6133 - timed out after 60000ms
Jul 27 06:29:01 db26.or2.omniture.com lrmd[20834]: notice: aws_eip_monitor_0:6133:stderr [ % Total % Received % Xferd Average Speed Time Time Time Current ]
Jul 27 06:29:01 db26.or2.omniture.com lrmd[20834]: notice: aws_eip_monitor_0:6133:stderr [ Dload Upload Total Spent Left Speed ]
Jul 27 06:29:01 db26.or2.omniture.com lrmd[20834]: notice: aws_eip_monitor_0:6133:stderr [ #015 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0#015100 56 100 56 0 0 65959 0 --:--:-- --:--:-- --:--:-- 56000 ]
Jul 27 06:29:01 db26.or2.omniture.com crmd[20837]: error: Result of probe operation for aws_eip on db26.or2.omniture.com: Timed Out
Jul 27 06:29:01 db26.or2.omniture.com crmd[20837]: notice: db26.or2.omniture.com-aws_eip_monitor_0:680 [ % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 56 100 56 0 0 65959 0 --:--:-- --:--:-- --:--:-- 56000\n ]
Jul 27 06:29:01 db26.or2.omniture.com crmd[20837]: warning: Action 4 (aws_eip_monitor_0) on db26.or2.omniture.com failed (target: 7 vs. rc: 1): Error
Jul 27 06:29:01 db26.or2.omniture.com crmd[20837]: notice: Transition 802 (Complete=3, Pending=0, Fired=0, Skipped=0, Incomplete=3, Source=/var/lib/pacemaker/pengine/pe-input-218.bz2): Complete
Jul 27 06:29:01 db26.or2.omniture.com pengine[20836]: notice: On loss of CCM Quorum: Ignore
Jul 27 06:29:01 db26.or2.omniture.com pengine[20836]: warning: Processing failed probe of aws_eip on db26.or2.omniture.com: unknown error
Jul 27 06:29:01 db26.or2.omniture.com pengine[20836]: notice: If it is not possible for aws_eip to run on db26.or2.omniture.com, see the resource-discovery option for location constraints
Jul 27 06:29:01 db26.or2.omniture.com pengine[20836]: warning: Processing failed probe of aws_eip on db26.or2.omniture.com: unknown error
Jul 27 06:29:01 db26.or2.omniture.com pengine[20836]: notice: If it is not possible for aws_eip to run on db26.or2.omniture.com, see the resource-discovery option for location constraints
Jul 27 06:29:01 db26.or2.omniture.com pengine[20836]: error: Resource aws_eip is active on 2 nodes (attempting recovery)
Jul 27 06:29:01 db26.or2.omniture.com pengine[20836]: notice: See https://wiki.clusterlabs.org/wiki/FAQ#Resource_is_Too_Active for more information
Jul 27 06:29:01 db26.or2.omniture.com pengine[20836]: notice: * Recover aws_eip ( db25.or2.omniture.com )
Jul 27 06:29:01 db26.or2.omniture.com pengine[20836]: error: Calculated transition 803 (with errors), saving inputs in /var/lib/pacemaker/pengine/pe-error-124.bz2
Jul 27 06:29:01 db26.or2.omniture.com crmd[20837]: notice: Initiating stop operation aws_eip_stop_0 on db25.or2.omniture.com
Jul 27 06:29:01 db26.or2.omniture.com crmd[20837]: notice: Initiating stop operation aws_eip_stop_0 locally on db26.or2.omniture.com
Jul 27 06:29:01 db26.or2.omniture.com systemd: Created slice User Slice of root.
Jul 27 06:29:01 db26.or2.omniture.com systemd: Started Session 24064 of user root.
Jul 27 06:29:02 db26.or2.omniture.com systemd: Removed slice User Slice of root.
Jul 27 06:29:31 db26.or2.omniture.com lrmd[20834]: warning: aws_eip_stop_0 process (PID 7056) timed out
Jul 27 06:29:31 db26.or2.omniture.com lrmd[20834]: warning: aws_eip_stop_0:7056 - timed out after 30000ms
Jul 27 06:29:31 db26.or2.omniture.com lrmd[20834]: notice: aws_eip_stop_0:7056:stderr [ % Total % Received % Xferd Average Speed Time Time Time Current ]
Jul 27 06:29:31 db26.or2.omniture.com lrmd[20834]: notice: aws_eip_stop_0:7056:stderr [ Dload Upload Total Spent Left Speed ]
Jul 27 06:29:31 db26.or2.omniture.com lrmd[20834]: notice: aws_eip_stop_0:7056:stderr [ #015 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0#015100 56 100 56 0 0 32054 0 --:--:-- --:--:-- --:--:-- 56000 ]
Jul 27 06:29:31 db26.or2.omniture.com crmd[20837]: error: Result of stop operation for aws_eip on db26.or2.omniture.com: Timed Out
Jul 27 06:29:31 db26.or2.omniture.com crmd[20837]: notice: db26.or2.omniture.com-aws_eip_stop_0:682 [ % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 56 100 56 0 0 32054 0 --:--:-- --:--:-- --:--:-- 56000\n ]
Jul 27 06:29:31 db26.or2.omniture.com crmd[20837]: warning: Action 3 (aws_eip_stop_0) on db26.or2.omniture.com failed (target: 0 vs. rc: 1): Error
Jul 27 06:29:31 db26.or2.omniture.com crmd[20837]: notice: Transition aborted by operation aws_eip_stop_0 'modify' on db26.or2.omniture.com: Event failed
Jul 27 06:29:31 db26.or2.omniture.com crmd[20837]: notice: Transition 803 (Complete=3, Pending=0, Fired=0, Skipped=0, Incomplete=5, Source=/var/lib/pacemaker/pengine/pe-error-124.bz2): Complete
Jul 27 06:29:31 db26.or2.omniture.com pengine[20836]: notice: On loss of CCM Quorum: Ignore
Jul 27 06:29:31 db26.or2.omniture.com pengine[20836]: warning: Processing failed stop of aws_eip on db26.or2.omniture.com: unknown error
Jul 27 06:29:31 db26.or2.omniture.com pengine[20836]: warning: Processing failed stop of aws_eip on db26.or2.omniture.com: unknown error
Jul 27 06:29:31 db26.or2.omniture.com pengine[20836]: notice: Calculated transition 804, saving inputs in /var/lib/pacemaker/pengine/pe-input-219.bz2
Jul 27 06:29:31 db26.or2.omniture.com pengine[20836]: notice: On loss of CCM Quorum: Ignore
Jul 27 06:29:31 db26.or2.omniture.com pengine[20836]: warning: Processing failed stop of aws_eip on db26.or2.omniture.com: unknown error
Jul 27 06:29:31 db26.or2.omniture.com pengine[20836]: warning: Processing failed stop of aws_eip on db26.or2.omniture.com: unknown error
Jul 27 06:29:31 db26.or2.omniture.com pengine[20836]: warning: Forcing aws_eip away from db26.or2.omniture.com after 1000000 failures (max=1000000)
Jul 27 06:29:31 db26.or2.omniture.com pengine[20836]: notice: Calculated transition 805, saving inputs in /var/lib/pacemaker/pengine/pe-input-220.bz2
Jul 27 06:29:31 db26.or2.omniture.com crmd[20837]: notice: Transition 805 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-220.bz2): Complete
Jul 27 06:29:31 db26.or2.omniture.com crmd[20837]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE
TRACE:
/var/lib/heartbeat/trace_ra/awseip # ls -ltr total 16 -rw-r-----. 1 root root 2785 Jul 27 06:25 aws_eip.monitor.2022-07-27.06:25:56 -rw-r-----. 1 root root 2853 Jul 27 06:26 aws_eip.stop.2022-07-27.06:26:56 -rw-r-----. 1 root root 2785 Jul 27 06:28 aws_eip.monitor.2022-07-27.06:28:01 -rw-r-----. 1 root root 2853 Jul 27 06:29 aws_eip.stop.2022-07-27.06:29:01 [root@db26.or2|cluster1] P,S: db25,db26, W,R: , /var/lib/heartbeat/trace_ra/awseip # cat aws_eip.monitor.2022-07-27.06:28:01 +++ 06:28:01: ocf_start_trace:999: echo +++ 06:28:01: ocf_start_trace:999: sort +++ 06:28:01: ocf_start_trace:999: printenv ++ 06:28:01: ocf_start_trace:999: env=' HA_LOGD=no HA_LOGFACILITY=daemon HA_LOGFILE=/var/log/cluster/corosync.log HA_cluster_type=corosync HA_debug=0 HA_logfacility=daemon HA_logfile=/var/log/cluster/corosync.log HA_mcp=true HA_quorum_type=corosync HA_use_logd=off LC_ALL=C OCF_EXIT_REASON_PREFIX=ocf-exit-reason: OCF_RA_VERSION_MAJOR=1 OCF_RA_VERSION_MINOR=0 OCF_RESKEY_CRM_meta_on_node=db26.or2.omniture.com OCF_RESKEY_CRM_meta_on_node_uuid=2 OCF_RESKEY_CRM_meta_timeout=60000 OCF_RESKEY_allocation_id=eipalloc-052c7f5348fc5d26c OCF_RESKEY_crm_feature_set=3.0.14 OCF_RESKEY_elastic_ip=35.86.13.200 OCF_RESKEY_trace_ra=1 OCF_RESOURCE_INSTANCE=aws_eip OCF_RESOURCE_PROVIDER=heartbeat OCF_RESOURCE_TYPE=awseip OCF_ROOT=/usr/lib/ocf PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/ucb PCMK_cluster_type=corosync PCMK_debug=0 PCMK_logfacility=daemon PCMK_logfile=/var/log/cluster/corosync.log PCMK_mcp=true PCMK_quorum_type=corosync PCMK_service=lrmd PCMK_use_logd=off PCMK_watchdog=false PWD=/var/lib/pacemaker/cores SHLVL=1 VALGRINDOPTS=--leak-check=full --trace-children=no --vgdb=no --num-callers=25 --log-file=/var/lib/pacemaker/valgrind-%p --suppressions=/usr/share/pacemaker/tests/valgrind-pcmk.suppressions --gen-suppressions=all =/usr/bin/printenv OCF_TRC_DEST=/var/lib/heartbeat/trace_ra/awseip/aws_eip.monitor.2022-07-27.06:28:01 OCF_TRC_MANAGE=1' ++ 06:28:01: 1053: ocf_is_true off ++ 06:28:01: ocf_is_true:103: case "$1" in ++ 06:28:01: ocf_is_true:103: case "$1" in ++ 06:28:01: ocf_is_true:105: false
/var/lib/heartbeat/trace_ra/awseip # cat aws_eip.stop.2022-07-27.06:29:01 +++ 06:29:01: ocf_start_trace:999: echo +++ 06:29:01: ocf_start_trace:999: sort +++ 06:29:01: ocf_start_trace:999: printenv ++ 06:29:01: ocf_start_trace:999: env=' HA_LOGD=no HA_LOGFACILITY=daemon HA_LOGFILE=/var/log/cluster/corosync.log HA_cluster_type=corosync HA_debug=0 HA_logfacility=daemon HA_logfile=/var/log/cluster/corosync.log HA_mcp=true HA_quorum_type=corosync HA_use_logd=off LC_ALL=C OCF_EXIT_REASON_PREFIX=ocf-exit-reason: OCF_RA_VERSION_MAJOR=1 OCF_RA_VERSION_MINOR=0 OCF_RESKEY_CRM_meta_name=stop OCF_RESKEY_CRM_meta_on_node=db26.or2.omniture.com OCF_RESKEY_CRM_meta_on_node_uuid=2 OCF_RESKEY_CRM_meta_timeout=30000 OCF_RESKEY_allocation_id=eipalloc-052c7f5348fc5d26c OCF_RESKEY_crm_feature_set=3.0.14 OCF_RESKEY_elastic_ip=35.86.13.200 OCF_RESKEY_trace_ra=1 OCF_RESOURCE_INSTANCE=aws_eip OCF_RESOURCE_PROVIDER=heartbeat OCF_RESOURCE_TYPE=awseip OCF_ROOT=/usr/lib/ocf PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/ucb PCMK_cluster_type=corosync PCMK_debug=0 PCMK_logfacility=daemon PCMK_logfile=/var/log/cluster/corosync.log PCMK_mcp=true PCMK_quorum_type=corosync PCMK_service=lrmd PCMK_use_logd=off PCMK_watchdog=false PWD=/var/lib/pacemaker/cores SHLVL=1 VALGRINDOPTS=--leak-check=full --trace-children=no --vgdb=no --num-callers=25 --log-file=/var/lib/pacemaker/valgrind-%p --suppressions=/usr/share/pacemaker/tests/valgrind-pcmk.suppressions --gen-suppressions=all =/usr/bin/printenv OCF_TRC_DEST=/var/lib/heartbeat/trace_ra/awseip/aws_eip.stop.2022-07-27.06:29:01 OCF_TRC_MANAGE=1' ++ 06:29:01: 1053: ocf_is_true off ++ 06:29:01: ocf_is_true:103: case "$1" in ++ 06:29:01: ocf_is_true:103: case "$1" in ++ 06:29:01: ocf_is_true:105: false
oalbrigt
commented
2 years ago It seems like it probably times out running aws --profile default ec2 describe-instances --instance-id i-08b22e8b666304490.
You can run time aws --profile default ec2 describe-instances --instance-id i-08b22e8b666304490 manually to see how long it takes.
infa-rammekal
commented
2 years ago yes. thank you. I just figured it out.. it is not working outside as well. I will work on it and do further tests.. thank you for all your support. keep you posted.
RC-Code
commented
2 years ago Hey hi,
I think you may have CC'ed the wrong person.
I'm going to delete the email on my end.
Regards Ron
On Fri., Jul. 22, 2022, 2:00 p.m. infa-rammekal, @.***> wrote:
AWSEIP is not able to disassociate and associate EIP between the nodes. And also keep erroring out.
Here is the commands I issued after setting corosync/pacemaker/mysql-M/S.
pcs resource create aws_eip ocf💓awseip elastic_ip=35.86.13.200 allocation_id=eipalloc-052c7f5348fc5d26c awscli="$(which aws)" --group aws_group op monitor timeout="40s" interval="20s" OCF_CHECK_LEVEL="10"
pcs constraint colocation add aws_eip with master ms_mysql pcs constraint order promote ms_mysql then start aws_eip
Stack: corosync Current DC: db26 (version 1.1.23-1.el7_9.1-9acf116022) - part ition with quorum Last updated: Fri Jul 22 13:49:22 2022 Last change: Fri Jul 22 13:48:09 2022 by hacluster via crmd on db26.or2.omnitu re.com
2 nodes configured 4 resource instances configured (1 BLOCKED from further action due to failure)
Online: [ db25.or2.omniture.com db26.or2.omniture.com ]
Active resources:
Master/Slave Set: ms_mysql [p_mysql] Masters: [ db25] Slaves: [ db26 ] Resource Group: aws_group aws_eip (ocf:💓awseip): FAILED (blocked)[ db25.or2.omn iture.com db26. ]
Failed Resource Actions:
- aws_eip_stop_0 on db25 'not running' (7): call=489, status= complete, exitreason='', last-rc-change='Fri Jul 22 13:48:50 2022', queued=0ms, exec=5649ms
- aws_eip_stop_0 on db26.or2.omniture.com 'unknown error' (1): call=455, statu s=Timed Out, exitreason='', last-rc-change='Fri Jul 22 13:48:50 2022', queued=0ms, exec=30003ms
here is corosync.log output..
@last-run=1658520877, @last-rc-change=1658520877, @exec-time=0 Jul 22 13:14:37 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=db25.or2.omniture.com/crmd/378, version=1.236.6) Jul 22 13:14:42 [13261] db25.or2.omniture.com cib: info: cib_process_ping: Reporting our current digest to db26.or2.omniture.com: ee253e652fc7abb3220c5d8613f25f39 for 1.236.6 (0x562ec48d2090 0) Jul 22 13:14:43 [13263] db25.or2.omniture.com lrmd: notice: operation_finished: aws_eip_stop_0:3035:stderr [ % Total % Received % Xferd Average Speed Time Time Time Current ] Jul 22 13:14:43 [13263] db25.or2.omniture.com lrmd: notice: operation_finished: aws_eip_stop_0:3035:stderr [ Dload Upload Total Spent Left Speed ] Jul 22 13:14:43 [13263] db25.or2.omniture.com lrmd: notice: operation_finished: aws_eip_stop_0:3035:s100 56 100 56 0 0 31728 0 --:--:-- --:--:-- --:--:-- 56000 ] Jul 22 13:14:43 [13263] db25.or2.omniture.com lrmd: notice: operation_finished: aws_eip_stop_0:3035:stderr [ ] Jul 22 13:14:43 [13263] db25.or2.omniture.com lrmd: notice: operation_finished: aws_eip_stop_0:3035:stderr [ An error occurred (UnauthorizedOperation) when calling the DisassociateAddress operation: You are not authorized to perform this operation. Encoded authorization failure message: fY4uLP5oYTunADmj5AoKUKnIYIt0fgaOjCVjVSfp4Q-zIObsDYLYOZZi94n_8nGVoRuBqv-zpykdtQbq6-TmwkH65_QzJVD9xgF1wFPxO-2vPTXJm0tPvqoZeM9c1og_qCX2zR0Qx8OvAoyZyF5UiDMgnAfrWMsVb7t5lP43DZcN-bFsyVDe-7GdZd0tl7nApFYZL-cZA6WGDRomQB7pOEX Jul 22 13:14:43 [13263] db25.or2.omniture.com lrmd: info: log_finished: finished - rsc:aws_eip action:stop call_id:481 pid:3035 exit-code:7 exec-time:6071ms queue-time:0ms Jul 22 13:14:43 [13266] db25.or2.omniture.com crmd: notice: process_lrm_event: Result of stop operation for aws_eip on db25.or2.omniture.com: 7 (not running) | call=481 key=aws_eip_stop_0 confirmed=true cib-update=379 Jul 22 13:14:43 [13266] db25.or2.omniture.com crmd: notice: process_lrm_event: db25.or2.omniture.com-aws_eip_stop_0:481 [ % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 56 100 56 0 0 31728 0 --:--:-- --:--:-- --:--:-- 56000\n\nAn error occurred (UnauthorizedOperation) when calling th Jul 22 13:14:43 [13266] db25.or2.omniture.com crmd: info: exec_alert_list: Sending resource alert via alert_file to /var/log/cluster/pcmk_alert_file.log Jul 22 13:14:43 [13263] db25.or2.omniture.com lrmd: info: process_lrmd_alert_exec: Executing alert alert_file for 18d3dbc1-e6ef-4e45-b801-ae2dd6c9273c Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/379) Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.236.6 2 Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.236.7 (null) Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- /cib: @num_updates=7 Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- @. @. @. @. https://github.com/id='aws_eip_last_0']: @transition-magic=0:7;34:215:0:600465a1-1af3-4f13-b91b-ceab68ecdc8f, @call-id=481, @RC-Code https://github.com/RC-Code=7, @op-status=0, @exec-time=6071 Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- @. @. @. @. https://github.com/id='aws_eip_last_failure_0']: @operation_key=aws_eip_stop_0, @operation https://github.com/operation=stop, @transition-key=34:215:0:600465a1-1af3-4f13-b91b-ceab68ecdc8f, @transition-magic=0:7;34:215:0:600465a1-1af3-4f13-b91b-ceab68ecdc8f, @call-id=481, @RC-Code https://github.com/RC-Code=7, @last-run=1658520877, @last-rc-change=1658520877, @exec-time=6071 Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=db25.or2.omniture.com/crmd/379, version=1.236.7) Jul 22 13:14:43 [13264] db25.or2.omniture.com attrd: info: attrd_peer_update: Setting fail-count-aws_eip#stop_0[db25.or2.omniture.com]: (null) -> INFINITY from db26.or2.omniture.com Jul 22 13:14:43 [13264] db25.or2.omniture.com attrd: info: write_attribute: Sent CIB request 298 with 2 changes for fail-count-aws_eip#stop_0 (id n/a, set n/a) Jul 22 13:14:43 [13264] db25.or2.omniture.com attrd: info: attrd_peer_update: Setting last-failure-aws_eip#stop_0[ db25.or2.omniture.com]: (null) -> 1658520883 from db26.or2.omniture.com Jul 22 13:14:43 [13264] db25.or2.omniture.com attrd: info: write_attribute: Sent CIB request 299 with 2 changes for last-failure-aws_eip#stop_0 (id n/a, set n/a) Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/attrd/298) Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/attrd/299) Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.236.7 2 Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.236.8 (null) Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- /cib: @num_updates=8 Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_perform_op: ++ @. https://github.com/id @. https://github.com/id @.** https://github.com/id='status-1']: Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=db25.or2.omniture.com/attrd/298, version=1.236.8) Jul 22 13:14:43 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: CIB update 298 result for fail-count-aws_eip#stop_0: OK | rc=0 Jul 22 13:14:43 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: fail-count-aws_eip#stop_0[db25.or2.omniture.com ]=INFINITY Jul 22 13:14:43 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: * fail-count-aws_eip#stop_0[db26.or2.omniture.com ]=(null) Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.236.8 2 Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.236.9 (null) Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- /cib: @num_updates=9 Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_perform_op: ++ @. https://github.com/id @. https://github.com/id @.** https://github.com/id='status-1']: Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=db25.or2.omniture.com/attrd/299, version=1.236.9) Jul 22 13:14:43 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: CIB update 299 result for last-failure-aws_eip#stop_0: OK | rc=0 Jul 22 13:14:43 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: last-failure-aws_eip#stop_0[db25.or2.omniture.com ]=1658520883 Jul 22 13:14:43 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: * last-failure-aws_eip#stop_0[db26.or2.omniture.com ]=(null) Jul 22 13:14:48 [13261] db25.or2.omniture.com cib: info: cib_process_ping: Reporting our current digest to db26.or2.omniture.com: 9dfd95c7549aebdc13f725c2f137d0d1 for 1.236.9 (0x562ec48d2090 0) Jul 22 13:15:07 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.236.9 2 Jul 22 13:15:07 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.236.10 (null) Jul 22 13:15:07 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- /cib: @num_updates=10 Jul 22 13:15:07 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- @. @. @. @. https://github.com/id='aws_eip_last_0']: @transition-magic=2:1;3:215:0:600465a1-1af3-4f13-b91b-ceab68ecdc8f, @call-id=447, @RC-Code https://github.com/RC-Code=1, @op-status=2, @exec-time=30003 Jul 22 13:15:07 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- @. @. @. @. https://github.com/id='aws_eip_last_failure_0']: @operation_key=aws_eip_stop_0, @operation https://github.com/operation=stop, @transition-key=3:215:0:600465a1-1af3-4f13-b91b-ceab68ecdc8f, @transition-magic=2:1;3:215:0:600465a1-1af3-4f13-b91b-ceab68ecdc8f, @call-id=447, @last-run=1658520877, @last-rc-change=1658520877, @exec-time=30003 Jul 22 13:15:07 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=db26.or2.omniture.com/crmd/837, version=1.236.10) Jul 22 13:15:07 [13264] db25.or2.omniture.com attrd: info: attrd_peer_update: Setting fail-count-aws_eip#stop_0[db26.or2.omniture.com]: (null) -> INFINITY from db26.or2.omniture.com Jul 22 13:15:07 [13264] db25.or2.omniture.com attrd: info: write_attribute: Sent CIB request 300 with 2 changes for fail-count-aws_eip#stop_0 (id n/a, set n/a) Jul 22 13:15:07 [13264] db25.or2.omniture.com attrd: info: attrd_peer_update: Setting last-failure-aws_eip#stop_0[ db26.or2.omniture.com]: (null) -> 1658520907 from db26.or2.omniture.com Jul 22 13:15:07 [13264] db25.or2.omniture.com attrd: info: write_attribute: Sent CIB request 301 with 2 changes for last-failure-aws_eip#stop_0 (id n/a, set n/a) Jul 22 13:15:07 [13261] db25.or2.omniture.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/attrd/300) Jul 22 13:15:07 [13261] db25.or2.omniture.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/attrd/301) Jul 22 13:15:07 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.236.10 2 Jul 22 13:15:07 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.236.11 (null) Jul 22 13:15:07 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- /cib: @num_updates=11 Jul 22 13:15:07 [13261] db25.or2.omniture.com cib: info: cib_perform_op: ++ @. https://github.com/id @. https://github.com/id @.** https://github.com/id='status-2']: Jul 22 13:15:07 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=db25.or2.omniture.com/attrd/300, version=1.236.11) Jul 22 13:15:07 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: CIB update 300 result for fail-count-aws_eip#stop_0: OK | rc=0 Jul 22 13:15:07 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: fail-count-aws_eip#stop_0[db25.or2.omniture.com ]=INFINITY Jul 22 13:15:07 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: * fail-count-aws_eip#stop_0[db26.or2.omniture.com ]=INFINITY Jul 22 13:15:07 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.236.11 2 Jul 22 13:15:07 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.236.12 (null) Jul 22 13:15:07 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- /cib: @num_updates=12 Jul 22 13:15:07 [13261] db25.or2.omniture.com cib: info: cib_perform_op: ++ @. https://github.com/id @. https://github.com/id @.** https://github.com/id='status-2']: Jul 22 13:15:07 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=db25.or2.omniture.com/attrd/301, version=1.236.12) Jul 22 13:15:07 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: CIB update 301 result for last-failure-aws_eip#stop_0: OK | rc=0 Jul 22 13:15:07 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: last-failure-aws_eip#stop_0[db25.or2.omniture.com ]=1658520883 Jul 22 13:15:07 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: * last-failure-aws_eip#stop_0[db26.or2.omniture.com ]=1658520907 Jul 22 13:15:12 [13261] db25.or2.omniture.com cib: info: cib_process_ping: Reporting our current digest to db26.or2.omniture.com: 743520915e8a23fa141f2b9d759e5e49 for 1.236.12 (0x562ec48d2090 0) Jul 22 13:47:42 [13261] db25.or2.omniture.com cib: info: cib_process_request: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 22 13:47:42 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.236.12 2 Jul 22 13:47:42 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.237.0 514ba7c689336bda8deed1b093c339c0 Jul 22 13:47:42 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- /cib: @epoch https://github.com/epoch=237, @num_updates=0 Jul 22 13:47:42 [13261] db25.or2.omniture.com cib: info: cib_perform_op: ++ /cib/configuration/constraints: <rsc_order first="ms_mysql" first-action="promote" id="order-ms_mysql-aws_eip-mandatory" then="aws_eip" then-action="start"/> Jul 22 13:47:42 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_replace operation for section configuration: OK (rc=0, origin=db25.or2.omniture.com/cibadmin/2, version=1.237.0) Jul 22 13:47:42 [13261] db25.or2.omniture.com cib: info: cib_file_backup: Archived previous version as /var/lib/pacemaker/cib/cib-95.raw Jul 22 13:47:42 [13261] db25.or2.omniture.com cib: info: cib_file_write_with_digest: Wrote version 1.237.0 of the CIB to disk (digest: 73d2eb377937149427fa85e1c5ea2d0d) Jul 22 13:47:42 [13261] db25.or2.omniture.com cib: info: cib_file_write_with_digest: Reading cluster configuration file /var/lib/pacemaker/cib/cib.KFJjZN (digest: /var/lib/pacemaker/cib/cib.eXU9f1) Jul 22 13:47:47 [13261] db25.or2.omniture.com cib: info: cib_process_ping: Reporting our current digest to db26.or2.omniture.com: 514ba7c689336bda8deed1b093c339c0 for 1.237.0 (0x562ec4879f50 0) Jul 22 13:48:09 [13264] db25.or2.omniture.com attrd: info: attrd_peer_update: Setting last-failure-aws_eip#stop_0[ db26.or2.omniture.com]: 1658520907 -> (null) from db26.or2.omniture.com Jul 22 13:48:09 [13264] db25.or2.omniture.com attrd: info: write_attribute: Sent CIB request 302 with 2 changes for last-failure-aws_eip#stop_0 (id n/a, set n/a) Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/attrd/302) Jul 22 13:48:09 [13264] db25.or2.omniture.com attrd: info: attrd_peer_update: Setting fail-count-aws_eip#stop_0[db26.or2.omniture.com]: INFINITY -> (null) from db26.or2.omniture.com Jul 22 13:48:09 [13264] db25.or2.omniture.com attrd: info: write_attribute: Sent CIB request 303 with 2 changes for fail-count-aws_eip#stop_0 (id n/a, set n/a) Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/attrd/303) Jul 22 13:48:09 [13264] db25.or2.omniture.com attrd: info: attrd_peer_update: Setting last-failure-aws_eip#stop_0[ db25.or2.omniture.com]: 1658520883 -> (null) from db26.or2.omniture.com Jul 22 13:48:09 [13264] db25.or2.omniture.com attrd: info: write_attribute: Write out of 'last-failure-aws_eip#stop_0' delayed: update 302 in progress Jul 22 13:48:09 [13264] db25.or2.omniture.com attrd: info: attrd_peer_update: Setting fail-count-aws_eip#stop_0[db25.or2.omniture.com]: INFINITY -> (null) from db26.or2.omniture.com Jul 22 13:48:09 [13264] db25.or2.omniture.com attrd: info: write_attribute: Write out of 'fail-count-aws_eip#stop_0' delayed: update 303 in progress Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.237.0 2 Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.237.1 (null) Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op: -- @. @. @.*** https://github.com/id='aws_eip'] Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- /cib: @num_updates=1 Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_delete operation for section @. https://github.com/uname='db26.or2.omniture.com'] @. https://github.com/id='aws_eip']: OK (rc=0, origin=db26.or2.omniture.com/crmd/844, version=1.237.0) Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.237.0 2 Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.237.1 (null) Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op: -- @. https://github.com/id @. https://github.com/id @.*** https://github.com/id='status-2']/nvpair[ @id https://github.com/id='status-2-last-failure-aws_eip.stop_0'] Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- /cib: @num_updates=1 Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=db25.or2.omniture.com/attrd/302, version=1.237.1) Jul 22 13:48:09 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: CIB update 302 result for last-failure-aws_eip#stop_0: OK | rc=0 Jul 22 13:48:09 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: last-failure-aws_eip#stop_0[db25.or2.omniture.com ]=1658520883 Jul 22 13:48:09 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: last-failure-aws_eip#stop_0[db26.or2.omniture.com ]=(null) Jul 22 13:48:09 [13264] db25.or2.omniture.com attrd: info: write_attribute: Sent CIB request 304 with 2 changes for last-failure-aws_eip#stop_0 (id n/a, set n/a) Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.237.1 2 Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.237.2 (null) Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op: -- @. https://github.com/id @. https://github.com/id @.*** https://github.com/id='status-2']/nvpair[ @id https://github.com/id='status-2-fail-count-aws_eip.stop_0'] Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- /cib: @num_updates=2 Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=db25.or2.omniture.com/attrd/303, version=1.237.2) Jul 22 13:48:09 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: CIB update 303 result for fail-count-aws_eip#stop_0: OK | rc=0 Jul 22 13:48:09 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: fail-count-aws_eip#stop_0[db25.or2.omniture.com ]=INFINITY Jul 22 13:48:09 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: fail-count-aws_eip#stop_0[db26.or2.omniture.com ]=(null) Jul 22 13:48:09 [13264] db25.or2.omniture.com attrd: info: write_attribute: Sent CIB request 305 with 2 changes for fail-count-aws_eip#stop_0 (id n/a, set n/a) Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.237.2 2 Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.237.3 (null) Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op: -- @. @. @.*** https://github.com/id='aws_eip'] Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- /cib: @num_updates=3 Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_delete operation for section @. https://github.com/uname='db26.or2.omniture.com'] @. https://github.com/id='aws_eip']: OK (rc=0, origin=db26.or2.omniture.com/crmd/845, version=1.237.3) Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.237.3 2 Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.238.0 (null) Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- /cib: @epoch https://github.com/epoch=238, @num_updates=0 Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- @. @. https://github.com/id='cib-bootstrap-options-last-lrm-refresh']: @value https://github.com/value=1658522889 Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=db26.or2.omniture.com/crmd/847, version=1.238.0) Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_process_request: Forwarding cib_delete operation for section @. https://github.com/uname='db25.or2.omniture.com'] @. https://github.com/id='aws_eip'] to all (origin=local/crmd/380) Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/attrd/304) Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/attrd/305) Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.238.0 2 Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.238.1 (null) Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op: -- @. @. @.*** https://github.com/id='aws_eip'] Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- /cib: @num_updates=1 Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_delete operation for section @. https://github.com/uname='db25.or2.omniture.com'] @. https://github.com/id='aws_eip']: OK (rc=0, origin=db25.or2.omniture.com/crmd/380, version=1.238.0) Jul 22 13:48:09 [13266] db25.or2.omniture.com crmd: info: delete_resource: Removing resource aws_eip for aef2720d-b36f-4fa6-a10d-fef392066011 (root) on db26.or2.omniture.com Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.238.0 2 Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.238.1 (null) Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op: -- @. https://github.com/id @. https://github.com/id @.*** https://github.com/id='status-1']/nvpair[ @id https://github.com/id='status-1-last-failure-aws_eip.stop_0'] Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- /cib: @num_updates=1 Jul 22 13:48:09 [13266] db25.or2.omniture.com crmd: info: controld_delete_resource_history: Clearing resource history for aws_eip on db25.or2.omniture.com (via CIB call 381) | @. https://github.com/uname='db25.or2.omniture.com'] @. https://github.com/id='aws_eip'] Jul 22 13:48:09 [13266] db25.or2.omniture.com crmd: info: notify_deleted: Notifying aef2720d-b36f-4fa6-a10d-fef392066011 on db26.or2.omniture.com that aws_eip was deleted Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=db25.or2.omniture.com/attrd/304, version=1.238.1) Jul 22 13:48:09 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: CIB update 304 result for last-failure-aws_eip#stop_0: OK | rc=0 Jul 22 13:48:09 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: last-failure-aws_eip#stop_0[db25.or2.omniture.com ]=(null) Jul 22 13:48:09 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: last-failure-aws_eip#stop_0[db26.or2.omniture.com ]=(null) Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.238.1 2 Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.238.2 (null) Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op: -- @. https://github.com/id @. https://github.com/id @.*** https://github.com/id='status-1']/nvpair[ @id https://github.com/id='status-1-fail-count-aws_eip.stop_0'] Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- /cib: @num_updates=2 Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=db25.or2.omniture.com/attrd/305, version=1.238.2) Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_process_request: Forwarding cib_delete operation for section @. https://github.com/uname='db25.or2.omniture.com'] @. https://github.com/id='aws_eip'] to all (origin=local/crmd/381) Jul 22 13:48:09 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: CIB update 305 result for fail-count-aws_eip#stop_0: OK | rc=0 Jul 22 13:48:09 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: fail-count-aws_eip#stop_0[db25.or2.omniture.com ]=(null) Jul 22 13:48:09 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: fail-count-aws_eip#stop_0[db26.or2.omniture.com ]=(null) Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.238.2 2 Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.238.3 (null) Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op: -- @. @. @.*** https://github.com/id='aws_eip'] Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- /cib: @num_updates=3 Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_delete operation for section @. https://github.com/uname='db25.or2.omniture.com'] @. https://github.com/id='aws_eip']: OK (rc=0, origin=db25.or2.omniture.com/crmd/381, version=1.238.3) Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_process_request: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/383) Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=db25.or2.omniture.com/crmd/383, version=1.238.3) Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_file_backup: Archived previous version as /var/lib/pacemaker/cib/cib-96.raw Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_file_write_with_digest: Wrote version 1.238.0 of the CIB to disk (digest: 32138ea0606429a16d4d07a258de99c3) Jul 22 13:48:09 [13261] db25.or2.omniture.com cib: info: cib_file_write_with_digest: Reading cluster configuration file /var/lib/pacemaker/cib/cib.AgX9Et (digest: /var/lib/pacemaker/cib/cib.m01FsT) Jul 22 13:48:10 [13263] db25.or2.omniture.com lrmd: info: process_lrmd_get_rsc_info: Resource 'aws_eip' not found (1 active resources) Jul 22 13:48:10 [13263] db25.or2.omniture.com lrmd: info: process_lrmd_rsc_register: Added 'aws_eip' to the rsc list (2 active resources) Jul 22 13:48:10 [13266] db25.or2.omniture.com crmd: info: do_lrm_rsc_op: Performing key=3:222:7:600465a1-1af3-4f13-b91b-ceab68ecdc8f op=aws_eip_monitor_0 Jul 22 13:48:10 [13261] db25.or2.omniture.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/385) Jul 22 13:48:10 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.238.3 2 Jul 22 13:48:10 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.238.4 (null) Jul 22 13:48:10 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- /cib: @num_updates=4 Jul 22 13:48:10 [13261] db25.or2.omniture.com cib: info: cib_perform_op: ++ @. @. https://github.com/id='1']/lrm_resources: <lrm_resource id="aws_eip" type="awseip" class="ocf" provider="heartbeat"/> Jul 22 13:48:10 [13261] db25.or2.omniture.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="aws_eip_last_0" operation_key="aws_eip_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="3:222:7:600465a1-1af3-4f13-b91b-ceab68ecdc8f" transition-magic="-1:193;3:222:7:600465a1-1af3-4f13-b91b-ceab68ecdc8f" exit-reason="" on_node="db25.or2.omniture.com" call-id="-1" rc-code="193" op-st Jul 22 13:48:10 [13261] db25.or2.omniture.com cib: info: cib_perform_op: ++ Jul 22 13:48:10 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=db25.or2.omniture.com/crmd/385, version=1.238.4) Jul 22 13:48:10 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.238.4 2 Jul 22 13:48:10 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.238.5 (null) Jul 22 13:48:10 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- /cib: @num_updates=5 Jul 22 13:48:10 [13261] db25.or2.omniture.com cib: info: cib_perform_op: ++ @. @. https://github.com/id='2']/lrm_resources: <lrm_resource id="aws_eip" type="awseip" class="ocf" provider="heartbeat"/> Jul 22 13:48:10 [13261] db25.or2.omniture.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="aws_eip_last_0" operation_key="aws_eip_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="4:222:7:600465a1-1af3-4f13-b91b-ceab68ecdc8f" transition-magic="-1:193;4:222:7:600465a1-1af3-4f13-b91b-ceab68ecdc8f" exit-reason="" on_node="db26.or2.omniture.com" call-id="-1" rc-code="193" op-st Jul 22 13:48:10 [13261] db25.or2.omniture.com cib: info: cib_perform_op: ++ Jul 22 13:48:10 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=db26.or2.omniture.com/crmd/857, version=1.238.5) Jul 22 13:48:13 [13263] db25.or2.omniture.com lrmd: notice: operation_finished: aws_eip_monitor_0:15692:stderr [ % Total % Received % Xferd Average Speed Time Time Time Current ] Jul 22 13:48:13 [13263] db25.or2.omniture.com lrmd: notice: operation_finished: aws_eip_monitor_0:15692:stderr [ Dload Upload Total Spent Left Speed ] Jul 22 13:48:13 [13263] db25.or2.omniture.com lrmd: notice: operation_finished: aws_eip_monitor_0:156100 56 100 56 0 0 27040 0 --:--:-- --:--:-- --:--:-- 56000 ] Jul 22 13:48:13 [13266] db25.or2.omniture.com crmd: notice: process_lrm_event: Result of probe operation for aws_eip on db25.or2.omniture.com: 0 (ok) | call=487 key=aws_eip_monitor_0 confirmed=true cib-update=386 Jul 22 13:48:13 [13266] db25.or2.omniture.com crmd: info: exec_alert_list: Sending resource alert via alert_file to /var/log/cluster/pcmk_alert_file.log Jul 22 13:48:13 [13261] db25.or2.omniture.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/386) Jul 22 13:48:13 [13263] db25.or2.omniture.com lrmd: info: process_lrmd_alert_exec: Executing alert alert_file for 18d3dbc1-e6ef-4e45-b801-ae2dd6c9273c Jul 22 13:48:13 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.238.5 2 Jul 22 13:48:13 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.238.6 (null) Jul 22 13:48:13 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- /cib: @num_updates=6 Jul 22 13:48:13 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- @. @. @. @. https://github.com/id='aws_eip_last_0']: @transition-magic=0:0;3:222:7:600465a1-1af3-4f13-b91b-ceab68ecdc8f, @call-id=487, @RC-Code https://github.com/RC-Code=0, @op-status=0, @exec-time=2834 Jul 22 13:48:13 [13261] db25.or2.omniture.com cib: info: cib_perform_op: ++ @. @. @.*** https://github.com/id='aws_eip']: <lrm_rsc_op id="aws_eip_last_failure_0" operation_key="aws_eip_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="3:222:7:600465a1-1af3-4f13-b91b-ceab68ecdc8f" transition-magic="0:0;3:222:7:600465a1-1af3-4f13-b91b-ceab68ecdc8f" exit-reason="" on_node="db25.or2.omniture.com" c Jul 22 13:48:13 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=db25.or2.omniture.com/crmd/386, version=1.238.6) Jul 22 13:48:18 [13261] db25.or2.omniture.com cib: info: cib_process_ping: Reporting our current digest to db26.or2.omniture.com: 9d9b0333341223e2bfd0680ba1137856 for 1.238.6 (0x562ec49f46f0 0) Jul 22 13:48:50 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.238.6 2 Jul 22 13:48:50 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.238.7 (null) Jul 22 13:48:50 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- /cib: @num_updates=7 Jul 22 13:48:50 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- @. @. @. @. https://github.com/id='aws_eip_last_0']: @transition-magic=2:1;4:222:7:600465a1-1af3-4f13-b91b-ceab68ecdc8f, @call-id=453, @RC-Code https://github.com/RC-Code=1, @op-status=2, @exec-time=40002 Jul 22 13:48:50 [13261] db25.or2.omniture.com cib: info: cib_perform_op: ++ @. @. @.*** https://github.com/id='aws_eip']: <lrm_rsc_op id="aws_eip_last_failure_0" operation_key="aws_eip_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="4:222:7:600465a1-1af3-4f13-b91b-ceab68ecdc8f" transition-magic="2:1;4:222:7:600465a1-1af3-4f13-b91b-ceab68ecdc8f" exit-reason="" on_node="db26.or2.omniture.com" c Jul 22 13:48:50 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=db26.or2.omniture.com/crmd/858, version=1.238.7) Jul 22 13:48:50 [13266] db25.or2.omniture.com crmd: info: do_lrm_rsc_op: Performing key=34:223:0:600465a1-1af3-4f13-b91b-ceab68ecdc8f op=aws_eip_stop_0 Jul 22 13:48:50 [13263] db25.or2.omniture.com lrmd: info: log_execute: executing - rsc:aws_eip action:stop call_id:489 Jul 22 13:48:50 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.238.7 2 Jul 22 13:48:50 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.238.8 (null) Jul 22 13:48:50 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- /cib: @num_updates=8 Jul 22 13:48:50 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- @. @. @. @. https://github.com/id='aws_eip_last_0']: @operation_key=aws_eip_stop_0, @operation https://github.com/operation=stop, @transition-key=3:223:0:600465a1-1af3-4f13-b91b-ceab68ecdc8f, @transition-magic=-1:193;3:223:0:600465a1-1af3-4f13-b91b-ceab68ecdc8f, @call-id=-1, @RC-Code https://github.com/RC-Code=193, @op-status=-1, @last-run=1658522930, @last-rc-change=1658522930, @exec-time=0 Jul 22 13:48:50 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=db26.or2.omniture.com/crmd/860, version=1.238.8) Jul 22 13:48:50 [13261] db25.or2.omniture.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/387) Jul 22 13:48:50 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.238.8 2 Jul 22 13:48:50 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.238.9 (null) Jul 22 13:48:50 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- /cib: @num_updates=9 Jul 22 13:48:50 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- @. @. @. @. https://github.com/id='aws_eip_last_0']: @operation_key=aws_eip_stop_0, @operation https://github.com/operation=stop, @transition-key=34:223:0:600465a1-1af3-4f13-b91b-ceab68ecdc8f, @transition-magic=-1:193;34:223:0:600465a1-1af3-4f13-b91b-ceab68ecdc8f, @call-id=-1, @RC-Code https://github.com/RC-Code=193, @op-status=-1, @last-run=1658522930, @last-rc-change=1658522930, @exec-time=0 Jul 22 13:48:50 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=db25.or2.omniture.com/crmd/387, version=1.238.9) Jul 22 13:48:55 [13261] db25.or2.omniture.com cib: info: cib_process_ping: Reporting our current digest to db26.or2.omniture.com: 2ca4791724020b9afd04277cb182b8ee for 1.238.9 (0x562ec49f46f0 0) Jul 22 13:48:56 [13263] db25.or2.omniture.com lrmd: notice: operation_finished: aws_eip_stop_0:15808:stderr [ % Total % Received % Xferd Average Speed Time Time Time Current ] Jul 22 13:48:56 [13263] db25.or2.omniture.com lrmd: notice: operation_finished: aws_eip_stop_0:15808:stderr [ Dload Upload Total Spent Left Speed ] Jul 22 13:48:56 [13263] db25.or2.omniture.com lrmd: notice: operation_finished: aws_eip_stop_0:15808:100 56 100 56 0 0 71519 0 --:--:-- --:--:-- --:--:-- 56000 ] Jul 22 13:48:56 [13263] db25.or2.omniture.com lrmd: notice: operation_finished: aws_eip_stop_0:15808:stderr [ ] Jul 22 13:48:56 [13263] db25.or2.omniture.com lrmd: notice: operation_finished: aws_eip_stop_0:15808:stderr [ An error occurred (UnauthorizedOperation) when calling the DisassociateAddress operation: You are not authorized to perform this operation. Encoded authorization failure message: pwzJpefrK-OBkN09iwXMerRsHYZLPx_GXTJk6bKtSSoaQvCMF0wd6rFcf18MFqKQfOUIYAoMAB62RuIZ-RAlP8L5LinRwXgKYPQPpTiCvqTlobUpwuUQCM6xeCoJuu0PkXBIYzv2_yAYTY9VWAxVpNdhnjql-Z5dNY5ALZnvSEqJaP4hc8DUt4QKUnSALwGY36oF3MGx-XL8Ekus2evzjG Jul 22 13:48:56 [13263] db25.or2.omniture.com lrmd: info: log_finished: finished - rsc:aws_eip action:stop call_id:489 pid:15808 exit-code:7 exec-time:5649ms queue-time:0ms Jul 22 13:48:56 [13266] db25.or2.omniture.com crmd: notice: process_lrm_event: Result of stop operation for aws_eip on db25.or2.omniture.com: 7 (not running) | call=489 key=aws_eip_stop_0 confirmed=true cib-update=388 Jul 22 13:48:56 [13266] db25.or2.omniture.com crmd: notice: process_lrm_event: db25.or2.omniture.com-aws_eip_stop_0:489 [ % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 56 100 56 0 0 71519 0 --:--:-- --:--:-- --:--:-- 56000\n\nAn error occurred (UnauthorizedOperation) when calling th Jul 22 13:48:56 [13266] db25.or2.omniture.com crmd: info: exec_alert_list: Sending resource alert via alert_file to /var/log/cluster/pcmk_alert_file.log Jul 22 13:48:56 [13261] db25.or2.omniture.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/388) Jul 22 13:48:56 [13263] db25.or2.omniture.com lrmd: info: process_lrmd_alert_exec: Executing alert alert_file for 18d3dbc1-e6ef-4e45-b801-ae2dd6c9273c Jul 22 13:48:56 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.238.9 2 Jul 22 13:48:56 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.238.10 (null) Jul 22 13:48:56 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- /cib: @num_updates=10 Jul 22 13:48:56 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- @. @. @. @. https://github.com/id='aws_eip_last_0']: @transition-magic=0:7;34:223:0:600465a1-1af3-4f13-b91b-ceab68ecdc8f, @call-id=489, @RC-Code https://github.com/RC-Code=7, @op-status=0, @exec-time=5649 Jul 22 13:48:56 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- @. @. @. @. https://github.com/id='aws_eip_last_failure_0']: @operation_key=aws_eip_stop_0, @operation https://github.com/operation=stop, @transition-key=34:223:0:600465a1-1af3-4f13-b91b-ceab68ecdc8f, @transition-magic=0:7;34:223:0:600465a1-1af3-4f13-b91b-ceab68ecdc8f, @call-id=489, @RC-Code https://github.com/RC-Code=7, @last-run=1658522930, @last-rc-change=1658522930, @exec-time=5649 Jul 22 13:48:56 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=db25.or2.omniture.com/crmd/388, version=1.238.10) Jul 22 13:48:56 [13264] db25.or2.omniture.com attrd: info: attrd_peer_update: Setting fail-count-aws_eip#stop_0[db25.or2.omniture.com]: (null) -> INFINITY from db26.or2.omniture.com Jul 22 13:48:56 [13264] db25.or2.omniture.com attrd: info: write_attribute: Sent CIB request 306 with 2 changes for fail-count-aws_eip#stop_0 (id n/a, set n/a) Jul 22 13:48:56 [13264] db25.or2.omniture.com attrd: info: attrd_peer_update: Setting last-failure-aws_eip#stop_0[ db25.or2.omniture.com]: (null) -> 1658522936 from db26.or2.omniture.com Jul 22 13:48:56 [13261] db25.or2.omniture.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/attrd/306) Jul 22 13:48:56 [13264] db25.or2.omniture.com attrd: info: write_attribute: Sent CIB request 307 with 2 changes for last-failure-aws_eip#stop_0 (id n/a, set n/a) Jul 22 13:48:56 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.238.10 2 Jul 22 13:48:56 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.238.11 (null) Jul 22 13:48:56 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- /cib: @num_updates=11 Jul 22 13:48:56 [13261] db25.or2.omniture.com cib: info: cib_perform_op: ++ @. https://github.com/id @. https://github.com/id @.** https://github.com/id='status-1']: Jul 22 13:48:56 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=db25.or2.omniture.com/attrd/306, version=1.238.11) Jul 22 13:48:56 [13261] db25.or2.omniture.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/attrd/307) Jul 22 13:48:56 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: CIB update 306 result for fail-count-aws_eip#stop_0: OK | rc=0 Jul 22 13:48:56 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: fail-count-aws_eip#stop_0[db25.or2.omniture.com ]=INFINITY Jul 22 13:48:56 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: * fail-count-aws_eip#stop_0[db26.or2.omniture.com ]=(null) Jul 22 13:48:56 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.238.11 2 Jul 22 13:48:56 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.238.12 (null) Jul 22 13:48:56 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- /cib: @num_updates=12 Jul 22 13:48:56 [13261] db25.or2.omniture.com cib: info: cib_perform_op: ++ @. https://github.com/id @. https://github.com/id @.** https://github.com/id='status-1']: Jul 22 13:48:56 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=db25.or2.omniture.com/attrd/307, version=1.238.12) Jul 22 13:48:56 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: CIB update 307 result for last-failure-aws_eip#stop_0: OK | rc=0 Jul 22 13:48:56 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: last-failure-aws_eip#stop_0[db25.or2.omniture.com ]=1658522936 Jul 22 13:48:56 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: * last-failure-aws_eip#stop_0[db26.or2.omniture.com ]=(null) Jul 22 13:49:01 [13261] db25.or2.omniture.com cib: info: cib_process_ping: Reporting our current digest to db26.or2.omniture.com: 455324d285e382bb511b6c0f7e2bf763 for 1.238.12 (0x562ec49f46f0 0) Jul 22 13:49:20 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.238.12 2 Jul 22 13:49:20 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.238.13 (null) Jul 22 13:49:20 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- /cib: @num_updates=13 Jul 22 13:49:20 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- @. @. @. @. https://github.com/id='aws_eip_last_0']: @transition-magic=2:1;3:223:0:600465a1-1af3-4f13-b91b-ceab68ecdc8f, @call-id=455, @RC-Code https://github.com/RC-Code=1, @op-status=2, @exec-time=30003 Jul 22 13:49:20 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- @. @. @. @. https://github.com/id='aws_eip_last_failure_0']: @operation_key=aws_eip_stop_0, @operation https://github.com/operation=stop, @transition-key=3:223:0:600465a1-1af3-4f13-b91b-ceab68ecdc8f, @transition-magic=2:1;3:223:0:600465a1-1af3-4f13-b91b-ceab68ecdc8f, @call-id=455, @last-run=1658522930, @last-rc-change=1658522930, @exec-time=30003 Jul 22 13:49:20 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=db26.or2.omniture.com/crmd/861, version=1.238.13) Jul 22 13:49:20 [13264] db25.or2.omniture.com attrd: info: attrd_peer_update: Setting fail-count-aws_eip#stop_0[db26.or2.omniture.com]: (null) -> INFINITY from db26.or2.omniture.com Jul 22 13:49:20 [13264] db25.or2.omniture.com attrd: info: write_attribute: Sent CIB request 308 with 2 changes for fail-count-aws_eip#stop_0 (id n/a, set n/a) Jul 22 13:49:20 [13264] db25.or2.omniture.com attrd: info: attrd_peer_update: Setting last-failure-aws_eip#stop_0[ db26.or2.omniture.com]: (null) -> 1658522960 from db26.or2.omniture.com Jul 22 13:49:20 [13264] db25.or2.omniture.com attrd: info: write_attribute: Sent CIB request 309 with 2 changes for last-failure-aws_eip#stop_0 (id n/a, set n/a) Jul 22 13:49:20 [13261] db25.or2.omniture.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/attrd/308) Jul 22 13:49:20 [13261] db25.or2.omniture.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/attrd/309) Jul 22 13:49:20 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.238.13 2 Jul 22 13:49:20 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.238.14 (null) Jul 22 13:49:20 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- /cib: @num_updates=14 Jul 22 13:49:20 [13261] db25.or2.omniture.com cib: info: cib_perform_op: ++ @. https://github.com/id @. https://github.com/id @.** https://github.com/id='status-2']: Jul 22 13:49:20 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=db25.or2.omniture.com/attrd/308, version=1.238.14) Jul 22 13:49:20 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: CIB update 308 result for fail-count-aws_eip#stop_0: OK | rc=0 Jul 22 13:49:20 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: fail-count-aws_eip#stop_0[db25.or2.omniture.com ]=INFINITY Jul 22 13:49:20 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: * fail-count-aws_eip#stop_0[db26.or2.omniture.com ]=INFINITY Jul 22 13:49:20 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.238.14 2 Jul 22 13:49:20 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.238.15 (null) Jul 22 13:49:20 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- /cib: @num_updates=15 Jul 22 13:49:20 [13261] db25.or2.omniture.com cib: info: cib_perform_op: ++ @. https://github.com/id @. https://github.com/id @.** https://github.com/id='status-2']: Jul 22 13:49:20 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=db25.or2.omniture.com/attrd/309, version=1.238.15) Jul 22 13:49:20 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: CIB update 309 result for last-failure-aws_eip#stop_0: OK | rc=0 Jul 22 13:49:20 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: last-failure-aws_eip#stop_0[db25.or2.omniture.com ]=1658522936 Jul 22 13:49:20 [13264] db25.or2.omniture.com attrd: info: attrd_cib_callback: * last-failure-aws_eip#stop_0[db26.or2.omniture.com ]=1658522960 Jul 22 13:49:25 [13261] db25.or2.omniture.com cib: info: cib_process_ping: Reporting our current digest to db26.or2.omniture.com: 535aab8ec227ffb8796b23b11350f283 for 1.238.15 (0x562ec49f46f0 0) Jul 22 13:56:23 [13261] db25.or2.omniture.com cib: info: cib_process_request: Forwarding cib_replace operation for section configuration to all (origin=local/cibadmin/2) Jul 22 13:56:23 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.238.15 2 Jul 22 13:56:23 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.239.0 59e95fd99a2f4a74eff7b3483486ae34 Jul 22 13:56:23 [13261] db25.or2.omniture.com cib: info: cib_perform_op:
- /cib: @epoch https://github.com/epoch=239, @num_updates=0 Jul 22 13:56:23 [13261] db25.or2.omniture.com cib: info: cib_perform_op: ++ /cib/configuration/constraints: <rsc_colocation id="colocation-aws_eip-ms_mysql-INFINITY" rsc="aws_eip" rsc-role="Started" score="INFINITY" with-rsc="ms_mysql" with-rsc-role="Master"/> Jul 22 13:56:23 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_replace operation for section configuration: OK (rc=0, origin=db25.or2.omniture.com/cibadmin/2, version=1.239.0) Jul 22 13:56:23 [13261] db25.or2.omniture.com cib: info: cib_file_backup: Archived previous version as /var/lib/pacemaker/cib/cib-97.raw Jul 22 13:56:23 [13261] db25.or2.omniture.com cib: info: cib_file_write_with_digest: Wrote version 1.239.0 of the CIB to disk (digest: 878006632767c93013d1f711bac202a6) Jul 22 13:56:23 [13261] db25.or2.omniture.com cib: info: cib_file_write_with_digest: Reading cluster configuration file /var/lib/pacemaker/cib/cib.CqkyD5 (digest: /var/lib/pacemaker/cib/cib.unvkwf) Jul 22 13:56:28 [13261] db25.or2.omniture.com cib: info: cib_process_ping: Reporting our current digest to db26.or2.omniture.com: 59e95fd99a2f4a74eff7b3483486ae34 for 1.239.0 (0x562ec49f4ce0 0)
— Reply to this email directly, view it on GitHub https://github.com/ClusterLabs/resource-agents/issues/1794, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADRZB2FJCVP2FJB6NC3ZMHTVVMDWHANCNFSM54MX6CFA . You are receiving this because you were mentioned.Message ID: @.***>
infa-rammekal
commented
2 years ago Thank you, so much for all your help, Oyvind Albrigtsen. We finally figured it out the issue. it issue with our end. from the current node we are not able to use aws cli because it not exposed to internet. Internet allowed to public ip which eip is pointed other node. that is why all actions failing on this node. With AWS help, created endpoints to corresponding vpc subnet which allowed to run aws commands.
oalbrigt
commented
2 years ago Nice. Glad to hear you found and solved the issue.
AWSEIP is not able to disassociate and associate EIP between the nodes. And also keep erroring out.
Here is the commands I issued after setting corosync/pacemaker/mysql-M/S.
pcs resource create aws_eip ocf:heartbeat:awseip elastic_ip=35.86.13.200 allocation_id=eipalloc-052c7f5348fc5d26c awscli="$(which aws)" --group aws_group op monitor timeout="40s" interval="20s" OCF_CHECK_LEVEL="10"
pcs constraint colocation add aws_eip with master ms_mysql pcs constraint order promote ms_mysql then start aws_eip
Stack: corosync Current DC: db26 (version 1.1.23-1.el7_9.1-9acf116022) - part ition with quorum Last updated: Fri Jul 22 13:49:22 2022 Last change: Fri Jul 22 13:48:09 2022 by hacluster via crmd on db26.or2.omnitu re.com
2 nodes configured 4 resource instances configured (1 BLOCKED from further action due to failure)
Online: [ db25.or2.omniture.com db26.or2.omniture.com ]
Active resources:
Master/Slave Set: ms_mysql [p_mysql] Masters: [ db25] Slaves: [ db26 ] Resource Group: aws_group aws_eip (ocf::heartbeat:awseip): FAILED (blocked)[ db25.or2.omn iture.com db26. ]
Failed Resource Actions:
here is corosync.log output..
@last-run=1658520877, @last-rc-change=1658520877, @exec-time=0 Jul 22 13:14:37 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=db25.or2.omniture.com/crmd/378, version=1.236.6) Jul 22 13:14:42 [13261] db25.or2.omniture.com cib: info: cib_process_ping: Reporting our current digest to db26.or2.omniture.com: ee253e652fc7abb3220c5d8613f25f39 for 1.236.6 (0x562ec48d2090 0) Jul 22 13:14:43 [13263] db25.or2.omniture.com lrmd: notice: operation_finished: aws_eip_stop_0:3035:stderr [ % Total % Received % Xferd Average Speed Time Time Time Current ] Jul 22 13:14:43 [13263] db25.or2.omniture.com lrmd: notice: operation_finished: aws_eip_stop_0:3035:stderr [ Dload Upload Total Spent Left Speed ] Jul 22 13:14:43 [13263] db25.or2.omniture.com lrmd: notice: operation_finished: aws_eip_stop_0:3035:s100 56 100 56 0 0 31728 0 --:--:-- --:--:-- --:--:-- 56000 ] Jul 22 13:14:43 [13263] db25.or2.omniture.com lrmd: notice: operation_finished: aws_eip_stop_0:3035:stderr [ ] Jul 22 13:14:43 [13263] db25.or2.omniture.com lrmd: notice: operation_finished: aws_eip_stop_0:3035:stderr [ An error occurred (UnauthorizedOperation) when calling the DisassociateAddress operation: You are not authorized to perform this operation. Encoded authorization failure message: fY4uLP5oYTunADmj5AoKUKnIYIt0fgaOjCVjVSfp4Q-zIObsDYLYOZZi94n_8nGVoRuBqv-zpykdtQbq6-TmwkH65_QzJVD9xgF1wFPxO-2vPTXJm0tPvqoZeM9c1og_qCX2zR0Qx8OvAoyZyF5UiDMgnAfrWMsVb7t5lP43DZcN-bFsyVDe-7GdZd0tl7nApFYZL-cZA6WGDRomQB7pOEX Jul 22 13:14:43 [13263] db25.or2.omniture.com lrmd: info: log_finished: finished - rsc:aws_eip action:stop call_id:481 pid:3035 exit-code:7 exec-time:6071ms queue-time:0ms Jul 22 13:14:43 [13266] db25.or2.omniture.com crmd: notice: process_lrm_event: Result of stop operation for aws_eip on db25.or2.omniture.com: 7 (not running) | call=481 key=aws_eip_stop_0 confirmed=true cib-update=379 Jul 22 13:14:43 [13266] db25.or2.omniture.com crmd: notice: process_lrm_event: db25.or2.omniture.com-aws_eip_stop_0:481 [ % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 56 100 56 0 0 31728 0 --:--:-- --:--:-- --:--:-- 56000\n\nAn error occurred (UnauthorizedOperation) when calling th Jul 22 13:14:43 [13266] db25.or2.omniture.com crmd: info: exec_alert_list: Sending resource alert via alert_file to /var/log/cluster/pcmk_alert_file.log Jul 22 13:14:43 [13263] db25.or2.omniture.com lrmd: info: process_lrmd_alert_exec: Executing alert alert_file for 18d3dbc1-e6ef-4e45-b801-ae2dd6c9273c Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/379) Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.236.6 2 Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.236.7 (null) Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_perform_op: + /cib: @num_updates=7 Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='aws_eip']/lrm_rsc_op[@id='aws_eip_last_0']: @transition-magic=0:7;34:215:0:600465a1-1af3-4f13-b91b-ceab68ecdc8f, @call-id=481, @rc-code=7, @op-status=0, @exec-time=6071 Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='aws_eip']/lrm_rsc_op[@id='aws_eip_last_failure_0']: @operation_key=aws_eip_stop_0, @operation=stop, @transition-key=34:215:0:600465a1-1af3-4f13-b91b-ceab68ecdc8f, @transition-magic=0:7;34:215:0:600465a1-1af3-4f13-b91b-ceab68ecdc8f, @call-id=481, @rc-code=7, @last-run=1658520877, @last-rc-change=1658520877, @exec-time=6071 Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=db25.or2.omniture.com/crmd/379, version=1.236.7) Jul 22 13:14:43 [13264] db25.or2.omniture.com attrd: info: attrd_peer_update: Setting fail-count-aws_eip#stop_0[db25.or2.omniture.com]: (null) -> INFINITY from db26.or2.omniture.com Jul 22 13:14:43 [13264] db25.or2.omniture.com attrd: info: write_attribute: Sent CIB request 298 with 2 changes for fail-count-aws_eip#stop_0 (id n/a, set n/a) Jul 22 13:14:43 [13264] db25.or2.omniture.com attrd: info: attrd_peer_update: Setting last-failure-aws_eip#stop_0[db25.or2.omniture.com]: (null) -> 1658520883 from db26.or2.omniture.com Jul 22 13:14:43 [13264] db25.or2.omniture.com attrd: info: write_attribute: Sent CIB request 299 with 2 changes for last-failure-aws_eip#stop_0 (id n/a, set n/a) Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/attrd/298) Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/attrd/299) Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: --- 1.236.7 2 Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_perform_op: Diff: +++ 1.236.8 (null) Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_perform_op: + /cib: @num_updates=8 Jul 22 13:14:43 [13261] db25.or2.omniture.com cib: info: cib_perform_op: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: