search

sonic-net / sonic-swss

SONiC Switch State Service (SwSS)
https://azure.github.io/SONiC
Other
175 stars 526 forks source link

Couldn't enable zmq for orchagent in docker-sonic-vs #2014

Open mint570 opened 2 years ago

mint570 commented 2 years ago

We tried to enable zmq for sync mode. We tested in docker-sonic-vs but found that orchagent was still using Redis (while syncd could successfully use zmq). We can see the zmq logging in syncd but not in orchagent.

Observation:

  1. Syncd command in docker-sonic-vs: "/usr/bin/syncd -u -z zmq_sync -p /usr/share/sonic/hwsku/sai.profile"
  2. Orchagent command in docker-sonic-vs: "/usr/bin/orchagent -d /var/log/swss -b 8192 -z zmq_sync -m 02:42:ac:11:00:02"
  3. Orchagent log in docker-sonic-vs: Nov 8 00:53:17.842864 e1e70738c505 INFO #supervisord 2021-11-08 00:53:17,842 INFO spawned: 'orchagent' with pid 88 Nov 8 00:53:18.061411 e1e70738c505 NOTICE #orchagent: :- main: --- Starting Orchestration Agent --- Nov 8 00:53:18.061455 e1e70738c505 NOTICE #orchagent: :- loadFromFile: no context config specified, will load default context config Nov 8 00:53:18.061464 e1e70738c505 NOTICE #orchagent: :- insert: added switch: idx 0, hwinfo '' Nov 8 00:53:18.062521 e1e70738c505 NOTICE #orchagent: :- RedisChannel: creating notification thread Nov 8 00:53:18.062683 e1e70738c505 NOTICE #orchagent: :- clear_local_state: clearing local state Nov 8 00:53:18.062749 e1e70738c505 NOTICE #orchagent: :- setRecordingFilename: setting recording filename : sairedis.rec Nov 8 00:53:18.062757 e1e70738c505 NOTICE #orchagent: :- stopRecording: stopped recording Nov 8 00:53:18.062893 e1e70738c505 NOTICE #orchagent: :- startRecording: started recording: sairedis.rec Nov 8 00:53:18.062903 e1e70738c505 NOTICE #orchagent: :- initSaiRedis: Enable redis pipeline Nov 8 00:53:18.062913 e1e70738c505 NOTICE #orchagent: :- notifySyncd: sending syncd: INIT_VIEW Nov 8 00:53:18.845971 e1e70738c505 INFO #supervisord 2021-11-08 00:53:18,845 INFO success: orchagent entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) Nov 8 00:53:18.848134 e1e70738c505 INFO #/supervisord: start.sh orchagent: started Nov 8 00:54:18.123076 e1e70738c505 ERR #orchagent: :- wait: SELECT operation result: TIMEOUT on notify Nov 8 00:54:18.123099 e1e70738c505 ERR #orchagent: :- wait: failed to get response for notify Nov 8 00:54:18.123232 e1e70738c505 ERR #orchagent: :- initSaiRedis: Failed to notify syncd INIT_VIEW, rv:-1 gSwitchId 0 Nov 8 00:54:18.123774 e1e70738c505 NOTICE #orchagent: :- uninitialize: begin Nov 8 00:54:18.123796 e1e70738c505 NOTICE #orchagent: :- uninitialize: begin Nov 8 00:54:18.123831 e1e70738c505 NOTICE #orchagent: :- ~RedisChannel: join ntf thread begin Nov 8 00:54:18.124010 e1e70738c505 NOTICE #orchagent: :- ~RedisChannel: join ntf thread end Nov 8 00:54:18.124118 e1e70738c505 NOTICE #orchagent: :- clear_local_state: clearing local state Nov 8 00:54:18.124149 e1e70738c505 NOTICE #orchagent: :- meta_init_db: begin Nov 8 00:54:18.124176 e1e70738c505 NOTICE #orchagent: :- meta_init_db: end Nov 8 00:54:18.124187 e1e70738c505 NOTICE #orchagent: :- uninitialize: end Nov 8 00:54:18.124242 e1e70738c505 NOTICE #orchagent: :- stopRecording: stopped recording Nov 8 00:54:18.124271 e1e70738c505 NOTICE #orchagent: :- stopRecording: closed recording file: sairedis.rec Nov 8 00:54:18.124292 e1e70738c505 NOTICE #orchagent: :- uninitialize: end Nov 8 00:54:19.119257 e1e70738c505 INFO #supervisord 2021-11-08 00:54:19,118 INFO exited: orchagent (exit status 1; not expected)

We used a snapshot of the 202106 version. The sonic-buildimage hash is 7916276330bd23d287465621028bd33077087039 (https://github.com/Azure/sonic-buildimage/tree/7916276330bd23d287465621028bd33077087039). The following is the corresponding submodules: sonic-swss-common: 11db9ba6403b92effd060bea33b4ca9812465f60 (https://github.com/Azure/sonic-swss-common/tree/11db9ba6403b92effd060bea33b4ca9812465f60) sonic-swss: 3226163654e51a09a24a2228fdc21c967ec998b5 (https://github.com/Azure/sonic-swss/tree/3226163654e51a09a24a2228fdc21c967ec998b5) sonic-sairedis: 0e2105a8e5c6e14c3551ec2bf65b1ad41bb5060b (https://github.com/Azure/sonic-sairedis/tree/0e2105a8e5c6e14c3551ec2bf65b1ad41bb5060b)

Our steps:

  1. In sonic-buildimage file platform/vs/docker-sonic-vs/orchagent.sh, replace "-s" with "-z zmq_sync" (https://github.com/Azure/sonic-buildimage/blob/7916276330bd23d287465621028bd33077087039/platform/vs/docker-sonic-vs/orchagent.sh#L30)
  2. In sonic-sairedis file syncd/scripts/syncd_init_common.sh, replace "-s" with "-z zmq_sync" (https://github.com/Azure/sonic-sairedis/blob/0e2105a8e5c6e14c3551ec2bf65b1ad41bb5060b/syncd/scripts/syncd_init_common.sh#L39)
  3. In sonic-buildimage file files/build_templates/init_cfg.json.j2, add a line, "synchronous_mode": "enable", into DEVICE_METADATA|localhost. This will enable sync mode. (https://github.com/Azure/sonic-buildimage/blob/7916276330bd23d287465621028bd33077087039/files/build_templates/init_cfg.json.j2#L6)
  4. Build the vs docker and run.
kcudnik commented 2 years ago

are you still passing "-s" parameter to orchagent? since that will override zmq if set by -z param, also could you provide syslog from entire orchagent in main.cpp there are those lines:

483 attr.id = SAI_REDIS_SWITCH_ATTR_REDIS_COMMUNICATION_MODE; 484 attr.value.s32 = gRedisCommunicationMode; 485 486 sai_switch_api->set_switch_attribute(gSwitchId, &attr);

which set communication mode, and in syslog there should be one of those messages logged: (from /sonic-sairedis/lib/RedisRemoteSaiInterface.cpp) SWSS_LOG_NOTICE("enabling redis async mode"); or SWSS_LOG_NOTICE("enabling redis sync mode"); or SWSS_LOG_NOTICE("zmq enabled, forcing sync mode") or SWSS_LOG_ERROR("invalid communication mode value

and those lines should be after main: --- Starting Orchestration Agent log line which branch on sonic-swss/ are you using ?, the above log lines are from master branch

kcudnik commented 2 years ago

can you confirm that branch you are using with OA is setting communication mode flag ? since i dont see any ofh the swss_log_notice messages in your pasted logs, so it seems like different branch is used which don't have this code implemented yet

kcudnik commented 2 years ago

PS. notifications in ZMQ mode will not work because of this issie: https://github.com/Azure/sonic-swss/issues/1145 which still was not fixed :/

PS2. why you want to use zmq?

mint570 commented 2 years ago

We didn't pass "-s". We just replace "-s" with "-z zmq_sync". I included the branch we used in the description. (It is a snapshot in the 202106 branch.) I do see the main.cpp code that makes the SAI call. I also see the logging code in RedisRemoteSaiInterface.cpp as well.

The orchagent log in the description is the full log. From the logging, the RedisRemoteSaiInterface.cpp function didn't seem to get called at all. I noticed that this message was in the log: "#orchagent: :- loadFromFile: no context config specified, will load default context config". Does that explain something?

Performance is the main reason why we want to use zmq. We tried to experiment the performance improvement if we enable zmq.

I have another question on zmq (unrelated to the orchagent issue). If we can successfully enable zmq, do we still record entries in ASIC DB? How about the sairedis.rec file? Thanks.

kcudnik commented 2 years ago

yea, entries will still be put in ASIC DB. as for log, this no default context don't matter, each communication mode should be changed, not sure why it's not logged, even if -s option itself is specified, log message should be in the syslog that it was changed, not sure why this is not happening

mint570 commented 2 years ago

If we only pass "-s", we do see the RedisRemoteSaiInterface.cpp log. Here is the log if we do "-s": 2021-11-09T09:38:18.028182-08:00 ju3u1m1 swss#orchagent: :- main: Enabling synchronous mode 2021-11-09T09:38:18.028198-08:00 ju3u1m1 swss#orchagent: :- main: --- Starting Orchestration Agent --- 2021-11-09T09:38:18.028963-08:00 ju3u1m1 swss#orchagent: :- loadFromFile: no context config specified, will load default context config 2021-11-09T09:38:18.028983-08:00 ju3u1m1 swss#orchagent: :- insert: added switch: idx 0, hwinfo '' 2021-11-09T09:38:18.030058-08:00 ju3u1m1 swss#orchagent: :- RedisChannel: creating notification thread 2021-11-09T09:38:18.030208-08:00 ju3u1m1 swss#orchagent: :- clear_local_state: clearing local state 2021-11-09T09:38:18.030309-08:00 ju3u1m1 swss#orchagent: :- setRecordingFilename: setting recording filename : sairedis.rec 2021-11-09T09:38:18.030322-08:00 ju3u1m1 swss#orchagent: :- stopRecording: stopped recording 2021-11-09T09:38:18.030454-08:00 ju3u1m1 swss#orchagent: :- startRecording: started recording: sairedis.rec 2021-11-09T09:38:18.030468-08:00 ju3u1m1 swss#orchagent: :- initSaiRedis: Enable redis pipeline 2021-11-09T09:38:18.030485-08:00 ju3u1m1 swss#orchagent: :- notifySyncd: sending syncd: INIT_VIEW 2021-11-09T09:38:18.031404-08:00 ju3u1m1 swss#orchagent: :- sai_redis_notify_syncd: switched ASIC to INIT VIEW 2021-11-09T09:38:18.031413-08:00 ju3u1m1 swss#orchagent: :- sai_redis_notify_syncd: clearing current local state since init view is called on initialized switch 2021-11-09T09:38:18.031418-08:00 ju3u1m1 swss#orchagent: :- clear_local_state: clearing local state 2021-11-09T09:38:18.031432-08:00 ju3u1m1 swss#orchagent: :- meta_init_db: begin 2021-11-09T09:38:18.031450-08:00 ju3u1m1 swss#orchagent: :- meta_init_db: end 2021-11-09T09:38:18.031459-08:00 ju3u1m1 swss#orchagent: :- initSaiRedis: Notify syncd INIT_VIEW 2021-11-09T09:38:18.032034-08:00 ju3u1m1 swss#orchagent: :- main: sync mode is depreacated, use -z param 2021-11-09T09:38:18.032044-08:00 ju3u1m1 swss#orchagent: :- set: unlocking api mutex for communication mode 2021-11-09T09:38:18.032072-08:00 ju3u1m1 swss#orchagent: :- ~RedisChannel: join ntf thread begin 2021-11-09T09:38:18.032186-08:00 ju3u1m1 swss#orchagent: :- ~RedisChannel: join ntf thread end 2021-11-09T09:38:18.032231-08:00 ju3u1m1 swss#orchagent: :- setRedisExtensionAttribute: enabling redis sync mode 2021-11-09T09:38:18.033200-08:00 ju3u1m1 swss#orchagent: :- RedisChannel: creating notification thread 2021-11-09T09:38:18.033269-08:00 ju3u1m1 swss#orchagent: :- create: request switch create with context 0 2021-11-09T09:38:18.033382-08:00 ju3u1m1 swss#orchagent: :- allocateNewSwitchObjectId: created SWITCH VID oid:0x21000000000000 for hwinfo: '' 2021-11-09T09:38:18.034052-08:00 ju3u1m1 swss#orchagent: :- Switch: created switch with hwinfo = '' 2021-11-09T09:38:18.034116-08:00 ju3u1m1 swss#orchagent: :- main: Create a switch, id:9288674231451648 2021-11-09T09:38:18.034808-08:00 ju3u1m1 swss#orchagent: :- main: Get switch virtual router ID 3000000000032 2021-11-09T09:38:18.035829-08:00 ju3u1m1 swss#orchagent: :- main: Created underlay router interface ID 6000000000866

In the "-s" log, the RedisRemoteSaiInterface.cpp log happens after "~RedisChannel: join ntf thread end". However, in the "-z" log, orchagent already starts "uninitialize: begin" before that (due to "Failed to notify syncd INIT_VIEW"). Syncd is in zmq mode in the "-z" case. Is it the same as the notification issue you pointed out in https://github.com/Azure/sonic-swss/issues/1145? That's why orchagent couldn't receive the INIT_VIEW response?

kcudnik commented 2 years ago

no, init_view should be working fine, since it's over zmq channel not over notification channel but with -s param you can clearly see:

main: sync mode is depreacated, use -z param
setRedisExtensionAttribute: enabling redis sync mode
mint570 commented 2 years ago

Syncd didn't seem to receive the init_view. Seems like zmq was not working? I ran the docker-sonic-vs directly on my machine. Is there anything I need to do on the machine to enable zmq support?

mint570 commented 2 years ago

Some new update. I tried syncd with the "-s" flag and orchagent with the "-z" flag, then INIT_VIEW worked! This is the orchagent log with "-s" in syncd and "-z" in orchagent: Nov 11 17:37:14.230278 bdb8071f02f3 NOTICE #orchagent: :- main: --- Starting Orchestration Agent --- Nov 11 17:37:14.230902 bdb8071f02f3 NOTICE #orchagent: :- loadFromFile: no context config specified, will load default context config Nov 11 17:37:14.230910 bdb8071f02f3 NOTICE #orchagent: :- insert: added switch: idx 0, hwinfo '' Nov 11 17:37:14.231764 bdb8071f02f3 NOTICE #orchagent: :- RedisChannel: creating notification thread Nov 11 17:37:14.231888 bdb8071f02f3 NOTICE #orchagent: :- clear_local_state: clearing local state Nov 11 17:37:14.231929 bdb8071f02f3 NOTICE #orchagent: :- setRecordingFilename: setting recording filename : sairedis.rec Nov 11 17:37:14.231935 bdb8071f02f3 NOTICE #orchagent: :- stopRecording: stopped recording Nov 11 17:37:14.232037 bdb8071f02f3 NOTICE #orchagent: :- startRecording: started recording: sairedis.rec Nov 11 17:37:14.232044 bdb8071f02f3 NOTICE #orchagent: :- initSaiRedis: Enable redis pipeline Nov 11 17:37:14.232051 bdb8071f02f3 NOTICE #orchagent: :- notifySyncd: sending syncd: INIT_VIEW Nov 11 17:37:14.232716 bdb8071f02f3 NOTICE #orchagent: :- sai_redis_notify_syncd: switched ASIC to INIT VIEW Nov 11 17:37:14.232721 bdb8071f02f3 NOTICE #orchagent: :- sai_redis_notify_syncd: clearing current local state since init view is called on initialized switch Nov 11 17:37:14.232724 bdb8071f02f3 NOTICE #orchagent: :- clear_local_state: clearing local state Nov 11 17:37:14.232731 bdb8071f02f3 NOTICE #orchagent: :- meta_init_db: begin Nov 11 17:37:14.232740 bdb8071f02f3 NOTICE #orchagent: :- meta_init_db: end Nov 11 17:37:14.232743 bdb8071f02f3 NOTICE #orchagent: :- initSaiRedis: Notify syncd INIT_VIEW Nov 11 17:37:14.233228 bdb8071f02f3 NOTICE #orchagent: :- set: unlocking api mutex for communication mode Nov 11 17:37:14.233238 bdb8071f02f3 NOTICE #orchagent: :- ~RedisChannel: join ntf thread begin Nov 11 17:37:14.233352 bdb8071f02f3 NOTICE #orchagent: :- ~RedisChannel: join ntf thread end Nov 11 17:37:14.235338 bdb8071f02f3 NOTICE #orchagent: :- ZeroMQChannel: opening zmq main endpoint: ipc:///tmp/zmq_ep Nov 11 17:37:14.235478 bdb8071f02f3 NOTICE #orchagent: :- ZeroMQChannel: opening zmq ntf endpoint: ipc:///tmp/zmq_ntf_ep Nov 11 17:37:14.235542 bdb8071f02f3 NOTICE #orchagent: :- ZeroMQChannel: creating notification thread Nov 11 17:37:14.235573 bdb8071f02f3 NOTICE #orchagent: :- setRedisExtensionAttribute: zmq enabled, forcing sync mode Nov 11 17:37:14.235577 bdb8071f02f3 NOTICE #orchagent: :- setRedisExtensionAttribute: disabling buffered pipeline in sync mode Nov 11 17:37:14.235586 bdb8071f02f3 NOTICE #orchagent: :- create: request switch create with context 0 Nov 11 17:37:14.235644 bdb8071f02f3 NOTICE #orchagent: :- allocateNewSwitchObjectId: created SWITCH VID oid:0x21000000000000 for hwinfo: '' Nov 11 17:37:14.235684 bdb8071f02f3 NOTICE #orchagent: :- notificationThreadFunction: start listening for notifications Nov 11 17:37:15.022934 bdb8071f02f3 INFO #supervisord 2021-11-11 17:37:15,022 INFO success: orchagent entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) Nov 11 17:37:15.025090 bdb8071f02f3 INFO #/supervisord: start.sh orchagent: started Nov 11 17:38:14.292138 bdb8071f02f3 ERR #orchagent: :- wait: zmq_poll timed out for: getresponse Nov 11 17:38:14.292208 bdb8071f02f3 ERR #orchagent: :- create: create status: SAI_STATUS_FAILURE Nov 11 17:38:14.292227 bdb8071f02f3 ERR #orchagent: :- main: Failed to create a switch, rv:-1 Nov 11 17:38:14.293249 bdb8071f02f3 NOTICE #orchagent: :- uninitialize: begin

However, zmq_poll timed out. That is expected since syncd is "-s". So looks like orchagent sends INIT_VIEW in Redis (instead of zmq) even with zmq enabled.

mint570 commented 2 years ago

Checking the code.

Orchagent sents INIT_VIEW during initSaiRedis (https://github.com/Azure/sonic-swss/blob/37c197d00d42d205243852827de63244c053e7df/orchagent/saihelper.cpp#L321). INIT_VIEW is only useful in warmboot but orchagent will send it in both warmboot and coldboot.

By default, RedisRemoteSaiInterface uses Redis during initialization, unless we overwrite it in context config (https://github.com/Azure/sonic-sairedis/blob/6f1a0eaf3662c5d736262769e9f7cf922cb07e0d/lib/RedisRemoteSaiInterface.cpp#L73). This is the reason that orchange sends INIT_VIEW in Redis. Later, RedisRemoteSaiInterface should change to use zmq in setRedisExtensionAttribute if orchagent makes the SAI call. However, orchagent already fails initSaiRedis and gives up.

To correctly config zmq, I guess we have to provide a context config that enables zmq?

kcudnik commented 2 years ago

depends, since there is default cofiguration for zmq, if you want to pass specific, then you will require to pass that config file

for zmq, both syncd and OA should be used with -z flag and "zmq_sync", unless there is some bug we dont know about

MaratGubaiev commented 2 years ago

@kcudnik @mint570 ¶Can we reproduce the issue on a real platform, let's say on the mellanox platform? ¶Also, can we reproduce the issue using sonic-mgmt-container? (Or could you provide documentation of how to use docker-sonic-vs?) ¶Orchagent log in docker-sonic-vs How to get this log? ¶Is this the first usage of zmq?

BTW, frr uses zmq too. Could it affect the matter somehow?

As for me, I have tried the steps + I have changed also dockers/docker-orchagent/orchagent.sh Then I checked build by sudo monit status. On mellanox after some time it showed

Program 'container_checker'
  status                       Status ok
  monitoring status            Monitored
  monitoring mode              active
  on reboot                    start
  last exit value              0
  last output                  -
  data collected               Tue, 30 Nov 2021 09:58:34

/usr/bin/syncd -u -z zmq_sync -p /tmp/sai.profile #in the syncd container /usr/bin/orchagent -d /var/log/swss -b 8192 -z zmq_sync #in the swss container

On vs it showed

Program 'container_checker'
  status                       Status failed
  monitoring status            Monitored
  monitoring mode              active
  on reboot                    start
  last exit value              3
  last output                  Expected containers not running: swss, teamd, dhcp_relay, radv
  data collected               Tue, 30 Nov 2021 10:15:38

/usr/bin/syncd -u -z zmq_sync -p /usr/share/sonic/hwsku/sai.profile #in the syncd container

orchagent not found

Also I have checked docker logs swss, docker logs syncd and the logs in /var/log/swss and have not found anything similar to your logs.

Now I should check it on the vs platform without my additions.

kcudnik commented 2 years ago

i dont think there should be a conflict with frr zmq if using different channel names

MaratGubaiev commented 2 years ago

@kcudnik The problem is with zmq or with docker-sonic-vs? Did you check how it works on other platforms?

kcudnik commented 2 years ago

didn't check other platforms, not sure if anyone is using zmq channel right now, and three should be no issues, the conflict im mentioning could happen if 2 different processes would use the same channel name

MaratGubaiev commented 2 years ago

@kcudnik ¶OK, I believe that frr's zmq doesn't affect the matter. ¶How to check if zmq works? Can monit status show it? ¶~May I create a fake pull request to see how it passes the tests?~ UPD It's impossible to just create a PR because sonic-sairedis is a submodule. Maybe there is another way to run the tests?

@mint570 ¶What does monit status show on your virtual device? ¶Where did you get that logs? ¶Could you provide documentation of how to use docker-sonic-vs?

MaratGubaiev commented 2 years ago

@prsunny @zhangyanzhao How can we run tests without creating pull request and merging?

The point is that I want to test this changes but they are made in two repos so pull request testing will stop because of merge conflict as it was here: https://github.com/Azure/sonic-buildimage/pull/9285

kcudnik commented 2 years ago

to check if zmq is enabled in syncd/OA you would need to check syslog, sairdis library and syncd logs notice message whether zmq or redis channel is used. there is no external way to check that as for tests there is no easy way to create full pipeline test with OA and syncd enabled zmq, since that requires configuration scripts to pass different parameters, and im not sure if i can help here, since my testing limits to sairedis library itself and tests there, where saiplayer acts as OA, and there is no need for swss package at all, but it should be possible to write vstests that would setup all configuration in right place

MaratGubaiev commented 2 years ago

@kcudnik I wanted to see how the standard pull request Azure pipeline would react on the steps proposed in the original comment.

mint570 commented 2 years ago

Sorry for the late reply. We saw the issue in both virtual and hardware platforms. I mentioned docker-sonic-vs here because I think we might have different hardware but the virtual one should be similar.

We have successfully enabled zmq by making some changes:

  1. Provide a context config file for orchagent and syncd to enable zmq. This is explained in my previous comment. Orchagent failed to perform syncd INIT_VIEW because by default zmq is disabled. https://github.com/Azure/sonic-sairedis/blob/26a8a1204e873109537c81462ad1457cf38c2f9e/lib/ContextConfig.cpp#L20 https://github.com/Azure/sonic-swss/blob/d352d5a9180a6eb8d5ea81af6e5332a2c8ba2dbe/orchagent/saihelper.cpp#L332 We provided a context config that enables zmq, so that orchagent will send INIT_VIEW in zmq. The context config in orchagent is hardcoded to be "/usr/share/sonic/hwsku/context_config.json": https://github.com/Azure/sonic-swss/blob/d352d5a9180a6eb8d5ea81af6e5332a2c8ba2dbe/orchagent/saihelper.cpp#L30

  2. Change this line to not exit: https://github.com/Azure/sonic-swss/blob/d352d5a9180a6eb8d5ea81af6e5332a2c8ba2dbe/orchagent/saihelper.cpp#L302 SAI_REDIS_SWITCH_ATTR_USE_PIPELINE is not supported in syncd mode: https://github.com/Azure/sonic-sairedis/blob/26a8a1204e873109537c81462ad1457cf38c2f9e/lib/RedisRemoteSaiInterface.cpp#L442

  3. In notification.cpp file, https://github.com/Azure/sonic-swss/blob/d352d5a9180a6eb8d5ea81af6e5332a2c8ba2dbe/orchagent/notifications.cpp, we forward the event into the old Redis channel. This is to address https://github.com/Azure/sonic-swss/issues/1145. What we did was to send the event in Redis channel in the handler function if it is SAI_REDIS_COMMUNICATION_MODE_ZMQ_SYNC mode. This might not be the best approach, but it worked.

The docker-sonic-vs.gz can be built by setting PLATFORM=vs option: https://github.com/Azure/sonic-buildimage. I just run the vs docker manually after loading the docker image: docker run --rm=true --privileged -i docker-sonic-vs:latest I can get into the vs docker: docker exec -i "docker ID" bash -C (The docker ID can be found by running "docker ps".) Inside the vs docker, I can check the log by doing "show logging". But I need to install sudo first. Just do "apt-get install -y sudo".)

MaratGubaiev commented 2 years ago

@mint570 Thank you!

We have successfully enabled zmq

Does it mean that the issue is closed?

BTW

Here is what have in log on my mellanox device:

``` admin@sonic:~$ show logging | grep -i "zmq\|ZeroMQ" Dec 2 03:32:16.942590 sonic NOTICE swss#orchagent: :- zmqPollThread: ending pool thread, since run is false Dec 2 03:32:16.942590 sonic NOTICE swss#orchagent: :- zmqPollThread: end Dec 2 03:32:16.943052 sonic NOTICE swss#orchagent: :- ~ZeroMQSelectableChannel: ending zmq poll thread for channel ipc:///tmp/saiServer Dec 2 03:32:16.943272 sonic NOTICE swss#orchagent: :- ~ZeroMQSelectableChannel: ended zmq poll thread for channel ipc:///tmp/saiServer Dec 2 03:33:36.883775 sonic NOTICE swss#orchagent: :- ZeroMQSelectableChannel: binding on ipc:///tmp/saiServer Dec 2 03:33:36.893411 sonic NOTICE swss#orchagent: :- zmqPollThread: begin Dec 2 03:33:41.107164 sonic NOTICE syncd#syncd: :- Syncd: command line: EnableDiagShell=NO EnableTempView=YES DisableExitSleep=NO EnableUnittests=NO EnableConsistencyCheck=NO EnableSyncMode=NO RedisCommunicationMode=zmq_sync EnableSaiBulkSuport=NO StartType=cold ProfileMapFile=/tmp/sai.profile GlobalContext=0 ContextConfig= BreakConfig= Dec 2 03:33:41.107164 sonic NOTICE syncd#syncd: :- Syncd: zmq sync mode enabled via cmd line Dec 2 03:33:41.119114 sonic NOTICE syncd#syncd: :- ZeroMQNotificationProducer: opening zmq ntf endpoint: ipc:///tmp/zmq_ntf_ep Dec 2 03:33:41.119114 sonic NOTICE syncd#syncd: :- Syncd: zmq enabled, forcing sync mode Dec 2 03:33:41.119114 sonic NOTICE syncd#syncd: :- ZeroMQSelectableChannel: binding on ipc:///tmp/zmq_ep Dec 2 03:33:41.122643 sonic NOTICE syncd#syncd: :- zmqPollThread: begin Dec 2 03:41:37.000369 sonic NOTICE swss#orchagent: :- zmqPollThread: ending pool thread, since run is false Dec 2 03:41:37.001779 sonic NOTICE swss#orchagent: :- zmqPollThread: end Dec 2 03:41:37.002583 sonic NOTICE swss#orchagent: :- ~ZeroMQSelectableChannel: ending zmq poll thread for channel ipc:///tmp/saiServer Dec 2 03:41:37.002683 sonic NOTICE swss#orchagent: :- ~ZeroMQSelectableChannel: ended zmq poll thread for channel ipc:///tmp/saiServer Dec 2 03:42:57.199125 sonic NOTICE swss#orchagent: :- ZeroMQSelectableChannel: binding on ipc:///tmp/saiServer Dec 2 03:42:57.219128 sonic NOTICE swss#orchagent: :- zmqPollThread: begin Dec 2 03:43:03.791619 sonic NOTICE syncd#syncd: :- Syncd: command line: EnableDiagShell=NO EnableTempView=YES DisableExitSleep=NO EnableUnittests=NO EnableConsistencyCheck=NO EnableSyncMode=NO RedisCommunicationMode=zmq_sync EnableSaiBulkSuport=NO StartType=cold ProfileMapFile=/tmp/sai.profile GlobalContext=0 ContextConfig= BreakConfig= Dec 2 03:43:03.791830 sonic NOTICE syncd#syncd: :- Syncd: zmq sync mode enabled via cmd line Dec 2 03:43:03.792872 sonic NOTICE syncd#syncd: :- ZeroMQNotificationProducer: opening zmq ntf endpoint: ipc:///tmp/zmq_ntf_ep Dec 2 03:43:03.809518 sonic NOTICE syncd#syncd: :- Syncd: zmq enabled, forcing sync mode Dec 2 03:43:03.811236 sonic NOTICE syncd#syncd: :- ZeroMQSelectableChannel: binding on ipc:///tmp/zmq_ep Dec 2 03:43:03.831463 sonic NOTICE syncd#syncd: :- zmqPollThread: begin Dec 2 03:50:57.315938 sonic NOTICE swss#orchagent: :- zmqPollThread: ending pool thread, since run is false Dec 2 03:50:57.315938 sonic NOTICE swss#orchagent: :- zmqPollThread: end Dec 2 03:50:57.315938 sonic NOTICE swss#orchagent: :- ~ZeroMQSelectableChannel: ending zmq poll thread for channel ipc:///tmp/saiServer Dec 2 03:50:57.315938 sonic NOTICE swss#orchagent: :- ~ZeroMQSelectableChannel: ended zmq poll thread for channel ipc:///tmp/saiServer Dec 2 03:52:21.071769 sonic NOTICE swss#orchagent: :- ZeroMQSelectableChannel: binding on ipc:///tmp/saiServer Dec 2 03:52:21.123135 sonic NOTICE swss#orchagent: :- zmqPollThread: begin Dec 2 03:52:24.847322 sonic NOTICE syncd#syncd: :- Syncd: command line: EnableDiagShell=NO EnableTempView=YES DisableExitSleep=NO EnableUnittests=NO EnableConsistencyCheck=NO EnableSyncMode=NO RedisCommunicationMode=zmq_sync EnableSaiBulkSuport=NO StartType=cold ProfileMapFile=/tmp/sai.profile GlobalContext=0 ContextConfig= BreakConfig= Dec 2 03:52:24.848058 sonic NOTICE syncd#syncd: :- Syncd: zmq sync mode enabled via cmd line Dec 2 03:52:24.849897 sonic NOTICE syncd#syncd: :- ZeroMQNotificationProducer: opening zmq ntf endpoint: ipc:///tmp/zmq_ntf_ep Dec 2 03:52:24.852509 sonic NOTICE syncd#syncd: :- Syncd: zmq enabled, forcing sync mode Dec 2 03:52:24.852864 sonic NOTICE syncd#syncd: :- ZeroMQSelectableChannel: binding on ipc:///tmp/zmq_ep Dec 2 03:52:24.888803 sonic NOTICE syncd#SAI: :- zmqPollThread: begin Dec 2 04:00:21.209198 sonic NOTICE swss#orchagent: :- zmqPollThread: ending pool thread, since run is false Dec 2 04:00:21.209198 sonic NOTICE swss#orchagent: :- zmqPollThread: end Dec 2 04:00:21.209198 sonic NOTICE swss#orchagent: :- ~ZeroMQSelectableChannel: ending zmq poll thread for channel ipc:///tmp/saiServer Dec 2 04:00:21.209198 sonic NOTICE swss#orchagent: :- ~ZeroMQSelectableChannel: ended zmq poll thread for channel ipc:///tmp/saiServer Dec 2 04:01:45.438428 sonic NOTICE swss#orchagent: :- ZeroMQSelectableChannel: binding on ipc:///tmp/saiServer Dec 2 04:01:45.438779 sonic NOTICE swss#orchagent: :- zmqPollThread: begin Dec 2 04:01:49.923128 sonic NOTICE syncd#syncd: :- Syncd: command line: EnableDiagShell=NO EnableTempView=YES DisableExitSleep=NO EnableUnittests=NO EnableConsistencyCheck=NO EnableSyncMode=NO RedisCommunicationMode=zmq_sync EnableSaiBulkSuport=NO StartType=cold ProfileMapFile=/tmp/sai.profile GlobalContext=0 ContextConfig= BreakConfig= Dec 2 04:01:49.923128 sonic NOTICE syncd#syncd: :- Syncd: zmq sync mode enabled via cmd line Dec 2 04:01:49.923128 sonic NOTICE syncd#syncd: :- ZeroMQNotificationProducer: opening zmq ntf endpoint: ipc:///tmp/zmq_ntf_ep Dec 2 04:01:49.923128 sonic NOTICE syncd#syncd: :- Syncd: zmq enabled, forcing sync mode Dec 2 04:01:49.923128 sonic NOTICE syncd#syncd: :- ZeroMQSelectableChannel: binding on ipc:///tmp/zmq_ep Dec 2 04:01:49.959153 sonic NOTICE syncd#syncd: :- zmqPollThread: begin Dec 2 04:09:45.556008 sonic NOTICE swss#orchagent: :- zmqPollThread: ending pool thread, since run is false Dec 2 04:09:45.556008 sonic NOTICE swss#orchagent: :- zmqPollThread: end Dec 2 04:09:45.556737 sonic NOTICE swss#orchagent: :- ~ZeroMQSelectableChannel: ending zmq poll thread for channel ipc:///tmp/saiServer Dec 2 04:09:45.556737 sonic NOTICE swss#orchagent: :- ~ZeroMQSelectableChannel: ended zmq poll thread for channel ipc:///tmp/saiServer Dec 2 04:11:06.923214 sonic NOTICE swss#orchagent: :- ZeroMQSelectableChannel: binding on ipc:///tmp/saiServer Dec 2 04:11:06.927370 sonic NOTICE swss#orchagent: :- zmqPollThread: begin Dec 2 04:11:10.858142 sonic NOTICE syncd#syncd: :- Syncd: command line: EnableDiagShell=NO EnableTempView=YES DisableExitSleep=NO EnableUnittests=NO EnableConsistencyCheck=NO EnableSyncMode=NO RedisCommunicationMode=zmq_sync EnableSaiBulkSuport=NO StartType=cold ProfileMapFile=/tmp/sai.profile GlobalContext=0 ContextConfig= BreakConfig= Dec 2 04:11:10.858142 sonic NOTICE syncd#syncd: :- Syncd: zmq sync mode enabled via cmd line Dec 2 04:11:10.858142 sonic NOTICE syncd#syncd: :- ZeroMQNotificationProducer: opening zmq ntf endpoint: ipc:///tmp/zmq_ntf_ep Dec 2 04:11:10.867156 sonic NOTICE syncd#syncd: :- Syncd: zmq enabled, forcing sync mode Dec 2 04:11:10.867156 sonic NOTICE syncd#syncd: :- ZeroMQSelectableChannel: binding on ipc:///tmp/zmq_ep Dec 2 04:11:10.875140 sonic NOTICE syncd#syncd: :- zmqPollThread: begin Dec 2 04:19:07.005090 sonic NOTICE swss#orchagent: :- ~ZeroMQSelectableChannel: ending zmq poll thread for channel ipc:///tmp/saiServer Dec 2 04:19:07.389714 sonic NOTICE swss#orchagent: :- zmqPollThread: ending pool thread, since run is false Dec 2 04:19:07.389963 sonic NOTICE swss#orchagent: :- zmqPollThread: end Dec 2 04:19:07.390327 sonic NOTICE swss#orchagent: :- ~ZeroMQSelectableChannel: ended zmq poll thread for channel ipc:///tmp/saiServer Dec 2 04:20:32.175159 sonic NOTICE swss#orchagent: :- ZeroMQSelectableChannel: binding on ipc:///tmp/saiServer Dec 2 04:20:32.175159 sonic NOTICE swss#orchagent: :- zmqPollThread: begin Dec 2 04:20:36.278394 sonic NOTICE syncd#syncd: :- Syncd: command line: EnableDiagShell=NO EnableTempView=YES DisableExitSleep=NO EnableUnittests=NO EnableConsistencyCheck=NO EnableSyncMode=NO RedisCommunicationMode=zmq_sync EnableSaiBulkSuport=NO StartType=cold ProfileMapFile=/tmp/sai.profile GlobalContext=0 ContextConfig= BreakConfig= Dec 2 04:20:36.278394 sonic NOTICE syncd#syncd: :- Syncd: zmq sync mode enabled via cmd line Dec 2 04:20:36.285241 sonic NOTICE syncd#syncd: :- ZeroMQNotificationProducer: opening zmq ntf endpoint: ipc:///tmp/zmq_ntf_ep Dec 2 04:20:36.285241 sonic NOTICE syncd#syncd: :- Syncd: zmq enabled, forcing sync mode Dec 2 04:20:36.285241 sonic NOTICE syncd#syncd: :- ZeroMQSelectableChannel: binding on ipc:///tmp/zmq_ep Dec 2 04:20:36.289856 sonic NOTICE syncd#syncd: :- zmqPollThread: begin Dec 2 04:28:32.270320 sonic NOTICE swss#orchagent: :- zmqPollThread: ending pool thread, since run is false Dec 2 04:28:32.270320 sonic NOTICE swss#orchagent: :- zmqPollThread: end Dec 2 04:28:32.270320 sonic NOTICE swss#orchagent: :- ~ZeroMQSelectableChannel: ending zmq poll thread for channel ipc:///tmp/saiServer Dec 2 04:28:32.270320 sonic NOTICE swss#orchagent: :- ~ZeroMQSelectableChannel: ended zmq poll thread for channel ipc:///tmp/saiServer Dec 2 04:29:54.019129 sonic NOTICE swss#orchagent: :- ZeroMQSelectableChannel: binding on ipc:///tmp/saiServer Dec 2 04:29:54.032704 sonic NOTICE swss#orchagent: :- zmqPollThread: begin Dec 2 04:29:59.047312 sonic NOTICE syncd#syncd: :- Syncd: command line: EnableDiagShell=NO EnableTempView=YES DisableExitSleep=NO EnableUnittests=NO EnableConsistencyCheck=NO EnableSyncMode=NO RedisCommunicationMode=zmq_sync EnableSaiBulkSuport=NO StartType=cold ProfileMapFile=/tmp/sai.profile GlobalContext=0 ContextConfig= BreakConfig= Dec 2 04:29:59.047312 sonic NOTICE syncd#syncd: :- Syncd: zmq sync mode enabled via cmd line Dec 2 04:29:59.048497 sonic NOTICE syncd#syncd: :- ZeroMQNotificationProducer: opening zmq ntf endpoint: ipc:///tmp/zmq_ntf_ep Dec 2 04:29:59.048497 sonic NOTICE syncd#syncd: :- Syncd: zmq enabled, forcing sync mode Dec 2 04:29:59.048497 sonic NOTICE syncd#syncd: :- ZeroMQSelectableChannel: binding on ipc:///tmp/zmq_ep Dec 2 04:29:59.073836 sonic NOTICE syncd#SAI: :- zmqPollThread: begin Dec 2 04:37:54.135803 sonic NOTICE swss#orchagent: :- zmqPollThread: ending pool thread, since run is false Dec 2 04:37:54.135840 sonic NOTICE swss#orchagent: :- zmqPollThread: end Dec 2 04:37:54.136006 sonic NOTICE swss#orchagent: :- ~ZeroMQSelectableChannel: ending zmq poll thread for channel ipc:///tmp/saiServer Dec 2 04:37:54.136006 sonic NOTICE swss#orchagent: :- ~ZeroMQSelectableChannel: ended zmq poll thread for channel ipc:///tmp/saiServer Dec 2 04:39:15.259158 sonic NOTICE swss#orchagent: :- ZeroMQSelectableChannel: binding on ipc:///tmp/saiServer Dec 2 04:39:15.291123 sonic NOTICE swss#orchagent: :- zmqPollThread: begin Dec 2 04:39:20.107147 sonic NOTICE syncd#syncd: :- Syncd: command line: EnableDiagShell=NO EnableTempView=YES DisableExitSleep=NO EnableUnittests=NO EnableConsistencyCheck=NO EnableSyncMode=NO RedisCommunicationMode=zmq_sync EnableSaiBulkSuport=NO StartType=cold ProfileMapFile=/tmp/sai.profile GlobalContext=0 ContextConfig= BreakConfig= Dec 2 04:39:20.107147 sonic NOTICE syncd#syncd: :- Syncd: zmq sync mode enabled via cmd line Dec 2 04:39:20.107147 sonic NOTICE syncd#syncd: :- ZeroMQNotificationProducer: opening zmq ntf endpoint: ipc:///tmp/zmq_ntf_ep Dec 2 04:39:20.107147 sonic NOTICE syncd#syncd: :- Syncd: zmq enabled, forcing sync mode Dec 2 04:39:20.107147 sonic NOTICE syncd#syncd: :- ZeroMQSelectableChannel: binding on ipc:///tmp/zmq_ep Dec 2 04:39:20.135119 sonic NOTICE syncd#syncd: :- zmqPollThread: begin admin@sonic:~$ ```

Looks like zmq works.

mint570 commented 2 years ago

We can only make it work with some code changes (as I mentioned previously). The main issue is that orchagent exits with error when initSaiRedis fail (in SAI_REDIS_NOTIFY_SYNCD_INIT_VIEW).

It is possible that this issue is specific to us since we made lots of code changes. But I saw the same code in the mainline (as I gave the code links). Not sure how the mellanox device could successfully do SAI_REDIS_NOTIFY_SYNCD_INIT_VIEW. Maybe a timing issue on when orchagent and syncd start?

MaratGubaiev commented 2 years ago

Maybe a timing issue on when orchagent and syncd start?

It takes quite lot of time to make all of the dockers start: 5-10 minutes. So monit status is OK only after that time after reboot.

@mint570, Just for case I made some more logs:

show logging | grep -i "INIT_VIEW"(it seems that I have some troubles)

Dec  3 03:19:49.443609 sonic NOTICE swss#orchagent: :- notifySyncd: sending syncd: INIT_VIEW
Dec  3 03:27:49.537070 sonic ERR swss#orchagent: :- initSaiRedis: Failed to notify syncd INIT_VIEW, rv:-1 gSwitchId 0
Dec  3 03:29:11.303628 sonic NOTICE swss#orchagent: :- notifySyncd: sending syncd: INIT_VIEW
Dec  3 03:37:11.394273 sonic ERR swss#orchagent: :- initSaiRedis: Failed to notify syncd INIT_VIEW, rv:-1 gSwitchId 0
Dec  3 03:39:29.286118 sonic NOTICE swss#orchagent: :- notifySyncd: sending syncd: INIT_VIEW
Dec  3 03:47:29.372309 sonic ERR swss#orchagent: :- initSaiRedis: Failed to notify syncd INIT_VIEW, rv:-1 gSwitchId 0
Dec  3 03:48:51.243272 sonic NOTICE swss#orchagent: :- notifySyncd: sending syncd: INIT_VIEW
Dec  3 03:56:51.332132 sonic ERR swss#orchagent: :- initSaiRedis: Failed to notify syncd INIT_VIEW, rv:-1 gSwitchId 0
Dec  3 03:58:14.859723 sonic NOTICE swss#orchagent: :- notifySyncd: sending syncd: INIT_VIEW
Dec  3 04:06:14.879792 sonic ERR swss#orchagent: :- initSaiRedis: Failed to notify syncd INIT_VIEW, rv:-1 gSwitchId 0
Dec  3 04:07:34.755278 sonic NOTICE swss#orchagent: :- notifySyncd: sending syncd: INIT_VIEW
Dec  3 04:15:34.856445 sonic ERR swss#orchagent: :- initSaiRedis: Failed to notify syncd INIT_VIEW, rv:-1 gSwitchId 0
Dec  3 04:16:56.187252 sonic NOTICE swss#orchagent: :- notifySyncd: sending syncd: INIT_VIEW


show logging | grep -i "swss#orchagent"

Dec  3 03:19:49.430520 sonic NOTICE swss#orchagent: :- main: --- Starting Orchestration Agent ---
Dec  3 03:19:49.430520 sonic NOTICE swss#orchagent: :- loadFromFile: no context config specified, will load default context config
Dec  3 03:19:49.430520 sonic NOTICE swss#orchagent: :- insert: added switch: idx 0, hwinfo ''
Dec  3 03:19:49.443609 sonic NOTICE swss#orchagent: :- RedisChannel: creating notification thread
Dec  3 03:19:49.443609 sonic NOTICE swss#orchagent: :- clear_local_state: clearing local state
Dec  3 03:19:49.443609 sonic NOTICE swss#orchagent: :- setRecordingFilename: setting recording filename : sairedis.rec
Dec  3 03:19:49.443609 sonic NOTICE swss#orchagent: :- stopRecording: stopped recording
Dec  3 03:19:49.443609 sonic NOTICE swss#orchagent: :- startRecording: started recording: sairedis.rec
Dec  3 03:19:49.443609 sonic NOTICE swss#orchagent: :- initSaiRedis: Enable redis pipeline
Dec  3 03:19:49.443609 sonic NOTICE swss#orchagent: :- setRedisExtensionAttribute: set response timeout to 480000 ms
Dec  3 03:19:49.443609 sonic NOTICE swss#orchagent: :- initSaiRedis: SAI REDIS response timeout set successfully to 480000
Dec  3 03:19:49.443609 sonic NOTICE swss#orchagent: :- notifySyncd: sending syncd: INIT_VIEW
Dec  3 03:27:49.536110 sonic ERR swss#orchagent: :- wait: SELECT operation result: TIMEOUT on notify
Dec  3 03:27:49.536366 sonic ERR swss#orchagent: :- wait: failed to get response for notify
Dec  3 03:27:49.537070 sonic ERR swss#orchagent: :- initSaiRedis: Failed to notify syncd INIT_VIEW, rv:-1 gSwitchId 0
Dec  3 03:27:49.537280 sonic NOTICE swss#orchagent: :- uninitialize: begin
Dec  3 03:27:49.537280 sonic NOTICE swss#orchagent: :- uninitialize: begin
Dec  3 03:27:49.537280 sonic NOTICE swss#orchagent: :- ~RedisChannel: join ntf thread begin
Dec  3 03:27:49.538007 sonic NOTICE swss#orchagent: :- ~RedisChannel: join ntf thread end
Dec  3 03:27:49.538007 sonic NOTICE swss#orchagent: :- clear_local_state: clearing local state
Dec  3 03:27:49.538007 sonic NOTICE swss#orchagent: :- meta_init_db: begin
Dec  3 03:27:49.538007 sonic NOTICE swss#orchagent: :- meta_init_db: end
Dec  3 03:27:49.538007 sonic NOTICE swss#orchagent: :- uninitialize: end
Dec  3 03:27:49.538060 sonic NOTICE swss#orchagent: :- stopRecording: stopped recording
Dec  3 03:27:49.538060 sonic NOTICE swss#orchagent: :- stopRecording: closed recording file: sairedis.rec
Dec  3 03:27:49.538092 sonic NOTICE swss#orchagent: :- uninitialize: end
Dec  3 03:29:11.291623 sonic NOTICE swss#orchagent: :- main: --- Starting Orchestration Agent ---
Dec  3 03:29:11.291623 sonic NOTICE swss#orchagent: :- loadFromFile: no context config specified, will load default context config
Dec  3 03:29:11.291623 sonic NOTICE swss#orchagent: :- insert: added switch: idx 0, hwinfo ''
Dec  3 03:29:11.291623 sonic NOTICE swss#orchagent: :- RedisChannel: creating notification thread
Dec  3 03:29:11.303628 sonic NOTICE swss#orchagent: :- clear_local_state: clearing local state
Dec  3 03:29:11.303628 sonic NOTICE swss#orchagent: :- setRecordingFilename: setting recording filename : sairedis.rec
Dec  3 03:29:11.303628 sonic NOTICE swss#orchagent: :- stopRecording: stopped recording
Dec  3 03:29:11.303628 sonic NOTICE swss#orchagent: :- startRecording: started recording: sairedis.rec
Dec  3 03:29:11.303628 sonic NOTICE swss#orchagent: :- initSaiRedis: Enable redis pipeline
Dec  3 03:29:11.303628 sonic NOTICE swss#orchagent: :- setRedisExtensionAttribute: set response timeout to 480000 ms
Dec  3 03:29:11.303628 sonic NOTICE swss#orchagent: :- initSaiRedis: SAI REDIS response timeout set successfully to 480000
Dec  3 03:29:11.303628 sonic NOTICE swss#orchagent: :- notifySyncd: sending syncd: INIT_VIEW
Dec  3 03:37:11.393839 sonic ERR swss#orchagent: :- wait: SELECT operation result: TIMEOUT on notify
Dec  3 03:37:11.393839 sonic ERR swss#orchagent: :- wait: failed to get response for notify
Dec  3 03:37:11.394273 sonic ERR swss#orchagent: :- initSaiRedis: Failed to notify syncd INIT_VIEW, rv:-1 gSwitchId 0
Dec  3 03:37:11.394273 sonic NOTICE swss#orchagent: :- uninitialize: begin
Dec  3 03:37:11.394273 sonic NOTICE swss#orchagent: :- uninitialize: begin
Dec  3 03:37:11.394319 sonic NOTICE swss#orchagent: :- ~RedisChannel: join ntf thread begin
Dec  3 03:37:11.394971 sonic NOTICE swss#orchagent: :- ~RedisChannel: join ntf thread end
Dec  3 03:37:11.394971 sonic NOTICE swss#orchagent: :- clear_local_state: clearing local state
Dec  3 03:37:11.394971 sonic NOTICE swss#orchagent: :- meta_init_db: begin
Dec  3 03:37:11.394971 sonic NOTICE swss#orchagent: :- meta_init_db: end
Dec  3 03:37:11.394971 sonic NOTICE swss#orchagent: :- uninitialize: end
Dec  3 03:37:11.394971 sonic NOTICE swss#orchagent: :- stopRecording: stopped recording
Dec  3 03:37:11.394971 sonic NOTICE swss#orchagent: :- stopRecording: closed recording file: sairedis.rec
Dec  3 03:37:11.395036 sonic NOTICE swss#orchagent: :- uninitialize: end
Dec  3 03:39:29.270406 sonic NOTICE swss#orchagent: :- main: --- Starting Orchestration Agent ---
Dec  3 03:39:29.270406 sonic NOTICE swss#orchagent: :- initialize: acting as a sairedis SERVER
Dec  3 03:39:29.270406 sonic NOTICE swss#orchagent: :- loadFromFile: no context config specified, will load default context config
Dec  3 03:39:29.270406 sonic NOTICE swss#orchagent: :- insert: added switch: idx 0, hwinfo ''
Dec  3 03:39:29.273969 sonic NOTICE swss#orchagent: :- RedisChannel: creating notification thread
Dec  3 03:39:29.274798 sonic NOTICE swss#orchagent: :- clear_local_state: clearing local state
Dec  3 03:39:29.275678 sonic NOTICE swss#orchagent: :- initialize: init client/server sai: SAI_STATUS_SUCCESS
Dec  3 03:39:29.275778 sonic NOTICE swss#orchagent: :- loadFromFile: no server config specified, will load default
Dec  3 03:39:29.275887 sonic NOTICE swss#orchagent: :- ZeroMQSelectableChannel: binding on ipc:///tmp/saiServer
Dec  3 03:39:29.283948 sonic NOTICE swss#orchagent: :- initialize: starting server thread
Dec  3 03:39:29.284090 sonic NOTICE swss#orchagent: :- initialize: init client/server sai: SAI_STATUS_SUCCESS
Dec  3 03:39:29.284245 sonic NOTICE swss#orchagent: :- setRecordingFilename: setting recording filename : sairedis.rec
Dec  3 03:39:29.284329 sonic NOTICE swss#orchagent: :- stopRecording: stopped recording
Dec  3 03:39:29.284626 sonic NOTICE swss#orchagent: :- serverThreadFunction: begin
Dec  3 03:39:29.286118 sonic NOTICE swss#orchagent: :- zmqPollThread: begin
Dec  3 03:39:29.286118 sonic NOTICE swss#orchagent: :- startRecording: started recording: sairedis.rec
Dec  3 03:39:29.286118 sonic NOTICE swss#orchagent: :- initSaiRedis: Enable redis pipeline
Dec  3 03:39:29.286118 sonic NOTICE swss#orchagent: :- setRedisExtensionAttribute: set response timeout to 480000 ms
Dec  3 03:39:29.286118 sonic NOTICE swss#orchagent: :- initSaiRedis: SAI REDIS response timeout set successfully to 480000
Dec  3 03:39:29.286118 sonic NOTICE swss#orchagent: :- notifySyncd: sending syncd: INIT_VIEW
Dec  3 03:47:29.372032 sonic ERR swss#orchagent: :- wait: SELECT operation result: TIMEOUT on notify
Dec  3 03:47:29.372208 sonic ERR swss#orchagent: :- wait: failed to get response for notify
Dec  3 03:47:29.372309 sonic ERR swss#orchagent: :- initSaiRedis: Failed to notify syncd INIT_VIEW, rv:-1 gSwitchId 0
Dec  3 03:47:29.372616 sonic NOTICE swss#orchagent: :- uninitialize: begin
Dec  3 03:47:29.372709 sonic NOTICE swss#orchagent: :- uninitialize: begin
Dec  3 03:47:29.372751 sonic NOTICE swss#orchagent: :- uninitialize: end server thread begin
Dec  3 03:47:29.372827 sonic NOTICE swss#orchagent: :- serverThreadFunction: end
Dec  3 03:47:29.372918 sonic NOTICE swss#orchagent: :- uninitialize: end server thread end
Dec  3 03:47:29.372991 sonic NOTICE swss#orchagent: :- uninitialize: begin
Dec  3 03:47:29.373116 sonic NOTICE swss#orchagent: :- uninitialize: begin
Dec  3 03:47:29.373156 sonic NOTICE swss#orchagent: :- ~RedisChannel: join ntf thread begin
Dec  3 03:47:29.373285 sonic NOTICE swss#orchagent: :- ~RedisChannel: join ntf thread end
Dec  3 03:47:29.373561 sonic NOTICE swss#orchagent: :- clear_local_state: clearing local state
Dec  3 03:47:29.373647 sonic NOTICE swss#orchagent: :- meta_init_db: begin
Dec  3 03:47:29.373717 sonic NOTICE swss#orchagent: :- meta_init_db: end
Dec  3 03:47:29.373786 sonic NOTICE swss#orchagent: :- uninitialize: end
Dec  3 03:47:29.373855 sonic NOTICE swss#orchagent: :- stopRecording: stopped recording
Dec  3 03:47:29.374015 sonic NOTICE swss#orchagent: :- stopRecording: closed recording file: sairedis.rec
Dec  3 03:47:29.374094 sonic NOTICE swss#orchagent: :- uninitialize: end
Dec  3 03:47:29.374175 sonic NOTICE swss#orchagent: :- uninitialize: end
Dec  3 03:47:29.374175 sonic NOTICE swss#orchagent: :- zmqPollThread: ending pool thread, since run is false
Dec  3 03:47:29.374175 sonic NOTICE swss#orchagent: :- zmqPollThread: end
Dec  3 03:47:29.374207 sonic NOTICE swss#orchagent: :- ~ZeroMQSelectableChannel: ending zmq poll thread for channel ipc:///tmp/saiServer
Dec  3 03:47:29.374207 sonic NOTICE swss#orchagent: :- ~ZeroMQSelectableChannel: ended zmq poll thread for channel ipc:///tmp/saiServer
Dec  3 03:47:29.374236 sonic NOTICE swss#orchagent: :- uninitialize: end
Dec  3 03:48:51.214230 sonic NOTICE swss#orchagent: :- main: --- Starting Orchestration Agent ---
Dec  3 03:48:51.225586 sonic NOTICE swss#orchagent: :- initialize: acting as a sairedis SERVER
Dec  3 03:48:51.225586 sonic NOTICE swss#orchagent: :- loadFromFile: no context config specified, will load default context config
Dec  3 03:48:51.225586 sonic NOTICE swss#orchagent: :- insert: added switch: idx 0, hwinfo ''
Dec  3 03:48:51.225586 sonic NOTICE swss#orchagent: :- RedisChannel: creating notification thread
Dec  3 03:48:51.243272 sonic NOTICE swss#orchagent: :- clear_local_state: clearing local state
Dec  3 03:48:51.243272 sonic NOTICE swss#orchagent: :- initialize: init client/server sai: SAI_STATUS_SUCCESS
Dec  3 03:48:51.243272 sonic NOTICE swss#orchagent: :- loadFromFile: no server config specified, will load default
Dec  3 03:48:51.243272 sonic NOTICE swss#orchagent: :- ZeroMQSelectableChannel: binding on ipc:///tmp/saiServer
Dec  3 03:48:51.243272 sonic NOTICE swss#orchagent: :- initialize: starting server thread
Dec  3 03:48:51.243272 sonic NOTICE swss#orchagent: :- initialize: init client/server sai: SAI_STATUS_SUCCESS
Dec  3 03:48:51.243272 sonic NOTICE swss#orchagent: :- setRecordingFilename: setting recording filename : sairedis.rec
Dec  3 03:48:51.243272 sonic NOTICE swss#orchagent: :- stopRecording: stopped recording
Dec  3 03:48:51.243272 sonic NOTICE swss#orchagent: :- startRecording: started recording: sairedis.rec
Dec  3 03:48:51.243272 sonic NOTICE swss#orchagent: :- initSaiRedis: Enable redis pipeline
Dec  3 03:48:51.243272 sonic NOTICE swss#orchagent: :- setRedisExtensionAttribute: set response timeout to 480000 ms
Dec  3 03:48:51.243272 sonic NOTICE swss#orchagent: :- initSaiRedis: SAI REDIS response timeout set successfully to 480000
Dec  3 03:48:51.243272 sonic NOTICE swss#orchagent: :- notifySyncd: sending syncd: INIT_VIEW
Dec  3 03:48:51.243272 sonic NOTICE swss#orchagent: :- serverThreadFunction: begin
Dec  3 03:48:51.243272 sonic NOTICE swss#orchagent: :- zmqPollThread: begin
Dec  3 03:56:51.331740 sonic ERR swss#orchagent: :- wait: SELECT operation result: TIMEOUT on notify
Dec  3 03:56:51.331999 sonic ERR swss#orchagent: :- wait: failed to get response for notify
Dec  3 03:56:51.332132 sonic ERR swss#orchagent: :- initSaiRedis: Failed to notify syncd INIT_VIEW, rv:-1 gSwitchId 0
Dec  3 03:56:51.332916 sonic NOTICE swss#orchagent: :- uninitialize: begin
Dec  3 03:56:51.333113 sonic NOTICE swss#orchagent: :- uninitialize: begin
Dec  3 03:56:51.333188 sonic NOTICE swss#orchagent: :- uninitialize: end server thread begin
Dec  3 03:56:51.333329 sonic NOTICE swss#orchagent: :- serverThreadFunction: end
Dec  3 03:56:51.333492 sonic NOTICE swss#orchagent: :- uninitialize: end server thread end
Dec  3 03:56:51.333620 sonic NOTICE swss#orchagent: :- uninitialize: begin
Dec  3 03:56:51.333739 sonic NOTICE swss#orchagent: :- uninitialize: begin
Dec  3 03:56:51.333804 sonic NOTICE swss#orchagent: :- ~RedisChannel: join ntf thread begin
Dec  3 03:56:51.333930 sonic NOTICE swss#orchagent: :- ~RedisChannel: join ntf thread end
Dec  3 03:56:51.334079 sonic NOTICE swss#orchagent: :- clear_local_state: clearing local state
Dec  3 03:56:51.334603 sonic NOTICE swss#orchagent: :- meta_init_db: begin
Dec  3 03:56:51.334748 sonic NOTICE swss#orchagent: :- meta_init_db: end
Dec  3 03:56:51.334867 sonic NOTICE swss#orchagent: :- uninitialize: end
Dec  3 03:56:51.335025 sonic NOTICE swss#orchagent: :- stopRecording: stopped recording
Dec  3 03:56:51.335153 sonic NOTICE swss#orchagent: :- stopRecording: closed recording file: sairedis.rec
Dec  3 03:56:51.335336 sonic NOTICE swss#orchagent: :- uninitialize: end
Dec  3 03:56:51.335420 sonic NOTICE swss#orchagent: :- uninitialize: end
Dec  3 03:56:51.335483 sonic NOTICE swss#orchagent: :- zmqPollThread: ending pool thread, since run is false
Dec  3 03:56:51.335555 sonic NOTICE swss#orchagent: :- zmqPollThread: end
Dec  3 03:56:51.335662 sonic NOTICE swss#orchagent: :- ~ZeroMQSelectableChannel: ending zmq poll thread for channel ipc:///tmp/saiServer
Dec  3 03:56:51.335662 sonic NOTICE swss#orchagent: :- ~ZeroMQSelectableChannel: ended zmq poll thread for channel ipc:///tmp/saiServer
Dec  3 03:56:51.335662 sonic NOTICE swss#orchagent: :- uninitialize: end
Dec  3 03:58:14.843331 sonic NOTICE swss#orchagent: :- main: --- Starting Orchestration Agent ---
Dec  3 03:58:14.843331 sonic NOTICE swss#orchagent: :- initialize: acting as a sairedis SERVER
Dec  3 03:58:14.843331 sonic NOTICE swss#orchagent: :- loadFromFile: no context config specified, will load default context config
Dec  3 03:58:14.843331 sonic NOTICE swss#orchagent: :- insert: added switch: idx 0, hwinfo ''
Dec  3 03:58:14.843331 sonic NOTICE swss#orchagent: :- RedisChannel: creating notification thread
Dec  3 03:58:14.843331 sonic NOTICE swss#orchagent: :- clear_local_state: clearing local state
Dec  3 03:58:14.843331 sonic NOTICE swss#orchagent: :- initialize: init client/server sai: SAI_STATUS_SUCCESS
Dec  3 03:58:14.843331 sonic NOTICE swss#orchagent: :- loadFromFile: no server config specified, will load default
Dec  3 03:58:14.843331 sonic NOTICE swss#orchagent: :- ZeroMQSelectableChannel: binding on ipc:///tmp/saiServer
Dec  3 03:58:14.849645 sonic NOTICE swss#orchagent: :- initialize: starting server thread
Dec  3 03:58:14.849645 sonic NOTICE swss#orchagent: :- initialize: init client/server sai: SAI_STATUS_SUCCESS
Dec  3 03:58:14.849645 sonic NOTICE swss#orchagent: :- setRecordingFilename: setting recording filename : sairedis.rec
Dec  3 03:58:14.849645 sonic NOTICE swss#orchagent: :- stopRecording: stopped recording
Dec  3 03:58:14.859723 sonic NOTICE swss#orchagent: :- startRecording: started recording: sairedis.rec
Dec  3 03:58:14.859723 sonic NOTICE swss#orchagent: :- initSaiRedis: Enable redis pipeline
Dec  3 03:58:14.859723 sonic NOTICE swss#orchagent: :- setRedisExtensionAttribute: set response timeout to 480000 ms
Dec  3 03:58:14.859723 sonic NOTICE swss#orchagent: :- initSaiRedis: SAI REDIS response timeout set successfully to 480000
Dec  3 03:58:14.859723 sonic NOTICE swss#orchagent: :- notifySyncd: sending syncd: INIT_VIEW
Dec  3 03:58:14.859723 sonic NOTICE swss#orchagent: :- zmqPollThread: begin
Dec  3 03:58:14.865338 sonic NOTICE swss#orchagent: :- serverThreadFunction: begin
Dec  3 04:06:14.879792 sonic ERR swss#orchagent: :- wait: SELECT operation result: TIMEOUT on notify
Dec  3 04:06:14.879792 sonic ERR swss#orchagent: :- wait: failed to get response for notify
Dec  3 04:06:14.879792 sonic ERR swss#orchagent: :- initSaiRedis: Failed to notify syncd INIT_VIEW, rv:-1 gSwitchId 0
Dec  3 04:06:14.880975 sonic NOTICE swss#orchagent: :- uninitialize: begin
Dec  3 04:06:14.881351 sonic NOTICE swss#orchagent: :- uninitialize: begin
Dec  3 04:06:14.881437 sonic NOTICE swss#orchagent: :- uninitialize: end server thread begin
Dec  3 04:06:14.881591 sonic NOTICE swss#orchagent: :- serverThreadFunction: end
Dec  3 04:06:14.881755 sonic NOTICE swss#orchagent: :- uninitialize: end server thread end
Dec  3 04:06:14.881905 sonic NOTICE swss#orchagent: :- uninitialize: begin
Dec  3 04:06:14.882213 sonic NOTICE swss#orchagent: :- uninitialize: begin
Dec  3 04:06:14.882290 sonic NOTICE swss#orchagent: :- ~RedisChannel: join ntf thread begin
Dec  3 04:06:14.882472 sonic NOTICE swss#orchagent: :- ~RedisChannel: join ntf thread end
Dec  3 04:06:14.882602 sonic NOTICE swss#orchagent: :- clear_local_state: clearing local state
Dec  3 04:06:14.882726 sonic NOTICE swss#orchagent: :- meta_init_db: begin
Dec  3 04:06:14.882845 sonic NOTICE swss#orchagent: :- meta_init_db: end
Dec  3 04:06:14.882963 sonic NOTICE swss#orchagent: :- uninitialize: end
Dec  3 04:06:14.883084 sonic NOTICE swss#orchagent: :- stopRecording: stopped recording
Dec  3 04:06:14.883229 sonic NOTICE swss#orchagent: :- stopRecording: closed recording file: sairedis.rec
Dec  3 04:06:14.883390 sonic NOTICE swss#orchagent: :- uninitialize: end
Dec  3 04:06:14.883470 sonic NOTICE swss#orchagent: :- uninitialize: end
Dec  3 04:06:14.883537 sonic NOTICE swss#orchagent: :- zmqPollThread: ending pool thread, since run is false
Dec  3 04:06:14.883606 sonic NOTICE swss#orchagent: :- zmqPollThread: end
Dec  3 04:06:14.883745 sonic NOTICE swss#orchagent: :- ~ZeroMQSelectableChannel: ending zmq poll thread for channel ipc:///tmp/saiServer
Dec  3 04:06:14.883745 sonic NOTICE swss#orchagent: :- ~ZeroMQSelectableChannel: ended zmq poll thread for channel ipc:///tmp/saiServer
Dec  3 04:06:14.883745 sonic NOTICE swss#orchagent: :- uninitialize: end
Dec  3 04:07:34.747322 sonic NOTICE swss#orchagent: :- main: --- Starting Orchestration Agent ---
Dec  3 04:07:34.747322 sonic NOTICE swss#orchagent: :- initialize: acting as a sairedis SERVER
Dec  3 04:07:34.747322 sonic NOTICE swss#orchagent: :- loadFromFile: no context config specified, will load default context config
Dec  3 04:07:34.747322 sonic NOTICE swss#orchagent: :- insert: added switch: idx 0, hwinfo ''
Dec  3 04:07:34.747322 sonic NOTICE swss#orchagent: :- RedisChannel: creating notification thread
Dec  3 04:07:34.755278 sonic NOTICE swss#orchagent: :- clear_local_state: clearing local state
Dec  3 04:07:34.755278 sonic NOTICE swss#orchagent: :- initialize: init client/server sai: SAI_STATUS_SUCCESS
Dec  3 04:07:34.755278 sonic NOTICE swss#orchagent: :- loadFromFile: no server config specified, will load default
Dec  3 04:07:34.755278 sonic NOTICE swss#orchagent: :- ZeroMQSelectableChannel: binding on ipc:///tmp/saiServer
Dec  3 04:07:34.755278 sonic NOTICE swss#orchagent: :- initialize: starting server thread
Dec  3 04:07:34.755278 sonic NOTICE swss#orchagent: :- initialize: init client/server sai: SAI_STATUS_SUCCESS
Dec  3 04:07:34.755278 sonic NOTICE swss#orchagent: :- setRecordingFilename: setting recording filename : sairedis.rec
Dec  3 04:07:34.755278 sonic NOTICE swss#orchagent: :- stopRecording: stopped recording
Dec  3 04:07:34.755278 sonic NOTICE swss#orchagent: :- startRecording: started recording: sairedis.rec
Dec  3 04:07:34.755278 sonic NOTICE swss#orchagent: :- initSaiRedis: Enable redis pipeline
Dec  3 04:07:34.755278 sonic NOTICE swss#orchagent: :- setRedisExtensionAttribute: set response timeout to 480000 ms
Dec  3 04:07:34.755278 sonic NOTICE swss#orchagent: :- initSaiRedis: SAI REDIS response timeout set successfully to 480000
Dec  3 04:07:34.755278 sonic NOTICE swss#orchagent: :- notifySyncd: sending syncd: INIT_VIEW
Dec  3 04:07:34.755278 sonic NOTICE swss#orchagent: :- zmqPollThread: begin
Dec  3 04:07:34.771281 sonic NOTICE swss#orchagent: :- serverThreadFunction: begin
Dec  3 04:15:34.856038 sonic ERR swss#orchagent: :- wait: SELECT operation result: TIMEOUT on notify
Dec  3 04:15:34.856117 sonic ERR swss#orchagent: :- wait: failed to get response for notify
Dec  3 04:15:34.856445 sonic ERR swss#orchagent: :- initSaiRedis: Failed to notify syncd INIT_VIEW, rv:-1 gSwitchId 0
Dec  3 04:15:34.856781 sonic NOTICE swss#orchagent: :- uninitialize: begin
Dec  3 04:15:34.856941 sonic NOTICE swss#orchagent: :- uninitialize: begin
Dec  3 04:15:34.857016 sonic NOTICE swss#orchagent: :- uninitialize: end server thread begin
Dec  3 04:15:34.857149 sonic NOTICE swss#orchagent: :- serverThreadFunction: end
Dec  3 04:15:34.857546 sonic NOTICE swss#orchagent: :- uninitialize: end server thread end
Dec  3 04:15:34.857687 sonic NOTICE swss#orchagent: :- uninitialize: begin
Dec  3 04:15:34.857809 sonic NOTICE swss#orchagent: :- uninitialize: begin
Dec  3 04:15:34.857877 sonic NOTICE swss#orchagent: :- ~RedisChannel: join ntf thread begin
Dec  3 04:15:34.858117 sonic NOTICE swss#orchagent: :- ~RedisChannel: join ntf thread end
Dec  3 04:15:34.858258 sonic NOTICE swss#orchagent: :- clear_local_state: clearing local state
Dec  3 04:15:34.858418 sonic NOTICE swss#orchagent: :- meta_init_db: begin
Dec  3 04:15:34.858544 sonic NOTICE swss#orchagent: :- meta_init_db: end
Dec  3 04:15:34.858664 sonic NOTICE swss#orchagent: :- uninitialize: end
Dec  3 04:15:34.858784 sonic NOTICE swss#orchagent: :- stopRecording: stopped recording
Dec  3 04:15:34.858904 sonic NOTICE swss#orchagent: :- stopRecording: closed recording file: sairedis.rec
Dec  3 04:15:34.859022 sonic NOTICE swss#orchagent: :- uninitialize: end
Dec  3 04:15:34.859152 sonic NOTICE swss#orchagent: :- uninitialize: end
Dec  3 04:15:34.859152 sonic NOTICE swss#orchagent: :- zmqPollThread: ending pool thread, since run is false
Dec  3 04:15:34.859152 sonic NOTICE swss#orchagent: :- zmqPollThread: end
Dec  3 04:15:34.859259 sonic NOTICE swss#orchagent: :- ~ZeroMQSelectableChannel: ending zmq poll thread for channel ipc:///tmp/saiServer
Dec  3 04:15:34.859259 sonic NOTICE swss#orchagent: :- ~ZeroMQSelectableChannel: ended zmq poll thread for channel ipc:///tmp/saiServer
Dec  3 04:15:34.859283 sonic NOTICE swss#orchagent: :- uninitialize: end
Dec  3 04:16:56.167275 sonic NOTICE swss#orchagent: :- main: --- Starting Orchestration Agent ---
Dec  3 04:16:56.167275 sonic NOTICE swss#orchagent: :- initialize: acting as a sairedis SERVER
Dec  3 04:16:56.167275 sonic NOTICE swss#orchagent: :- loadFromFile: no context config specified, will load default context config
Dec  3 04:16:56.167275 sonic NOTICE swss#orchagent: :- insert: added switch: idx 0, hwinfo ''
Dec  3 04:16:56.167275 sonic NOTICE swss#orchagent: :- RedisChannel: creating notification thread
Dec  3 04:16:56.167275 sonic NOTICE swss#orchagent: :- clear_local_state: clearing local state
Dec  3 04:16:56.167275 sonic NOTICE swss#orchagent: :- initialize: init client/server sai: SAI_STATUS_SUCCESS
Dec  3 04:16:56.167275 sonic NOTICE swss#orchagent: :- loadFromFile: no server config specified, will load default
Dec  3 04:16:56.167275 sonic NOTICE swss#orchagent: :- ZeroMQSelectableChannel: binding on ipc:///tmp/saiServer
Dec  3 04:16:56.187252 sonic NOTICE swss#orchagent: :- initialize: starting server thread
Dec  3 04:16:56.187252 sonic NOTICE swss#orchagent: :- initialize: init client/server sai: SAI_STATUS_SUCCESS
Dec  3 04:16:56.187252 sonic NOTICE swss#orchagent: :- setRecordingFilename: setting recording filename : sairedis.rec
Dec  3 04:16:56.187252 sonic NOTICE swss#orchagent: :- stopRecording: stopped recording
Dec  3 04:16:56.187252 sonic NOTICE swss#orchagent: :- startRecording: started recording: sairedis.rec
Dec  3 04:16:56.187252 sonic NOTICE swss#orchagent: :- initSaiRedis: Enable redis pipeline
Dec  3 04:16:56.187252 sonic NOTICE swss#orchagent: :- setRedisExtensionAttribute: set response timeout to 480000 ms
Dec  3 04:16:56.187252 sonic NOTICE swss#orchagent: :- initSaiRedis: SAI REDIS response timeout set successfully to 480000
Dec  3 04:16:56.187252 sonic NOTICE swss#orchagent: :- notifySyncd: sending syncd: INIT_VIEW
Dec  3 04:16:56.187252 sonic NOTICE swss#orchagent: :- serverThreadFunction: begin
Dec  3 04:16:56.187252 sonic NOTICE swss#orchagent: :- zmqPollThread: begin
Dec  3 04:24:56.284970 sonic ERR swss#orchagent: :- wait: SELECT operation result: TIMEOUT on notify
Dec  3 04:24:56.284970 sonic ERR swss#orchagent: :- wait: failed to get response for notify
Dec  3 04:24:56.284970 sonic ERR swss#orchagent: :- initSaiRedis: Failed to notify syncd INIT_VIEW, rv:-1 gSwitchId 0
Dec  3 04:24:56.284970 sonic NOTICE swss#orchagent: :- uninitialize: begin
Dec  3 04:24:56.284970 sonic NOTICE swss#orchagent: :- uninitialize: begin
Dec  3 04:24:56.284970 sonic NOTICE swss#orchagent: :- uninitialize: end server thread begin
Dec  3 04:24:56.284970 sonic NOTICE swss#orchagent: :- serverThreadFunction: end
Dec  3 04:24:56.284970 sonic NOTICE swss#orchagent: :- uninitialize: end server thread end
Dec  3 04:24:56.284970 sonic NOTICE swss#orchagent: :- uninitialize: begin
Dec  3 04:24:56.284970 sonic NOTICE swss#orchagent: :- uninitialize: begin
Dec  3 04:24:56.284970 sonic NOTICE swss#orchagent: :- ~RedisChannel: join ntf thread begin
Dec  3 04:24:56.284970 sonic NOTICE swss#orchagent: :- ~RedisChannel: join ntf thread end
Dec  3 04:24:56.285049 sonic NOTICE swss#orchagent: :- clear_local_state: clearing local state
Dec  3 04:24:56.285049 sonic NOTICE swss#orchagent: :- meta_init_db: begin
Dec  3 04:24:56.285049 sonic NOTICE swss#orchagent: :- meta_init_db: end
Dec  3 04:24:56.285269 sonic NOTICE swss#orchagent: :- uninitialize: end
Dec  3 04:24:56.285269 sonic NOTICE swss#orchagent: :- stopRecording: stopped recording
Dec  3 04:24:56.285269 sonic NOTICE swss#orchagent: :- stopRecording: closed recording file: sairedis.rec
Dec  3 04:24:56.285269 sonic NOTICE swss#orchagent: :- uninitialize: end
Dec  3 04:24:56.285269 sonic NOTICE swss#orchagent: :- uninitialize: end
Dec  3 04:24:56.285269 sonic NOTICE swss#orchagent: :- zmqPollThread: ending pool thread, since run is false
Dec  3 04:24:56.285269 sonic NOTICE swss#orchagent: :- zmqPollThread: end
Dec  3 04:24:56.285770 sonic NOTICE swss#orchagent: :- ~ZeroMQSelectableChannel: ending zmq poll thread for channel ipc:///tmp/saiServer
Dec  3 04:24:56.285770 sonic NOTICE swss#orchagent: :- ~ZeroMQSelectableChannel: ended zmq poll thread for channel ipc:///tmp/saiServer
Dec  3 04:24:56.285770 sonic NOTICE swss#orchagent: :- uninitialize: end
Dec  3 04:26:16.127254 sonic NOTICE swss#orchagent: :- main: --- Starting Orchestration Agent ---
Dec  3 04:26:16.127254 sonic NOTICE swss#orchagent: :- initialize: acting as a sairedis SERVER
Dec  3 04:26:16.127254 sonic NOTICE swss#orchagent: :- loadFromFile: no context config specified, will load default context config
Dec  3 04:26:16.127254 sonic NOTICE swss#orchagent: :- insert: added switch: idx 0, hwinfo ''
Dec  3 04:26:16.127254 sonic NOTICE swss#orchagent: :- RedisChannel: creating notification thread
Dec  3 04:26:16.143256 sonic NOTICE swss#orchagent: :- clear_local_state: clearing local state
Dec  3 04:26:16.143256 sonic NOTICE swss#orchagent: :- initialize: init client/server sai: SAI_STATUS_SUCCESS
Dec  3 04:26:16.143256 sonic NOTICE swss#orchagent: :- loadFromFile: no server config specified, will load default
Dec  3 04:26:16.143256 sonic NOTICE swss#orchagent: :- ZeroMQSelectableChannel: binding on ipc:///tmp/saiServer
Dec  3 04:26:16.143256 sonic NOTICE swss#orchagent: :- initialize: starting server thread
Dec  3 04:26:16.143256 sonic NOTICE swss#orchagent: :- initialize: init client/server sai: SAI_STATUS_SUCCESS
Dec  3 04:26:16.143256 sonic NOTICE swss#orchagent: :- setRecordingFilename: setting recording filename : sairedis.rec
Dec  3 04:26:16.143256 sonic NOTICE swss#orchagent: :- stopRecording: stopped recording
Dec  3 04:26:16.143256 sonic NOTICE swss#orchagent: :- startRecording: started recording: sairedis.rec
Dec  3 04:26:16.143256 sonic NOTICE swss#orchagent: :- initSaiRedis: Enable redis pipeline
Dec  3 04:26:16.143256 sonic NOTICE swss#orchagent: :- setRedisExtensionAttribute: set response timeout to 480000 ms
Dec  3 04:26:16.143256 sonic NOTICE swss#orchagent: :- initSaiRedis: SAI REDIS response timeout set successfully to 480000
Dec  3 04:26:16.143256 sonic NOTICE swss#orchagent: :- notifySyncd: sending syncd: INIT_VIEW
Dec  3 04:26:16.143256 sonic NOTICE swss#orchagent: :- serverThreadFunction: begin
Dec  3 04:26:16.143256 sonic NOTICE swss#orchagent: :- zmqPollThread: begin


show logging | grep -i "syncd#syncd"

Dec  3 03:19:56.630078 sonic DEBUG syncd#syncd: :> syncd_main: enter
Dec  3 03:19:56.650575 sonic NOTICE syncd#syncd: :- initialize: initializeing metadata log function
Dec  3 03:19:56.658764 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SWITCH
Dec  3 03:19:56.658764 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_PORT
Dec  3 03:19:56.658764 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_FDB
Dec  3 03:19:56.671665 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_VLAN
Dec  3 03:19:56.671665 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_VIRTUAL_ROUTER
Dec  3 03:19:56.671665 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ROUTE
Dec  3 03:19:56.671665 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_NEXT_HOP
Dec  3 03:19:56.671665 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_NEXT_HOP_GROUP
Dec  3 03:19:56.671665 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ROUTER_INTERFACE
Dec  3 03:19:56.695651 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_NEIGHBOR
Dec  3 03:19:56.695651 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ACL
Dec  3 03:19:56.695651 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_HOSTIF
Dec  3 03:19:56.695651 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_MIRROR
Dec  3 03:19:56.695651 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SAMPLEPACKET
Dec  3 03:19:56.695651 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_STP
Dec  3 03:19:56.695651 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_LAG
Dec  3 03:19:56.703720 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_POLICER
Dec  3 03:19:56.715648 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_WRED
Dec  3 03:19:56.715648 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_QOS_MAP
Dec  3 03:19:56.715648 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_QUEUE
Dec  3 03:19:56.727689 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SCHEDULER
Dec  3 03:19:56.729444 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SCHEDULER_GROUP
Dec  3 03:19:56.733426 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_BUFFER
Dec  3 03:19:56.759530 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_HASH
Dec  3 03:19:56.759530 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_UDF
Dec  3 03:19:56.761955 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_TUNNEL
Dec  3 03:19:56.767602 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[494]- sai_log_set: mlnx_sai_interfacequery.c[494]- sai_log_set: SAI_API_L2MC not implemented
Dec  3 03:19:56.773542 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[494]- sai_log_set: mlnx_sai_interfacequery.c[494]- sai_log_set: SAI_API_IPMC not implemented
Dec  3 03:19:56.783639 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[494]- sai_log_set: mlnx_sai_interfacequery.c[494]- sai_log_set: SAI_API_RPF_GROUP not implemented
Dec  3 03:19:56.783639 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_L2MC_GROUP
Dec  3 03:19:56.791678 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[494]- sai_log_set: mlnx_sai_interfacequery.c[494]- sai_log_set: SAI_API_IPMC_GROUP not implemented
Dec  3 03:19:56.791678 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[494]- sai_log_set: mlnx_sai_interfacequery.c[494]- sai_log_set: SAI_API_MCAST_FDB not implemented
Dec  3 03:19:56.791678 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_BRIDGE
Dec  3 03:19:56.802417 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[494]- sai_log_set: mlnx_sai_interfacequery.c[494]- sai_log_set: SAI_API_TAM not implemented
Dec  3 03:19:56.802417 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[494]- sai_log_set: mlnx_sai_interfacequery.c[494]- sai_log_set: SAI_API_SEGMENTROUTE not implemented
Dec  3 03:19:56.808333 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[494]- sai_log_set: mlnx_sai_interfacequery.c[494]- sai_log_set: SAI_API_MPLS not implemented
Dec  3 03:19:56.808333 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[494]- sai_log_set: mlnx_sai_interfacequery.c[494]- sai_log_set: SAI_API_DTEL not implemented
Dec  3 03:19:56.814374 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_BFD
Dec  3 03:19:56.827297 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ISOLATION_GROUP
Dec  3 03:19:56.839610 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[494]- sai_log_set: mlnx_sai_interfacequery.c[494]- sai_log_set: SAI_API_NAT not implemented
Dec  3 03:19:56.839610 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_COUNTER
Dec  3 03:19:56.839610 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_DEBUG_COUNTER
Dec  3 03:19:56.851689 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[494]- sai_log_set: mlnx_sai_interfacequery.c[494]- sai_log_set: SAI_API_MACSEC not implemented
Dec  3 03:19:56.851689 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[494]- sai_log_set: mlnx_sai_interfacequery.c[494]- sai_log_set: SAI_API_SYSTEM_PORT not implemented
Dec  3 03:19:56.851689 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[494]- sai_log_set: mlnx_sai_interfacequery.c[494]- sai_log_set: SAI_API_BMTOR not implemented
Dec  3 03:19:56.851689 sonic NOTICE syncd#syncd: :- Syncd: command line:  EnableDiagShell=NO EnableTempView=YES DisableExitSleep=NO EnableUnittests=NO EnableConsistencyCheck=NO EnableSyncMode=NO RedisCommunicationMode=zmq_sync EnableSaiBulkSuport=NO StartType=cold ProfileMapFile=/tmp/sai.profile GlobalContext=0 ContextConfig= BreakConfig=
Dec  3 03:19:56.851689 sonic NOTICE syncd#syncd: :- loadFromFile: no context config specified, will load default context config
Dec  3 03:19:56.851689 sonic NOTICE syncd#syncd: :- insert: added switch: idx 0, hwinfo ''
Dec  3 03:19:56.851689 sonic NOTICE syncd#syncd: :- Syncd: zmq sync mode enabled via cmd line
Dec  3 03:19:56.851689 sonic NOTICE syncd#syncd: :- ZeroMQNotificationProducer: opening zmq ntf endpoint: ipc:///tmp/zmq_ntf_ep
Dec  3 03:19:56.851689 sonic NOTICE syncd#syncd: :- Syncd: zmq enabled, forcing sync mode
Dec  3 03:19:56.859597 sonic NOTICE syncd#syncd: :- zmqPollThread: begin
Dec  3 03:19:56.895666 sonic NOTICE syncd#syncd: :- isVeryFirstRun: First Run: True
Dec  3 03:29:15.923735 sonic DEBUG syncd#syncd: :> syncd_main: enter
Dec  3 03:29:15.923735 sonic NOTICE syncd#syncd: :- initialize: initializeing metadata log function
Dec  3 03:29:15.923735 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SWITCH
Dec  3 03:29:15.923735 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_PORT
Dec  3 03:29:15.923735 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_FDB
Dec  3 03:29:15.929090 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_VLAN
Dec  3 03:29:15.929090 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_VIRTUAL_ROUTER
Dec  3 03:29:15.929090 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ROUTE
Dec  3 03:29:15.929090 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_NEXT_HOP
Dec  3 03:29:15.936535 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_NEXT_HOP_GROUP
Dec  3 03:29:15.936535 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ROUTER_INTERFACE
Dec  3 03:29:15.936535 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_NEIGHBOR
Dec  3 03:29:15.936535 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ACL
Dec  3 03:29:15.942813 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_HOSTIF
Dec  3 03:29:15.942813 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_MIRROR
Dec  3 03:29:15.942813 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SAMPLEPACKET
Dec  3 03:29:15.942813 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_STP
Dec  3 03:29:15.942813 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_LAG
Dec  3 03:29:15.942813 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_POLICER
Dec  3 03:29:15.947600 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_WRED
Dec  3 03:29:15.947600 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_QOS_MAP
Dec  3 03:29:15.955604 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_QUEUE
Dec  3 03:29:15.955604 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SCHEDULER
Dec  3 03:29:15.955604 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SCHEDULER_GROUP
Dec  3 03:29:15.955604 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_BUFFER
Dec  3 03:29:15.955604 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_HASH
Dec  3 03:29:15.955604 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_UDF
Dec  3 03:29:15.965861 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_TUNNEL
Dec  3 03:29:15.965861 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[494]- sai_log_set: mlnx_sai_interfacequery.c[494]- sai_log_set: SAI_API_L2MC not implemented
Dec  3 03:29:15.965861 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[494]- sai_log_set: mlnx_sai_interfacequery.c[494]- sai_log_set: SAI_API_IPMC not implemented
Dec  3 03:29:15.965861 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[494]- sai_log_set: mlnx_sai_interfacequery.c[494]- sai_log_set: SAI_API_RPF_GROUP not implemented
Dec  3 03:29:15.965861 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_L2MC_GROUP
Dec  3 03:29:15.965861 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[494]- sai_log_set: mlnx_sai_interfacequery.c[494]- sai_log_set: SAI_API_IPMC_GROUP not implemented
Dec  3 03:29:15.965861 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[494]- sai_log_set: mlnx_sai_interfacequery.c[494]- sai_log_set: SAI_API_MCAST_FDB not implemented
Dec  3 03:29:15.965861 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_BRIDGE
Dec  3 03:29:15.965861 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[494]- sai_log_set: mlnx_sai_interfacequery.c[494]- sai_log_set: SAI_API_TAM not implemented
Dec  3 03:29:15.977518 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[494]- sai_log_set: mlnx_sai_interfacequery.c[494]- sai_log_set: SAI_API_SEGMENTROUTE not implemented
Dec  3 03:29:15.977518 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[494]- sai_log_set: mlnx_sai_interfacequery.c[494]- sai_log_set: SAI_API_MPLS not implemented
Dec  3 03:29:15.977518 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[494]- sai_log_set: mlnx_sai_interfacequery.c[494]- sai_log_set: SAI_API_DTEL not implemented
Dec  3 03:29:15.977518 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_BFD
Dec  3 03:29:15.977518 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ISOLATION_GROUP
Dec  3 03:29:15.977518 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[494]- sai_log_set: mlnx_sai_interfacequery.c[494]- sai_log_set: SAI_API_NAT not implemented
Dec  3 03:29:15.977518 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_COUNTER
Dec  3 03:29:15.977518 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_DEBUG_COUNTER
Dec  3 03:29:15.981573 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[494]- sai_log_set: mlnx_sai_interfacequery.c[494]- sai_log_set: SAI_API_MACSEC not implemented
Dec  3 03:29:15.981573 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[494]- sai_log_set: mlnx_sai_interfacequery.c[494]- sai_log_set: SAI_API_SYSTEM_PORT not implemented
Dec  3 03:29:15.981573 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[494]- sai_log_set: mlnx_sai_interfacequery.c[494]- sai_log_set: SAI_API_BMTOR not implemented
Dec  3 03:29:15.991669 sonic NOTICE syncd#syncd: :- Syncd: command line:  EnableDiagShell=NO EnableTempView=YES DisableExitSleep=NO EnableUnittests=NO EnableConsistencyCheck=NO EnableSyncMode=NO RedisCommunicationMode=zmq_sync EnableSaiBulkSuport=NO StartType=cold ProfileMapFile=/tmp/sai.profile GlobalContext=0 ContextConfig= BreakConfig=
Dec  3 03:29:15.991669 sonic NOTICE syncd#syncd: :- loadFromFile: no context config specified, will load default context config
Dec  3 03:29:15.991669 sonic NOTICE syncd#syncd: :- insert: added switch: idx 0, hwinfo ''
Dec  3 03:29:15.991669 sonic NOTICE syncd#syncd: :- Syncd: zmq sync mode enabled via cmd line
Dec  3 03:29:15.991669 sonic NOTICE syncd#syncd: :- ZeroMQNotificationProducer: opening zmq ntf endpoint: ipc:///tmp/zmq_ntf_ep
Dec  3 03:29:15.991669 sonic NOTICE syncd#syncd: :- Syncd: zmq enabled, forcing sync mode
Dec  3 03:29:15.999600 sonic NOTICE syncd#syncd: :- zmqPollThread: begin
Dec  3 03:29:15.999600 sonic NOTICE syncd#syncd: :- isVeryFirstRun: First Run: True
Dec  3 03:39:35.270908 sonic DEBUG syncd#syncd: :> syncd_main: enter
Dec  3 03:39:35.280080 sonic NOTICE syncd#syncd: :- initialize: initializeing metadata log function
Dec  3 03:39:35.295406 sonic NOTICE syncd#syncd: :- threadFunction: starting timer watchdog thread
Dec  3 03:39:35.303407 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SWITCH
Dec  3 03:39:35.307099 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_PORT
Dec  3 03:39:35.323311 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_FDB
Dec  3 03:39:35.323311 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_VLAN
Dec  3 03:39:35.323311 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_VIRTUAL_ROUTER
Dec  3 03:39:35.323311 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ROUTE
Dec  3 03:39:35.323311 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_NEXT_HOP
Dec  3 03:39:35.323311 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_NEXT_HOP_GROUP
Dec  3 03:39:35.323311 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ROUTER_INTERFACE
Dec  3 03:39:35.326195 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_NEIGHBOR
Dec  3 03:39:35.326195 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ACL
Dec  3 03:39:35.326195 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_HOSTIF
Dec  3 03:39:35.337568 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_MIRROR
Dec  3 03:39:35.337568 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SAMPLEPACKET
Dec  3 03:39:35.337568 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_STP
Dec  3 03:39:35.341502 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_LAG
Dec  3 03:39:35.341502 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_POLICER
Dec  3 03:39:35.354648 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_WRED
Dec  3 03:39:35.354648 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_QOS_MAP
Dec  3 03:39:35.354648 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_QUEUE
Dec  3 03:39:35.354648 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SCHEDULER
Dec  3 03:39:35.363309 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SCHEDULER_GROUP
Dec  3 03:39:35.371307 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_BUFFER
Dec  3 03:39:35.371307 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_HASH
Dec  3 03:39:35.371307 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_UDF
Dec  3 03:39:35.371307 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_TUNNEL
Dec  3 03:39:35.387310 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_L2MC not implemented
Dec  3 03:39:35.387310 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_IPMC not implemented
Dec  3 03:39:35.387310 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_RPF_GROUP not implemented
Dec  3 03:39:35.406395 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_L2MC_GROUP
Dec  3 03:39:35.406395 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_IPMC_GROUP not implemented
Dec  3 03:39:35.406395 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_MCAST_FDB not implemented
Dec  3 03:39:35.406395 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_BRIDGE
Dec  3 03:39:35.406395 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_TAM not implemented
Dec  3 03:39:35.406395 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_SRV6 not implemented
Dec  3 03:39:35.406395 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_MPLS not implemented
Dec  3 03:39:35.406395 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_DTEL not implemented
Dec  3 03:39:35.406395 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_BFD
Dec  3 03:39:35.406395 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ISOLATION_GROUP
Dec  3 03:39:35.406395 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_NAT not implemented
Dec  3 03:39:35.406395 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_COUNTER
Dec  3 03:39:35.417732 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_DEBUG_COUNTER
Dec  3 03:39:35.417732 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_MACSEC not implemented
Dec  3 03:39:35.417732 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_SYSTEM_PORT not implemented
Dec  3 03:39:35.417732 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_MY_MAC not implemented
Dec  3 03:39:35.431593 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_IPSEC not implemented
Dec  3 03:39:35.431593 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_BMTOR not implemented
Dec  3 03:39:35.431593 sonic NOTICE syncd#syncd: :- Syncd: command line:  EnableDiagShell=NO EnableTempView=YES DisableExitSleep=NO EnableUnittests=NO EnableConsistencyCheck=NO EnableSyncMode=NO RedisCommunicationMode=zmq_sync EnableSaiBulkSuport=NO StartType=cold ProfileMapFile=/tmp/sai.profile GlobalContext=0 ContextConfig= BreakConfig=
Dec  3 03:39:35.431593 sonic NOTICE syncd#syncd: :- loadFromFile: no context config specified, will load default context config
Dec  3 03:39:35.431593 sonic NOTICE syncd#syncd: :- insert: added switch: idx 0, hwinfo ''
Dec  3 03:39:35.431593 sonic NOTICE syncd#syncd: :- Syncd: zmq sync mode enabled via cmd line
Dec  3 03:39:35.431593 sonic NOTICE syncd#syncd: :- ZeroMQNotificationProducer: opening zmq ntf endpoint: ipc:///tmp/zmq_ntf_ep
Dec  3 03:39:35.431593 sonic NOTICE syncd#syncd: :- Syncd: zmq enabled, forcing sync mode
Dec  3 03:39:35.431593 sonic NOTICE syncd#syncd: :- ZeroMQSelectableChannel: binding on ipc:///tmp/zmq_ep
Dec  3 03:39:35.543499 sonic NOTICE syncd#syncd: :- zmqPollThread: begin
Dec  3 03:39:35.543499 sonic NOTICE syncd#syncd: :- isVeryFirstRun: First Run: True
Dec  3 03:48:57.351646 sonic DEBUG syncd#syncd: :> syncd_main: enter
Dec  3 03:48:57.365926 sonic NOTICE syncd#syncd: :- initialize: initializeing metadata log function
Dec  3 03:48:57.366641 sonic NOTICE syncd#syncd: :- threadFunction: starting timer watchdog thread
Dec  3 03:48:57.367562 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SWITCH
Dec  3 03:48:57.390169 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_PORT
Dec  3 03:48:57.390169 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_FDB
Dec  3 03:48:57.390169 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_VLAN
Dec  3 03:48:57.390169 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_VIRTUAL_ROUTER
Dec  3 03:48:57.390169 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ROUTE
Dec  3 03:48:57.390169 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_NEXT_HOP
Dec  3 03:48:57.390169 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_NEXT_HOP_GROUP
Dec  3 03:48:57.390169 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ROUTER_INTERFACE
Dec  3 03:48:57.402829 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_NEIGHBOR
Dec  3 03:48:57.402829 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ACL
Dec  3 03:48:57.402829 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_HOSTIF
Dec  3 03:48:57.402829 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_MIRROR
Dec  3 03:48:57.402829 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SAMPLEPACKET
Dec  3 03:48:57.402829 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_STP
Dec  3 03:48:57.402829 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_LAG
Dec  3 03:48:57.409656 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_POLICER
Dec  3 03:48:57.409656 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_WRED
Dec  3 03:48:57.409656 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_QOS_MAP
Dec  3 03:48:57.409656 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_QUEUE
Dec  3 03:48:57.409656 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SCHEDULER
Dec  3 03:48:57.409656 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SCHEDULER_GROUP
Dec  3 03:48:57.412370 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_BUFFER
Dec  3 03:48:57.412370 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_HASH
Dec  3 03:48:57.421747 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_UDF
Dec  3 03:48:57.421747 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_TUNNEL
Dec  3 03:48:57.421747 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_L2MC not implemented
Dec  3 03:48:57.421747 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_IPMC not implemented
Dec  3 03:48:57.421747 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_RPF_GROUP not implemented
Dec  3 03:48:57.421747 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_L2MC_GROUP
Dec  3 03:48:57.427052 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_IPMC_GROUP not implemented
Dec  3 03:48:57.427052 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_MCAST_FDB not implemented
Dec  3 03:48:57.427052 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_BRIDGE
Dec  3 03:48:57.427052 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_TAM not implemented
Dec  3 03:48:57.433369 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_SRV6 not implemented
Dec  3 03:48:57.433369 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_MPLS not implemented
Dec  3 03:48:57.433369 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_DTEL not implemented
Dec  3 03:48:57.433369 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_BFD
Dec  3 03:48:57.433369 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ISOLATION_GROUP
Dec  3 03:48:57.436269 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_NAT not implemented
Dec  3 03:48:57.436269 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_COUNTER
Dec  3 03:48:57.441210 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_DEBUG_COUNTER
Dec  3 03:48:57.441210 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_MACSEC not implemented
Dec  3 03:48:57.441210 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_SYSTEM_PORT not implemented
Dec  3 03:48:57.441210 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_MY_MAC not implemented
Dec  3 03:48:57.444290 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_IPSEC not implemented
Dec  3 03:48:57.444290 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_BMTOR not implemented
Dec  3 03:48:57.448728 sonic NOTICE syncd#syncd: :- Syncd: command line:  EnableDiagShell=NO EnableTempView=YES DisableExitSleep=NO EnableUnittests=NO EnableConsistencyCheck=NO EnableSyncMode=NO RedisCommunicationMode=zmq_sync EnableSaiBulkSuport=NO StartType=cold ProfileMapFile=/tmp/sai.profile GlobalContext=0 ContextConfig= BreakConfig=
Dec  3 03:48:57.448728 sonic NOTICE syncd#syncd: :- loadFromFile: no context config specified, will load default context config
Dec  3 03:48:57.448728 sonic NOTICE syncd#syncd: :- insert: added switch: idx 0, hwinfo ''
Dec  3 03:48:57.448728 sonic NOTICE syncd#syncd: :- Syncd: zmq sync mode enabled via cmd line
Dec  3 03:48:57.458579 sonic NOTICE syncd#syncd: :- ZeroMQNotificationProducer: opening zmq ntf endpoint: ipc:///tmp/zmq_ntf_ep
Dec  3 03:48:57.458579 sonic NOTICE syncd#syncd: :- Syncd: zmq enabled, forcing sync mode
Dec  3 03:48:57.458579 sonic NOTICE syncd#syncd: :- ZeroMQSelectableChannel: binding on ipc:///tmp/zmq_ep
Dec  3 03:48:57.467252 sonic NOTICE syncd#syncd: :- zmqPollThread: begin
Dec  3 03:48:57.467252 sonic NOTICE syncd#syncd: :- isVeryFirstRun: First Run: True
Dec  3 03:58:19.410428 sonic DEBUG syncd#syncd: :> syncd_main: enter
Dec  3 03:58:19.410428 sonic NOTICE syncd#syncd: :- initialize: initializeing metadata log function
Dec  3 03:58:19.425369 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SWITCH
Dec  3 03:58:19.425369 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_PORT
Dec  3 03:58:19.425369 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_FDB
Dec  3 03:58:19.425369 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_VLAN
Dec  3 03:58:19.425369 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_VIRTUAL_ROUTER
Dec  3 03:58:19.425369 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ROUTE
Dec  3 03:58:19.425369 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_NEXT_HOP
Dec  3 03:58:19.425369 sonic NOTICE syncd#syncd: :- threadFunction: starting timer watchdog thread
Dec  3 03:58:19.425369 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_NEXT_HOP_GROUP
Dec  3 03:58:19.425369 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ROUTER_INTERFACE
Dec  3 03:58:19.425369 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_NEIGHBOR
Dec  3 03:58:19.425369 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ACL
Dec  3 03:58:19.425369 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_HOSTIF
Dec  3 03:58:19.445395 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_MIRROR
Dec  3 03:58:19.445395 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SAMPLEPACKET
Dec  3 03:58:19.449502 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_STP
Dec  3 03:58:19.449502 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_LAG
Dec  3 03:58:19.462109 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_POLICER
Dec  3 03:58:19.462109 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_WRED
Dec  3 03:58:19.462109 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_QOS_MAP
Dec  3 03:58:19.462109 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_QUEUE
Dec  3 03:58:19.462109 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SCHEDULER
Dec  3 03:58:19.473509 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SCHEDULER_GROUP
Dec  3 03:58:19.473509 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_BUFFER
Dec  3 03:58:19.473509 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_HASH
Dec  3 03:58:19.473509 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_UDF
Dec  3 03:58:19.473509 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_TUNNEL
Dec  3 03:58:19.473509 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_L2MC not implemented
Dec  3 03:58:19.487324 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_IPMC not implemented
Dec  3 03:58:19.487324 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_RPF_GROUP not implemented
Dec  3 03:58:19.487324 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_L2MC_GROUP
Dec  3 03:58:19.487324 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_IPMC_GROUP not implemented
Dec  3 03:58:19.487324 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_MCAST_FDB not implemented
Dec  3 03:58:19.487324 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_BRIDGE
Dec  3 03:58:19.500851 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_TAM not implemented
Dec  3 03:58:19.500851 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_SRV6 not implemented
Dec  3 03:58:19.500851 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_MPLS not implemented
Dec  3 03:58:19.500851 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_DTEL not implemented
Dec  3 03:58:19.500851 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_BFD
Dec  3 03:58:19.500851 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ISOLATION_GROUP
Dec  3 03:58:19.500851 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_NAT not implemented
Dec  3 03:58:19.500851 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_COUNTER
Dec  3 03:58:19.500851 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_DEBUG_COUNTER
Dec  3 03:58:19.500851 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_MACSEC not implemented
Dec  3 03:58:19.500851 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_SYSTEM_PORT not implemented
Dec  3 03:58:19.500851 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_MY_MAC not implemented
Dec  3 03:58:19.500851 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_IPSEC not implemented
Dec  3 03:58:19.500851 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_BMTOR not implemented
Dec  3 03:58:19.500851 sonic NOTICE syncd#syncd: :- Syncd: command line:  EnableDiagShell=NO EnableTempView=YES DisableExitSleep=NO EnableUnittests=NO EnableConsistencyCheck=NO EnableSyncMode=NO RedisCommunicationMode=zmq_sync EnableSaiBulkSuport=NO StartType=cold ProfileMapFile=/tmp/sai.profile GlobalContext=0 ContextConfig= BreakConfig=
Dec  3 03:58:19.511296 sonic NOTICE syncd#syncd: :- loadFromFile: no context config specified, will load default context config
Dec  3 03:58:19.511296 sonic NOTICE syncd#syncd: :- insert: added switch: idx 0, hwinfo ''
Dec  3 03:58:19.511296 sonic NOTICE syncd#syncd: :- Syncd: zmq sync mode enabled via cmd line
Dec  3 03:58:19.511296 sonic NOTICE syncd#syncd: :- ZeroMQNotificationProducer: opening zmq ntf endpoint: ipc:///tmp/zmq_ntf_ep
Dec  3 03:58:19.511296 sonic NOTICE syncd#syncd: :- Syncd: zmq enabled, forcing sync mode
Dec  3 03:58:19.511296 sonic NOTICE syncd#syncd: :- ZeroMQSelectableChannel: binding on ipc:///tmp/zmq_ep
Dec  3 03:58:19.544761 sonic NOTICE syncd#syncd: :- zmqPollThread: begin
Dec  3 03:58:19.555664 sonic NOTICE syncd#syncd: :- isVeryFirstRun: First Run: True
Dec  3 04:07:39.283256 sonic DEBUG syncd#syncd: :> syncd_main: enter
Dec  3 04:07:39.295278 sonic NOTICE syncd#syncd: :- initialize: initializeing metadata log function
Dec  3 04:07:39.295278 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SWITCH
Dec  3 04:07:39.295278 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_PORT
Dec  3 04:07:39.295278 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_FDB
Dec  3 04:07:39.295278 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_VLAN
Dec  3 04:07:39.295278 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_VIRTUAL_ROUTER
Dec  3 04:07:39.295278 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ROUTE
Dec  3 04:07:39.295278 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_NEXT_HOP
Dec  3 04:07:39.307280 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_NEXT_HOP_GROUP
Dec  3 04:07:39.307280 sonic NOTICE syncd#syncd: :- threadFunction: starting timer watchdog thread
Dec  3 04:07:39.307280 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ROUTER_INTERFACE
Dec  3 04:07:39.307280 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_NEIGHBOR
Dec  3 04:07:39.307280 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ACL
Dec  3 04:07:39.307280 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_HOSTIF
Dec  3 04:07:39.307280 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_MIRROR
Dec  3 04:07:39.307280 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SAMPLEPACKET
Dec  3 04:07:39.307280 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_STP
Dec  3 04:07:39.307280 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_LAG
Dec  3 04:07:39.319232 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_POLICER
Dec  3 04:07:39.331281 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_WRED
Dec  3 04:07:39.331281 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_QOS_MAP
Dec  3 04:07:39.331281 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_QUEUE
Dec  3 04:07:39.331281 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SCHEDULER
Dec  3 04:07:39.331281 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SCHEDULER_GROUP
Dec  3 04:07:39.331281 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_BUFFER
Dec  3 04:07:39.331281 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_HASH
Dec  3 04:07:39.331281 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_UDF
Dec  3 04:07:39.331281 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_TUNNEL
Dec  3 04:07:39.331281 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_L2MC not implemented
Dec  3 04:07:39.331281 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_IPMC not implemented
Dec  3 04:07:39.331281 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_RPF_GROUP not implemented
Dec  3 04:07:39.331281 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_L2MC_GROUP
Dec  3 04:07:39.331281 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_IPMC_GROUP not implemented
Dec  3 04:07:39.331281 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_MCAST_FDB not implemented
Dec  3 04:07:39.331281 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_BRIDGE
Dec  3 04:07:39.331281 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_TAM not implemented
Dec  3 04:07:39.349650 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_SRV6 not implemented
Dec  3 04:07:39.349650 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_MPLS not implemented
Dec  3 04:07:39.349650 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_DTEL not implemented
Dec  3 04:07:39.349650 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_BFD
Dec  3 04:07:39.349650 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ISOLATION_GROUP
Dec  3 04:07:39.349650 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_NAT not implemented
Dec  3 04:07:39.349650 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_COUNTER
Dec  3 04:07:39.349650 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_DEBUG_COUNTER
Dec  3 04:07:39.349650 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_MACSEC not implemented
Dec  3 04:07:39.363246 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_SYSTEM_PORT not implemented
Dec  3 04:07:39.363246 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_MY_MAC not implemented
Dec  3 04:07:39.363246 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_IPSEC not implemented
Dec  3 04:07:39.363246 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_BMTOR not implemented
Dec  3 04:07:39.363246 sonic NOTICE syncd#syncd: :- Syncd: command line:  EnableDiagShell=NO EnableTempView=YES DisableExitSleep=NO EnableUnittests=NO EnableConsistencyCheck=NO EnableSyncMode=NO RedisCommunicationMode=zmq_sync EnableSaiBulkSuport=NO StartType=cold ProfileMapFile=/tmp/sai.profile GlobalContext=0 ContextConfig= BreakConfig=
Dec  3 04:07:39.363246 sonic NOTICE syncd#syncd: :- loadFromFile: no context config specified, will load default context config
Dec  3 04:07:39.363246 sonic NOTICE syncd#syncd: :- insert: added switch: idx 0, hwinfo ''
Dec  3 04:07:39.363246 sonic NOTICE syncd#syncd: :- Syncd: zmq sync mode enabled via cmd line
Dec  3 04:07:39.373537 sonic NOTICE syncd#syncd: :- ZeroMQNotificationProducer: opening zmq ntf endpoint: ipc:///tmp/zmq_ntf_ep
Dec  3 04:07:39.373537 sonic NOTICE syncd#syncd: :- Syncd: zmq enabled, forcing sync mode
Dec  3 04:07:39.373537 sonic NOTICE syncd#syncd: :- ZeroMQSelectableChannel: binding on ipc:///tmp/zmq_ep
Dec  3 04:07:39.383277 sonic NOTICE syncd#syncd: :- zmqPollThread: begin
Dec  3 04:07:39.407327 sonic NOTICE syncd#syncd: :- isVeryFirstRun: First Run: True
Dec  3 04:17:02.960856 sonic DEBUG syncd#syncd: :> syncd_main: enter
Dec  3 04:17:02.977585 sonic NOTICE syncd#syncd: :- initialize: initializeing metadata log function
Dec  3 04:17:02.978951 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SWITCH
Dec  3 04:17:02.979356 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_PORT
Dec  3 04:17:02.984942 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_FDB
Dec  3 04:17:02.996422 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_VLAN
Dec  3 04:17:02.996422 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_VIRTUAL_ROUTER
Dec  3 04:17:02.996422 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ROUTE
Dec  3 04:17:02.999558 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_NEXT_HOP
Dec  3 04:17:03.000043 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_NEXT_HOP_GROUP
Dec  3 04:17:03.000481 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ROUTER_INTERFACE
Dec  3 04:17:03.003243 sonic NOTICE syncd#syncd: :- threadFunction: starting timer watchdog thread
Dec  3 04:17:03.003243 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_NEIGHBOR
Dec  3 04:17:03.003243 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ACL
Dec  3 04:17:03.003243 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_HOSTIF
Dec  3 04:17:03.003243 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_MIRROR
Dec  3 04:17:03.003243 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SAMPLEPACKET
Dec  3 04:17:03.003752 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_STP
Dec  3 04:17:03.011918 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_LAG
Dec  3 04:17:03.012759 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_POLICER
Dec  3 04:17:03.013375 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_WRED
Dec  3 04:17:03.014052 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_QOS_MAP
Dec  3 04:17:03.014755 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_QUEUE
Dec  3 04:17:03.019484 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SCHEDULER
Dec  3 04:17:03.020373 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SCHEDULER_GROUP
Dec  3 04:17:03.021176 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_BUFFER
Dec  3 04:17:03.021837 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_HASH
Dec  3 04:17:03.022468 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_UDF
Dec  3 04:17:03.023312 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_TUNNEL
Dec  3 04:17:03.027222 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_L2MC not implemented
Dec  3 04:17:03.027222 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_IPMC not implemented
Dec  3 04:17:03.027222 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_RPF_GROUP not implemented
Dec  3 04:17:03.027222 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_L2MC_GROUP
Dec  3 04:17:03.027884 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_IPMC_GROUP not implemented
Dec  3 04:17:03.028565 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_MCAST_FDB not implemented
Dec  3 04:17:03.031234 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_BRIDGE
Dec  3 04:17:03.031234 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_TAM not implemented
Dec  3 04:17:03.035977 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_SRV6 not implemented
Dec  3 04:17:03.036530 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_MPLS not implemented
Dec  3 04:17:03.036921 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_DTEL not implemented
Dec  3 04:17:03.037325 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_BFD
Dec  3 04:17:03.037723 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ISOLATION_GROUP
Dec  3 04:17:03.038102 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_NAT not implemented
Dec  3 04:17:03.038486 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_COUNTER
Dec  3 04:17:03.038861 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_DEBUG_COUNTER
Dec  3 04:17:03.039262 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_MACSEC not implemented
Dec  3 04:17:03.043639 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_SYSTEM_PORT not implemented
Dec  3 04:17:03.044098 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_MY_MAC not implemented
Dec  3 04:17:03.044492 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_IPSEC not implemented
Dec  3 04:17:03.044887 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_BMTOR not implemented
Dec  3 04:17:03.045127 sonic NOTICE syncd#syncd: :- Syncd: command line:  EnableDiagShell=NO EnableTempView=YES DisableExitSleep=NO EnableUnittests=NO EnableConsistencyCheck=NO EnableSyncMode=NO RedisCommunicationMode=zmq_sync EnableSaiBulkSuport=NO StartType=cold ProfileMapFile=/tmp/sai.profile GlobalContext=0 ContextConfig= BreakConfig=
Dec  3 04:17:03.045206 sonic NOTICE syncd#syncd: :- loadFromFile: no context config specified, will load default context config
Dec  3 04:17:03.045286 sonic NOTICE syncd#syncd: :- insert: added switch: idx 0, hwinfo ''
Dec  3 04:17:03.045364 sonic NOTICE syncd#syncd: :- Syncd: zmq sync mode enabled via cmd line
Dec  3 04:17:03.046877 sonic NOTICE syncd#syncd: :- ZeroMQNotificationProducer: opening zmq ntf endpoint: ipc:///tmp/zmq_ntf_ep
Dec  3 04:17:03.047041 sonic NOTICE syncd#syncd: :- Syncd: zmq enabled, forcing sync mode
Dec  3 04:17:03.047157 sonic NOTICE syncd#syncd: :- ZeroMQSelectableChannel: binding on ipc:///tmp/zmq_ep
Dec  3 04:17:03.079446 sonic NOTICE syncd#syncd: :- isVeryFirstRun: First Run: True
Dec  3 04:26:19.845224 sonic DEBUG syncd#syncd: :> syncd_main: enter
Dec  3 04:26:19.847398 sonic NOTICE syncd#syncd: :- initialize: initializeing metadata log function
Dec  3 04:26:19.847398 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SWITCH
Dec  3 04:26:19.847398 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_PORT
Dec  3 04:26:19.847578 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_FDB
Dec  3 04:26:19.847964 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_VLAN
Dec  3 04:26:19.849964 sonic NOTICE syncd#syncd: :- threadFunction: starting timer watchdog thread
Dec  3 04:26:19.850094 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_VIRTUAL_ROUTER
Dec  3 04:26:19.851021 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ROUTE
Dec  3 04:26:19.851647 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_NEXT_HOP
Dec  3 04:26:19.852153 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_NEXT_HOP_GROUP
Dec  3 04:26:19.852659 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ROUTER_INTERFACE
Dec  3 04:26:19.853151 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_NEIGHBOR
Dec  3 04:26:19.853639 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ACL
Dec  3 04:26:19.854116 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_HOSTIF
Dec  3 04:26:19.854586 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_MIRROR
Dec  3 04:26:19.855082 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SAMPLEPACKET
Dec  3 04:26:19.855610 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_STP
Dec  3 04:26:19.856085 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_LAG
Dec  3 04:26:19.856569 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_POLICER
Dec  3 04:26:19.857046 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_WRED
Dec  3 04:26:19.857512 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_QOS_MAP
Dec  3 04:26:19.857979 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_QUEUE
Dec  3 04:26:19.858432 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SCHEDULER
Dec  3 04:26:19.858915 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SCHEDULER_GROUP
Dec  3 04:26:19.859426 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_BUFFER
Dec  3 04:26:19.859895 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_HASH
Dec  3 04:26:19.860348 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_UDF
Dec  3 04:26:19.860810 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_TUNNEL
Dec  3 04:26:19.862675 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_L2MC not implemented
Dec  3 04:26:19.863305 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_IPMC not implemented
Dec  3 04:26:19.863850 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_RPF_GROUP not implemented
Dec  3 04:26:19.864371 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_L2MC_GROUP
Dec  3 04:26:19.864922 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_IPMC_GROUP not implemented
Dec  3 04:26:19.865496 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_MCAST_FDB not implemented
Dec  3 04:26:19.866090 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_BRIDGE
Dec  3 04:26:19.866672 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_TAM not implemented
Dec  3 04:26:19.867213 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_SRV6 not implemented
Dec  3 04:26:19.867736 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_MPLS not implemented
Dec  3 04:26:19.868275 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_DTEL not implemented
Dec  3 04:26:19.868780 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_BFD
Dec  3 04:26:19.869278 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ISOLATION_GROUP
Dec  3 04:26:19.869758 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_NAT not implemented
Dec  3 04:26:19.870262 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_COUNTER
Dec  3 04:26:19.871866 sonic NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_DEBUG_COUNTER
Dec  3 04:26:19.873755 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_MACSEC not implemented
Dec  3 04:26:19.874423 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_SYSTEM_PORT not implemented
Dec  3 04:26:19.875005 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_MY_MAC not implemented
Dec  3 04:26:19.875658 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_IPSEC not implemented
Dec  3 04:26:19.876291 sonic WARNING syncd#syncd: [SAI_INTERFACE_QUERY.WARNING] mlnx_sai_interfacequery.c[495]- sai_log_set: mlnx_sai_interfacequery.c[495]- sai_log_set: SAI_API_BMTOR not implemented
Dec  3 04:26:19.876686 sonic NOTICE syncd#syncd: :- Syncd: command line:  EnableDiagShell=NO EnableTempView=YES DisableExitSleep=NO EnableUnittests=NO EnableConsistencyCheck=NO EnableSyncMode=NO RedisCommunicationMode=zmq_sync EnableSaiBulkSuport=NO StartType=cold ProfileMapFile=/tmp/sai.profile GlobalContext=0 ContextConfig= BreakConfig=
Dec  3 04:26:19.876889 sonic NOTICE syncd#syncd: :- loadFromFile: no context config specified, will load default context config
Dec  3 04:26:19.877074 sonic NOTICE syncd#syncd: :- insert: added switch: idx 0, hwinfo ''
Dec  3 04:26:19.877257 sonic NOTICE syncd#syncd: :- Syncd: zmq sync mode enabled via cmd line
Dec  3 04:26:19.879343 sonic NOTICE syncd#syncd: :- ZeroMQNotificationProducer: opening zmq ntf endpoint: ipc:///tmp/zmq_ntf_ep
Dec  3 04:26:19.879723 sonic NOTICE syncd#syncd: :- Syncd: zmq enabled, forcing sync mode
Dec  3 04:26:19.879963 sonic NOTICE syncd#syncd: :- ZeroMQSelectableChannel: binding on ipc:///tmp/zmq_ep
Dec  3 04:26:19.885712 sonic NOTICE syncd#syncd: :- isVeryFirstRun: First Run: True

MaratGubaiev commented 2 years ago

We can only make it work with some code changes (as I mentioned previously).

Do you intend to commit them to the master?

mint570 commented 2 years ago

We can only make it work with some code changes (as I mentioned previously).

Do you intend to commit them to the master?

The changes we made are listed in my previous comment. The first one is to provide a context config file to orchagent and syncd to enable zmq. The synd file can be passed in command line argument. But the orchagent file is hard-coded to be "/usr/share/sonic/hwsku/context_config.json". We just modify that to pass our specific file to enable zmq. That is not upstreamable. I think the clean way to fix it is to let sairedis know orchagent wants to do zmq in the init step, so it doesn't start with using redis by default. But I don't see an obvious way to do it.

The second one is a small change. We can upstream that.

The last one is to address an existing zmq issue: https://github.com/Azure/sonic-swss/issues/1145. Our solution might not be the best. But if you prefer our solution, we can upstream it.

MaratGubaiev commented 2 years ago

@kcudnik @mint570 I'm going to apply the changes described by @mint570 and create a pull request to commit them into the master as a result. But I want to be sure that we really need them in the master. What is our policy towards ZMQ?

notify @pshulik

MaratGubaiev commented 2 years ago

@mint570 You provided the general description of your changes. Can you also provide us what you exactly did?

kcudnik commented 2 years ago

@mint570 i dont like the idea to forward the event to original redis channel when using zmq mode, this will work, since OA is listening for that channel an manually handling each notification, but this is was workaround for handling async event by design, and your solution add another workaround on top of that, all this should be fixed at the core, but from long time nobody from swss owners handled this issue, but i dont have better solution in mind right now :/

@mint570 on your comment https://github.com/Azure/sonic-swss/issues/2014#issuecomment-966498311 and https://github.com/Azure/sonic-swss/issues/2014#issue-1046914900

when both syncd and OA passing "-z zmq_sync" context_confing.json should not be required, and default zmq configuration channels should be used, but not sure why this not happening in OA, from 1st post, it's cleary seend that RedisChannel is used, so it means that there is some bug there that is not properly handling that -z flag, when context_config is not specified, this is ether OA bug or sairedis lib bug,

@MaratGubaiev what changes are you talking about here? do you want to enabled zmq buy default on sonic master? adding @lguohan and @qiluo-msft, internally we do not use zmq yet in sonic, so this patch is not fully tested, and potential issues could arise that we are not aware yet, and i have no knowledge about zmq when it would be enabled by default

i think we currently have only 2 basic tests for test communication on zmq mode: sonic-sairedis/tests/BCM56850.pl / test_brcm_buffer_pool_zmq_sync_flag and test_brcm_buffer_pool_zmq one tests flags passed via command line, and other tests context config passed via command line

if you generate PR, azure pipeline shoul execute vs tests using zmq, but not sure if those are all tests

MaratGubaiev commented 2 years ago

@kcudnik I said about the steps mentioned in the issue itself, and also about the changes proposed in the comment: https://github.com/Azure/sonic-swss/issues/2014#issuecomment-984013783

Also I thought that the issue is a bug to be fixed, so I wandered was it a true issue with the existing code or not. I see that it is like an "issue to be discussed".

mint570 commented 2 years ago

@MaratGubaiev I can prepare the pr if you need. But our solution is only a workaround as @kcudnik said.

@kcudnik The reason we need to passed a context config is explained here: https://github.com/Azure/sonic-swss/issues/2014#issuecomment-966521738. In initSaiRedis(), orchagent does not apply zmq yet. It only applies zmq later by setRedisExtensionAttribute. In the initSaiRedis(), orchagent still uses Redis to do INIT_VIEW since that's the default. And INIT_VIEW will fail because syncd is in zmq mode. Orchagent will exit with error after INIT_VIEW fails.

kcudnik commented 2 years ago

then we can move set SAI_REDIS_SWITCH_ATTR_REDIS_COMMUNICATION_MODE into iniSaiRedis() this will eliminate then need for context_config.json fille, since still this is a bug